Apache symlink security issue fix/patch

by brian on January 14, 2013

There is a serious security hole in the way that Apache handles symlinks on shared servers.

This allows an exploited account on a server to view .php files owned by other accounts, thus escalating a single-account exploit to potentially many accounts on the one server.  This post describes how to plug these holes very portably.

Rather disappointingly, for various reasons to do with ownership of the problem, nobody has taken responsibility for this hack which is nearly as serious as the old suphp/DSO issue which allowed much the same sort of exploit (ie allowed users to read other users’ PHP files).

The exploit, in general terms, is to create a symbolic link file (eg public_html/fred.txt) pointing to a wp-config.php file (eg /home/otheracct/public_html/wp-config.php) which contains database user and password which will occasionally be the cpanel username/password.  The file is then readable via a web browser and the hacker proceeds.  If the user has been unwise enough to use their cpanel username/password for the database, it’s even worse for the account.

Many of you are probably aware that this is now actively being exploited by hackers in very, very large scale in the wild.  It  appears to have  affected several very large hosting companies recently  (eg one of Australia’s top companies recently reported an exploit like this).   Surprisingly, at the time of writing (Jan 2013)  it seems many large companies still have their servers unpatched against this issue.

Fixes

At present there are two possible fixes –

1. In .htaccess or global httpd.conf, add SymLinksIfOwnerMatch

2. Change permissions on config .php files (or all executable/data files) to be mode 600

Both of these have weaknesses – #1 as exploiter can often simply disable SymLinksIfOwnerMatch by overwriting .htaccess, #2 as users have to remember to secure their files and many users will not even know this is needed.

Important point: changing permissions blocks the symlink hack in the kernel.  The weakness with changing permissions is only if you leave it up to users; if you enforce restricted permissions on .php files I believe the protection is 100%.

Below I’m publishing/advertising two fixes, the first from cPanel’s forum a few years ago.  There are other fixes that are more involved that change the way the kernel works.  While these also resolve the issue, installing both of these fixes is currently a 100% protection.

Fix 1 – Forcing SymLinksIfOwnerMatch to always be on:

Steven of Rack911 has published an easyapache patch which adds the file /scripts/before-apache-make to force SymLinksIfOwnerMatch to be always on.  This largely mitigates the issue, although there is apparently a race condition which can be exploited to work around the patch. Still, it makes the hole harder to exploit as is.

tinyurl.com/symlink-patch  (tinyurl created by me) or the actual link:
http://forums.cpanel.net/f185/how-prevent-creating-symbolic-links-non-root-users-202242-p2.html#post996441

This is a cPanel specific patch, but should be adaptable to other control panel varieties.  On cPanel, you’ll need to run cPanel’s easyapache or other httpd rebuild process after installing the patch file.  This avoids the quick and easy skript kiddie exploits, but does not avoid the race condition (which would be a lot harder to exploit and may not yet be fully in the wild).

I’ve been using this successfully on our servers for about 8 months (since April 2012) and it has been widely adopted and used as you can see in the thread above.  At January 2013 my guess is that just this one step mitigates most of this security hole.

It’s worth stressing that this does not provide 100% mitigation; it just makes it harder.  Changing permissions does, AFAIK, provide 100% mitigation.

[News 20/1: Bluehost appear to have developed a patch which closes the race exploit – see http://tinyurl.com/apache-fstat-patch-bluehost]

[News Feb 2013: cPanel have released a patch which is selectable in easyapache.  I beleive this is the bluehost patch above, though they haven’t made it clear what it does.  If you check the patch and discover details, please let us know.  The bluehost patch uses fstat() to check file ownership *after* the file has been opened, which is the only correct way to implement the SymLinksIfOwnerMatch check.]

 

Fix 2 – periodically force .php file permissions to be mode 600

This is a complete fix as it prevents the symlink file from being readable.

I’ve written a couple of scripts to change permissions on new .php files.

php-mode-fast – this should be run hourly to change all new *config*.php files from /etc/cron.hourly.  It changes permissions on the most obvious and easily found configuration files.

php-mode-600 – this should be run daily from /etc/cron.daily.  It changes all PHP files on the server to remove group and all read permission.

I’ve been using these two scripts on our servers for 4 months (since Sept 2012 approx).

An important point here (as made above, but bears repeating): changing permissions gives 100% protection against the symlink hack.  The reason for this is that the hack relies on the webserver (apache) being able to read the file.  When you change permissions to 600, you are denying the webserver direct file access.  The file still runs when accessed though, as the webserver changes the PHP user to the end user (account owner) every time it runs a script, and the account owner has permission to read the file.  So, a .php file that is mode 600 cannot be read by Apache and thus cannot be read by Apache on behalf of another account.  Writing here as someone who has been around Unix for a long time, I’m pretty sure this is solid and I’ll update this if I ever learn otherwise.  Lack of widespread understanding of this point doesn’t make it any less true!

Note: for this fix to work, you will need to be running PHP as a separate user for each account!  If you are not doing that, I’m sorry, but there is no point in installing this fix as users are already able to read all .php files on the server.  In some installations this is called suphp, but there are other more recent modes that also support running users under separate partitions.  If you’re not already doing this you should switch as a matter of urgency.

Second note: There is a workaround if a .php file needs public permission – the unusual mode 744 is never changed.  As far as I’ve experienced, this has only been needed in the unusual circumstance that a .php file under /home is include()d by PHP scripts running in other accounts. We needed this for our (outdated) holding page for empty accounts and it’s theoretically possible a user might need it under very unusual circumstances.  I would not advertise the workaround – rather wait till you are asked by a user; you’ll probably find nobody asks – nobody has asked us yet!

A third note: If you have been comprehensively hacked, you must change all database passwords on your server.  Unfortunately, whether all accounts have been hacked or not, the bad guys probably now have database passwords for every single account on the server (assuming they could locate a wp-config.php or configuration.php file).  This is a lot of work and I haven’t yet seen an automated tool that changes database passwords in mysql then edits the files for you (if you find one, please comment here).

Stop press 26 Jan 2013: Installatron have been setting mode on config files when installed to 600 when the server permits) as mentioned here.  They’ve probably saved thousands of sites from being hacked!

Update March 2013:  cPanel have finally decided to help out and have released an easyApache option, which has to be selected manually, unfortunately.  Note that if you have been systematically hacked with this hack, you need to go through and change every single database password on the server to be secure, and you need to inform your users that the information in their databases, including any credit card information, has been compromised.  I beleive this update is based on the bluehost patch given on this page.

 

To install the cron scripts:

Download the tar archive of files first: Download php-mode.tgz [6553 downloads] tgz

tar xvzf php-mode.tgz
cd php-mode
cp php-mode-600 /etc/cron.daily/php-mode-600
cp php-mode-fast /etc/cron.hourly/php-mode-fast
chmod 750 /etc/cron*/php-mode*
chown root.root /etc/cron*/php-mode*

There is also a simple install script called install_script which does the above, so an alternative (beta) install technique would be:

tar xvzf php-mode.tgz && sh php-mode/install_scripts

[update July 2013]
If you are looking for a well written and very readable explanation of the problem, check out this link:
http://blog.sucuri.net/2013/05/from-a-site-compromise-to-full-root-access-symlinks-to-root-part-i.html

  • Ed

    Thank you Brian for bringing this all together in one location.

    You mention a workaround if php files need public permission. Is that simply to chmod the file to 744?

    As I’ve followed that cPanel thread for some time would be interested to know if you are seeing these account exploits happen under any specific circumstances?

  • re the workaround – yes, 744 is the “leave my permissions alone” workaround. Rarely needed.

    I haven’t seen much specific targeting happen and as far as I can tell they are generically looking for vulnerable sites. However, I wouldn’t put it past the hacker community to use these to target specific sites. They need to exploit at least one site on the server (or server farm, if a NAS/SAN is being used for user files, which is one reason why this is a nasty bug!)

  • Ed

    Thank you Brian for the reply.

    Just thought I might mention that ConfigServer has updated their CXS application to address this attack vector.

  • Hi Ed, I’ve been chatting with Configserver and asked them to look into it. As usual the ever great Jonathan updated CXS to give some assistance. Trying to work out how to test whether the server is vulnerable but it’s desperately hard to automate.

    Also, bluehost appear to have written a patch which closes even the race vulnerability (the Apache patch mentioned above has a race vulnerability, this patch closes that hole) at http://tinyurl.com/apache-fstat-patch-bluehost.

  • Scott

    Hi Brian,

    Thanks for this post, one question though..would I need to first uninstall the SteveC patch and then install the bluehost patch? Or can both be used together?

    Also can I use the same install method that SteveC used with the BlueHost patch?

    Thanks.

    • Hi Scott,

      I think both work together and have slightly different intents. The SteveC patch forces SymLinksIfOwnerMatch on, always a good idea.

      The bluehost patch changes the way the symlink ownership is checked to not be subject to a race conditions, also a good idea. I don’t think the bluehost patch overlaps the SteveC patch. I haven’t checked though, please do take a look for us.

    • ps: I’d definitely uses SteveC’s “before_apache_make” method to ensure Apache is repatched every time it is downloaded.

  • Pingback: WordPress hackeado con el XSS UTF-7 descargar gratis | Zonadictz()

  • Even with all this tweaks, a malicious script cannot acces through a webpage resources located on that server, but if it has ftp access can list, download, etc.. everything on the server, except /home directory. The only option (which is not an option for customers) is totally disable “symlink” function from php.ini. Bytheway how do you apply the Bluehost patch because is the only thing I didn’t made it until now because I do not understand how to use it.

    • Yes, these are patches to assist with the Apache symlink issue and do not close every other security weakness in the system 🙂

      If there are weaknesses in ftp, it would need to be patched or correctly configured so one can’t escape the home directory. Almost no protection is provided by disabling symlinks in php.ini as symlinks can be created in many other ways – through perl, binaries, commands, etc – it’s impossible to close off all the options.

      The bluehost patch is applied with the “patch” command from the correct folder. I haven’t worked out the detailed commands to do that yet, but once done, you should change the “before_apache_make” file to include the correct commands to repatch every time.

  • Hello

    this fix run in server with suphp ?

    Fix 2 – periodically force .php file permissions to be mode 600

    SuPHP does not execute php with permission 600 in my server ( tested )

    I see ( php-mode-600 )

    #===========================================================================
    #
    # Do a run through all home dirs looking for readable PHP files.
    # And change them to mode 600. This works fine with PHP.
    # These could be attacked by the Apache symlink vulnerability
    # if they were left readable to all.
    #
    # This is suitable to be run daily; it is likely to be cpu-intensive.
    #
    # BrianC August 2012 http://www.whmscripts.net
    #
    #===========================================================================

    Thank you
    Marcelo Konrath

    • Marcelo,

      This fix is intended for a server running suphp. Suphp works with native Unix file permissions and runs each user’s PHP scripts under their own userid, so they thus cannot normally read other users’ files. The apache hack uses the fact that Apache has more access to read the text files, and the symlink check and permission “fix” scripts make this impossible.

      If your site does not work when you change the script to mode 600 it probably means that all PHP scripts run under a single common user. Under that scenario, the symlink protection may not work at all, and the PHP scripts, if they are smart enough can often get access to all the other user accounts on the server anyway. It’s considered a lot less secure than suphp by most hosters.

      Cheers,

      Brian

  • Anonymous

    There seems to be an illusion that the Rack911 patch some how resolved people’s security concerns.
    This simply isn’t true.

    The Rack911 effectively did one thing; change ‘FollowSymlinks’ to ‘SymlinksIfOwnerMatch’. So basically you’re left with an option to turn ON or OFF symlinks that must match the owner. That’s it.. nothing more. It saves you roughly 20 characters worth of typing.

    The core issues being:
    1. Most httpd.conf files are set up such that their users can create an .htaccess file with ‘Options -FollowSymlinks -SymlinksIfOwnerMatch’, then get around the Rack911 patch anyways. And, due to how Apache allows an admin to blanketly disable/enable Options, but not specific KINDS of options, it’s just a slide of hand.
    2. More importantly, the Rack911 patch doesn’t resolve the race condition that allows an attacker to switch out the symlink in between the check of “does the owner match the symlink” and “access the file”.

    The new Bluehost.com patch does a great job of stopping the race condition. However, it also prevents users from accessing ANY kind of file they don’t “own”; “own” being a loose word due to how the patch determines who “owns” a file.

    While the new patch doesn’t resolve 100% of a user’s ability to access files they don’t own, it does a much better job than the Rack911 patch.

    • So embarrassed that this well-researched comment didn’t get approved quickly; I apologize!

      You’re completely correct on all points. The original Rack911 patch did stop exploits at the time, but isn’t a complete fix. I’ve updated the article to make it clearer that a permission fix is the only real fix. Obviously the bluehost patch fixes the problem. Cloudlinux and a few other packages appear to have comprehensive and well executed solutions as well.

  • Anonymous

    Does it mean that the Rack911 patch fails if the .htaccess files are allowed to disable the SymlinksIfOwnerMatch directive?

    So, now an attacker can place a .htaccess file with the option:

    Options -SymlinksIfOwnerMatch and as a result of this disable SymlinkIfOwnerMatch?

    how will that allow the attacker to make Apache follow symlinks?

    because, FollowSymLinks is required for Apache to follow the symlinks. And with the Rack911 patch, FollowSymlinks is translated to SymlinkIfOwnerMatch which is disabled now. As a result of this, even FollowSymlinks is disabled.

    So, how is the Rack911 patch not effective if someone disables SymlinksIfOwnerMatch directive in the .htaccess file?

    • No, this isn’t the way it works at all. The Rack911 patches locks SymLinksIfOwnerMatch permanently on, so it can’t be disabled in .htaccess or anywhere else. I suspect that attempts to disable SymLinksIfOwnerMatch appear to succeed, but they are ignored internally and it stays enabled.

      If FollowSymLinks is turned off, symlinks won’t be followed and the exploit won’t work.

  • Pingback: Linux Stuff | My WordPress Website()

  • Francis Ndungu

    Thank you for this info.
    I got hacked once and all wordpress files injected with codes. Today i got malware report and on checking the site which was suspended, i got all folders from root directory and a specific file to hack all wordpress sites
    wpmass.php
    I have installed php-mode-600.

    I am now trying to do SymLinksIfOwnerMatch.

    Thanks how can i buy you a beer?

    see below codes

    Wordpress MassDeface(ReCoded By G3mbel_mOxer)



    Thank you for this

  • quietFinn

    Seems php-mode-600 finds lots of files under directory /home/virtfs so it’s maybe good to change line:
    find /home \( -iname ‘*.php’ -o -iname ‘*.phtml’ \) \
    to:
    find /home -path /home/virtfs -prune -o \( -iname ‘*.php’ -o -iname ‘*.phtml’ \) \

  • Francis Ndungu

    Brian I have a problem with all webmail clients and phpmyadmin. I get “Access Denied” on all accounts. After contacting cpanel they responded as follows

    The custom cron script at /etc/cron.daily/php-mode-600 and /etc/cron.hourly/php-mode-fast traverses the /home directory looking for PHP scripts which are not mode 600 and attempts to change the permissions on these scripts.

    WHM/cPanel uses VirtFS mounts under /home/virtfs whenever ‘Jailed Shell’ is set for an account under WHM >> Home >> Account Functions >> Manage Shell Access, and WHM/cPanel 11.38 also started using VirtFS for any shell set to ‘Disabled’ in order to execute cron jobs and piped mail programs for additional security. These VirtFS mounts replicate other filesystem mounts under /home/virtfs, and so if VirtFS is mounted at the time the cron script runs, the ‘find’ command will unintentionally also find all of the PHP scripts under /usr/local/cpanel (via /home/virtfs/*/usr/local/cpanel) and attempts to “fix” these when they should be left untouched.

    phpMyAdmin is currently working, but unless the /etc/cron.daily/php-mode-600 and php-mode-fast scripts are modified to exclude scanning /home/virtfs, or else disabled, this problem will likely re-occur.

    Just in case someone else have the same problem. You can remodify the script to check only public_html folders not the whole /home/ directory

    I have disabled the crone for now until hoping that we will get it fixed soon. Its a great tool.

    Francis

    • Francis, Thanks for letting us know this; we’ve had the same problem. Give me a few hours and I’ll publish a fix.

      It seems to affect squirrelmail and roundcube more than the others.

      An easy quick fix for many people would be to add “-xdev” to the find options early on, but this won’t work for people with all filesystems in one.

  • OK – fix published and downloadable; the php-mode-600 script was the one causing this problem, and it now will only change modes for .php/.phtml files under public_html folders.

    This won’t correct modes on any add-on domains in folders not under public_html.

    If you have been affected by this, there are some shell commands you can run to restore the correct file permissions:
    HOME=/root /usr/local/cpanel/bin/update-roundcube –force
    HOME=/root /usr/local/cpanel/bin/update-squirrelmail –force
    HOME=/root /usr/local/cpanel/bin/update-horde –force

    The HOME=/root may be required to pick up the root mysql password from /root/.my.cnf on some systems, and may not be needed on others.

    If you are in a hurry for now, you can also prevent the daily cron script with:
    chmod 0 /etc/cron.daily/php-mode-600

  • Francis Ndungu

    Hi Brian,
    many thanks for the fixing. I have downloaded and uploaded and re-enabled the crons back to 750.
    My softaculous was also affected and returns an error. I have updated cpanel to fix most permissions.

    I need to reupdate softaculous but cant gt how to any idea?

    Once again many thanks

    • jackb

      Please check the permissions of /usr/local/cpanel/php/cpanel.php

      Once it is set to 644, Softaculous will work again. If it’s already 644, you may need to reinstall Softaculous to ensure that all permissions are correct.

      Hope this helps!

  • Francis Ndungu

    Brian,
    Just an update i had to do it manually changing phpmyadmin .php files back 0644
    /usr/local/cpanel/base/3rdparty/phpMyAdmin
    And Softaculous as well to 0644
    /usr/local/cpanel/whostmgr/docroot/cgi/softaculous/enduser

    I will continue doing that until all files are back

    The problem is now sorted out. Waiting to see if the cron tab will affect them again.

  • Ishan Garg

    Hello Brian,

    I am new to hosting and cpanel operation. While I understand the issue but I am u able to figure out how to apply the above fixes. Would be glad if you can assist us with a simple procedure to do so..

    Thanks,
    Ishan

    • Hi, I’m afraid helping with copying files to a cron folder is not something I do, just for lack of time. If you read up a little on Unix you should be able to manage it!

  • Xin

    Brian,

    Thank you for the solution. It worked excellently.

    Based on your scripts, I’v created another one for me that runs every minute to detect any change on public_html and applies 600 to *.php. Since it checks for the time stamp on public_html only, it very fast.

    Here is the script. Assume the script resides in /root/tools and added to crontab as “*/1 * * * * /root/tools/php-mode-go”. The first time it runs, it compares for time stamp at the beginning of today. The rest, a minute ago.

    With that, the php-mode-600 will cover any possible *php not processed by php-mode-go due to the race condition.

    #!/bin/bash
    #
    # BrianC July 2013 wd3.com.au/whitedoggreenfrog.com
    #
    mytmpfile=/tmp/php-mode-go
    touch $mytmpfile
    LAST=/root/tools/php-mode-go.last
    if [ ! -e $LAST ]
    then
    touch -t `date +%m%d0000` $LAST
    fi
    cd /home
    for dir in *
    do
    if [ ! -d “$dir/public_html” ]
    then
    continue
    fi
    CHANGED=`find “$dir” -maxdepth 1 -type d -name public_html -newer $LAST`
    if [ “$CHANGED” != “” ]
    then
    #echo $dir/public_html
    find “$dir/public_html” \( -iname ‘*.php’ -o -iname ‘*.phtml’ \) -perm /044 ! -perm 744 -print0 2>/dev/null | xargs -0 chmod go= $mytmpfile
    fi
    done
    touch -t `date +\%y\%m\%d\%H\%M\.\%S` $LAST
    rm “$mytmpfile” 2>/dev/null

    • Thanks Xin! The problem with that approach, of course, is that it will miss files in subfolders. Granted, this won’t happen often, and of course running the other script will catch those anyway. The main files you want to protect are the configuration.php/wp-config.php files.

      I have to say that as a sysadmin I always cringe at the thought of running cron jobs every minute. Having files unprotected for a few minutes is a very small risk, given the attacker would have to be checking constantly. If you really want instant protection, you could look at the INOTIFY kernel feature, which will notify a script when a file is created or changed.

  • Host malabar

Previous post: