Apache symlink security issue fix/patch

by brian on January 14, 2013

There is a serious security hole in the way that Apache handles symlinks on shared servers.

This allows an exploited account on a server to view .php files owned by other accounts, thus escalating a single-account exploit to potentially many accounts on the one server.  This post describes how to plug these holes very portably.

Rather disappointingly, for various reasons to do with ownership of the problem, nobody has taken responsibility for this hack which is nearly as serious as the old suphp/DSO issue which allowed much the same sort of exploit (ie allowed users to read other users’ PHP files).

The exploit, in general terms, is to create a symbolic link file (eg public_html/fred.txt) pointing to a wp-config.php file (eg /home/otheracct/public_html/wp-config.php) which contains database user and password which will occasionally be the cpanel username/password.  The file is then readable via a web browser and the hacker proceeds.  If the user has been unwise enough to use their cpanel username/password for the database, it’s even worse for the account.

Many of you are probably aware that this is now actively being exploited by hackers in very, very large scale in the wild.  It  appears to have  affected several very large hosting companies recently  (eg one of Australia’s top companies recently reported an exploit like this).   Surprisingly, at the time of writing (Jan 2013)  it seems many large companies still have their servers unpatched against this issue.

Fixes

At present there are two possible fixes –

1. In .htaccess or global httpd.conf, add SymLinksIfOwnerMatch

2. Change permissions on config .php files (or all executable/data files) to be mode 600

Both of these have weaknesses – #1 as exploiter can often simply disable SymLinksIfOwnerMatch by overwriting .htaccess, #2 as users have to remember to secure their files and many users will not even know this is needed.

Important point: changing permissions blocks the symlink hack in the kernel.  The weakness with changing permissions is only if you leave it up to users; if you enforce restricted permissions on .php files I believe the protection is 100%.

Below I’m publishing/advertising two fixes, the first from cPanel’s forum a few years ago.  There are other fixes that are more involved that change the way the kernel works.  While these also resolve the issue, installing both of these fixes is currently a 100% protection.

Fix 1 – Forcing SymLinksIfOwnerMatch to always be on:

Steven of Rack911 has published an easyapache patch which adds the file /scripts/before-apache-make to force SymLinksIfOwnerMatch to be always on.  This largely mitigates the issue, although there is apparently a race condition which can be exploited to work around the patch. Still, it makes the hole harder to exploit as is.

tinyurl.com/symlink-patch  (tinyurl created by me) or the actual link:
http://forums.cpanel.net/f185/how-prevent-creating-symbolic-links-non-root-users-202242-p2.html#post996441

This is a cPanel specific patch, but should be adaptable to other control panel varieties.  On cPanel, you’ll need to run cPanel’s easyapache or other httpd rebuild process after installing the patch file.  This avoids the quick and easy skript kiddie exploits, but does not avoid the race condition (which would be a lot harder to exploit and may not yet be fully in the wild).

I’ve been using this successfully on our servers for about 8 months (since April 2012) and it has been widely adopted and used as you can see in the thread above.  At January 2013 my guess is that just this one step mitigates most of this security hole.

It’s worth stressing that this does not provide 100% mitigation; it just makes it harder.  Changing permissions does, AFAIK, provide 100% mitigation.

[News 20/1: Bluehost appear to have developed a patch which closes the race exploit – see http://tinyurl.com/apache-fstat-patch-bluehost]

[News Feb 2013: cPanel have released a patch which is selectable in easyapache.  I beleive this is the bluehost patch above, though they haven’t made it clear what it does.  If you check the patch and discover details, please let us know.  The bluehost patch uses fstat() to check file ownership *after* the file has been opened, which is the only correct way to implement the SymLinksIfOwnerMatch check.]

 

Fix 2 – periodically force .php file permissions to be mode 600

This is a complete fix as it prevents the symlink file from being readable.

I’ve written a couple of scripts to change permissions on new .php files.

php-mode-fast – this should be run hourly to change all new *config*.php files from /etc/cron.hourly.  It changes permissions on the most obvious and easily found configuration files.

php-mode-600 – this should be run daily from /etc/cron.daily.  It changes all PHP files on the server to remove group and all read permission.

I’ve been using these two scripts on our servers for 4 months (since Sept 2012 approx).

An important point here (as made above, but bears repeating): changing permissions gives 100% protection against the symlink hack.  The reason for this is that the hack relies on the webserver (apache) being able to read the file.  When you change permissions to 600, you are denying the webserver direct file access.  The file still runs when accessed though, as the webserver changes the PHP user to the end user (account owner) every time it runs a script, and the account owner has permission to read the file.  So, a .php file that is mode 600 cannot be read by Apache and thus cannot be read by Apache on behalf of another account.  Writing here as someone who has been around Unix for a long time, I’m pretty sure this is solid and I’ll update this if I ever learn otherwise.  Lack of widespread understanding of this point doesn’t make it any less true!

Note: for this fix to work, you will need to be running PHP as a separate user for each account!  If you are not doing that, I’m sorry, but there is no point in installing this fix as users are already able to read all .php files on the server.  In some installations this is called suphp, but there are other more recent modes that also support running users under separate partitions.  If you’re not already doing this you should switch as a matter of urgency.

Second note: There is a workaround if a .php file needs public permission – the unusual mode 744 is never changed.  As far as I’ve experienced, this has only been needed in the unusual circumstance that a .php file under /home is include()d by PHP scripts running in other accounts. We needed this for our (outdated) holding page for empty accounts and it’s theoretically possible a user might need it under very unusual circumstances.  I would not advertise the workaround – rather wait till you are asked by a user; you’ll probably find nobody asks – nobody has asked us yet!

A third note: If you have been comprehensively hacked, you must change all database passwords on your server.  Unfortunately, whether all accounts have been hacked or not, the bad guys probably now have database passwords for every single account on the server (assuming they could locate a wp-config.php or configuration.php file).  This is a lot of work and I haven’t yet seen an automated tool that changes database passwords in mysql then edits the files for you (if you find one, please comment here).

Stop press 26 Jan 2013: Installatron have been setting mode on config files when installed to 600 when the server permits) as mentioned here.  They’ve probably saved thousands of sites from being hacked!

Update March 2013:  cPanel have finally decided to help out and have released an easyApache option, which has to be selected manually, unfortunately.  Note that if you have been systematically hacked with this hack, you need to go through and change every single database password on the server to be secure, and you need to inform your users that the information in their databases, including any credit card information, has been compromised.  I beleive this update is based on the bluehost patch given on this page.

 

To install the cron scripts:

Download the tar archive of files first: [download id=”11″]

tar xvzf php-mode.tgz
cd php-mode
cp php-mode-600 /etc/cron.daily/php-mode-600
cp php-mode-fast /etc/cron.hourly/php-mode-fast
chmod 750 /etc/cron*/php-mode*
chown root.root /etc/cron*/php-mode*

There is also a simple install script called install_script which does the above, so an alternative (beta) install technique would be:

tar xvzf php-mode.tgz && sh php-mode/install_scripts

[update July 2013]
If you are looking for a well written and very readable explanation of the problem, check out this link:
http://blog.sucuri.net/2013/05/from-a-site-compromise-to-full-root-access-symlinks-to-root-part-i.html

Previous post: