Cleaning up a compromised Drupal site
When running a web-based CMS like Drupal or WordPress, it is not uncommon to have your site compromised by hackers who inject malicious files to use your site as a hosting platform for spaming, botnets, or other malicious activies. If this happens to you, don't fret, as it is very unlikely you were specifically targeted. Instead, it's most likely your site was found with automated scanners and targeted simply because it was identified as being out of date with security patches. To protect yourself, there's a few things you should do:
- Regularily review and install security patches on the core CMS. Both Drupal and Wordpress will give you notifications when there are available updates you should install.
- Regularily review and install security paches on your plugins. Again, both CMS will give you notification on this.
- Keep regular backups of your site files and databases so you can quickly restore to the last known-good state.
- Rotate all your passwords. Connections to the server for FTP/SSH, your database connections, and your site administrator passwords should all be changed regularily so that if they become compromised, there is a short lifespan of them being valid. This reduces the window where your site can be hacked.
- If you can afford to spend more on hosting, split your admin interface from your publishing interface, and run 2 seperate servers. Keep your admin interface behind a private secure login (like a VPN), and make sure your public interface cannot modify files on the server or make database changes. This can be a challenge if you support comments or user-generated articles, but both Drupal and WordPress can be run in this mode. I'll write up tutorials in the future on how to design and build this model.
Needless to say in the 5 years since I had last done significant work on my site, I had gotten lazy and not stayed on top of security updates for my Drupal site. This ended up biting me, as my site got compromised by one of the well-known Drupal exploits. Here's how I noticed I had been hacked, and how I went about cleaning it up:
Noticing the site is compromised
My Drupal installation was out of date, and I knew I should address any security updates before messing with other changes I wanted to put in place. So I immediately started by backing up the site files and databases first. As a precaution, I first ssh into my server and do a directory listing. In the past, automated exploits for Drupal have meant that files get added or modified to the filesystem. Usually these are easy to find by date stamps or doing a diff of what files I maintain in source control. This is when I noticed something was wrong, in the root of my site there was a suspicious file:
Well look at that, a file, wp-post.php was added to my system this past year, and my hosting provider caught it, renamed it and changed permissions so it can’t be executed. I’m pretty sure that’s an exploit masquerading as WordPress based on the file name. Let’s see what it contains by running less wp-poast.php.suspected:
Oh yeah, that’s definitely an exploit based on hijacking WordPress templates and search forms. It’s using PHP’s eval() to parse and execute a base64 encoded PHP script. The base64 encoding is to make it illegible and is a poor attempt to get past security scans. Since I’m running Drupal, not WordPress, this one is pretty obviously out of place and I need to take the time to clean it up properly. Luckily, I keep the entire state of my site backed up and saved in source control, so it’s easy to copy it down of the current filesystem and do a diff to see what’s changed from the last time I was in here.
Compare to a known-good backup
I make sure to always keep a full backup of my site every time I make changes. If you don't have a backup, then you'll have to go through the much more tedious process of getting the exact version of the Drupal Core and the various plugins that you use on your site. You need a clean set of files to compare against so you know what the hackers have modified.
My backup process is as follows:
- Use rsync over ssh to download all files to my local git repo
rsync -vzr --delete --exclude '.gitignore' email@example.com:domains/anthonymclin.com/html/ ~/git/anthonymclin.com/html
- Compare the diff to my site saved in source control
Yep, definitely got infected by some kind of automated script. The PHP scripts are all full of obfuscated code snippets. Since I have a known-good backup, this is easy to cleanup. I can simply delete any of the added files, restore the files of mine the script replaced (it renamed an index.html to index.html.bak.bak) and review the modified files to cleanup what it changed.
But before that, I want to make sure they didn’t get database access. I check the Drupal users admin page to make sure there aren’t any users added that I don’t expect to see, and it’s clean.
Looking at the files that change is always instructive. A fun trick these hackers like to use is to pad a bunch of line with whitespace, so that it looks like the diff on a file doesn’t contain any changes:
But if you scroll over horizontally in your editor, you’ll see there’s a bunch code injected at the end of the line after a bunch of whitespace:
The hackers are trying to trick the lazy. By padding their changes with a bunch of whitespace, they push the changes off the horizontally viewable area of any editor, so a reviewer checking the code would likely think there's nothing but a simple whitespace change and miss the hidden code. Always make sure to scroll over and verify this before committing the cleanup, otherwise there's a good chance you'll leave your site in an infected state where the hackers can quickly compromise it again even if you've done the security patches.
Publishing a known-clean state of the site
I don’t keep the generated cache files in source control, and it’s likely they may be infected with whatever the hacking toolkit injected into pages. So I delete everything under /sites/default/files/css/ and sites/default/files/js/ Those can easily be regenerated later.
After reverting everything the hackers changed I delete everything on my server, and push up the known-good files for a clean site state.
Applying security patches
To prevent the hackers from getting back in, let’s deal with the Drupal core security upgrades as well as any modules.
First the Drupal Core, I’m on 7.5 and the latest for this version is 7.61. Drupal doesn’t support online upgrades of the core system, so I need to download 7.61 manually, copy it into my local working directories, diff and commit any changes, then deploy that to the server. From my local project folder I run this command to do it in one step:
curl https://ftp.drupal.org/files/projects/drupal-7.61.tar.gz | tar xvz --strip 1
I review what files are added and changed, and commit them to my local repo. I also manually review /sites/default/default.settings.php to see what has changed and to determine if any of those changes should be copied into my settings.php. I also review the .htaccess to make sure the settings for my hosting provider and the redirects I prefer aren’t wiped out. Once satisfied with the changes, I commit them to my source control, so I have a record of this state.
Next I put my site in maintenance mode, then upload the clean version of the filesystem for the latest Drupal from my local copy. Now the report shows Drupal 7.61, but we’re not done since database updates haven’t been run:
A couple of quick steps at this point:
- Put the site in maintenance mode: /?q=admin/config/development/maintenance
- Run the database update scripts: /update.php
- Bring the site back out of maintenance mode: /?q=admin/config/development/maintenance
At this point, my Drupal core is up to date, and backed up, but I need a copy of the database changes as well. So I backup the database and save that in my source control too. Again, this is really important so you always have known-good states to work from in the future.
Update modules with security patches
Now we can move on to the module updates by visiting /admin/reports/updates. I review the list of modules with updates and identify which ones I want to upgrade in place, starting with the ones highlighted as security upgrades. For each, I run the upgrade on the server:
- XML Sitemap 7.x-2.3 to 7.x-2.6
- File (Field) Paths 7.x-1.0 to 7.x-1.1
- Lightbox2 7.x-1.0-beta1 to 7.x-1.0-beta2
- Views 7.x-3.14 to 7.x-3.20
- Entity Reference 7.x-1.2 to 7.x-1.5
- Entity API 7.x-1.8 to 7.x-1.9
- Bbcode 7.x-1.0 to 7.x-2.0
Naturally I read the release notes for each before upgrading to understand why the change and if there are any concerns or potential breakages I should be aware of.
Now that the modules with security updates are patched, I go through the modules with non-critical updates and do the same upgrade process:
- Advanced help
- Chaos tool suite
- Global Redirect
- Job Schedule
- Libraries API
- Semantic Views
- Views Bulk Operations
Now everything is up to date. But each time I run update.php, I’m getting an error message. Apparently I missed that at some point in the past I had installed a module without making a local backup. So in my haste to cleanup modified files, I ended up deleting the module verify_site and so I'm getting the error "User warning: The following module is missing from the file system: site_verify":
This is easily remedied. I visit the page for the module on Drupal.org, find the link for the .tar.gz file and right-click to copy the URL:
Then in /admin/modules/install I just paste in the copied URL and hit install:
After enabling the module, I go back to /update.php since this module is likely newer than what I was using and there may be a database update. But not this time:
Now that my running installation is up to date, there’s three last steps to make sure I have a copy of everything backed up for next time.
- Using SFTP, I copy down the updated file system to my local project
- Save another database backup
- Commit all the changes to source control