Server issues

I sent the following to all the sites I host earlier this afternoon. Needless to say, it has been a stressful few days.

You have no doubt noticed by now that your website has been down since Monday (12/20) at approximately 8 pm EST. I am writing to explain what happened, why it happened, and what steps I am taking to prevent similar situations in the future.

On Monday evening, the server was hit with a “worm” described at http://news.com.com/Net+worm+using+Google+to+spread/2100-7349_3-5499725.html?tag=nl. When I first noticed this, I thought my server had been compromised by an intruder and went about working with the server provider to close any security breeches and restore the corrupted html/php files from the daily backup. (at that time, no one knew it was a worm.) That restore was complete by 11 pm, and within 30 minutes the files were defaced.

The server provider, by default, keeps one nightly backup (around 11 pm est) and does a weekly backup every Wednesday. Because of the timing of this event, the daily backup was also corrupted.

This worm rewrote the contents of index.* files. Because you are on a “shared server”, you were affected regardless of whether or not you use php

When I discovered I had been hit a second time, I opted to take the entire webserver down, upgrade php to address the issues, and only then restore from the 12/15 tape backup. I was told by the provider that they would have a fully supported upgrade of php in place by last night. Rather than doing a custom installation, I opted to wait.

I have been assured that the upgrade will be in place by later this morning’. I will then restore all files to the 12/15 backup and restart all the sites by mid afternoon.

Unfortunately, this means that I weeks worth of data has been lost. Going forward, I intend to prevent this by making a local copy of the provider’s nightly backup.