At around 13:30UTC today I received a mail for a customer saying that he could not access this site.
Naturally I got on the case straight away. My investigations found that his site was actually accessible, but only after waiting an unacceptably long period of time. And occasionally the page never appeared at all. On studying the server logs, I found there was quite a bit of activity. Most notably we had three search engine “robots”simultaneously crawling the web server. And it seems there one a lot of Windows 8 laptop users in mainland China interested in one of our image sites. Strange, I thought…
Whilst one might expect the server to slow down a bit with all this traffic, if would not explain such slow speed. Added to which examining all the running processes indicated that e MySQL server was using most of the server’s processing power, along with a number of power hungry Apache2 threads. I restarted both APache2 and MySQL. This caused a temporary speed up. But then things went back to a snail’s pace.
So I was forced to reboot the entire server at 14:25UTC. It remained offline for around seven minutes. That resolved the speed issue for about ten minutes. So I studied the logs again, particularly all those Chinese Windows 8 laptops and noticed they were all looking at the same few files.
I came to the conclusion that we were under a DDoS attack. Without boring readers with all the gory details I attempted to harden our server against such attacks, using a method similar to that detailed here. Unfortunately that didn’t work. All it did was ban the Google bot from crawling the site – which was a somewhat less than desirable outcome.
So I tried a more dramatic approach – using a web cache application called “Varnish“:-
Sadly that didn’t fix the problem either.
At the moment only one site bearing the brunt of attack, and thus hogging computing resources needed by other sites, and the DEOSS mail server. So I have temporarily taken that site off line while my investigations continue. The rest of the web server now appears to be running reasonably satisfactorily – though this is only a very temporary fix.
I will be closely monitoring the situation, while I seek a more permanent and adaptive solution.