Since the forums were down for nearly a week, I thought it would be appropriate to fill you in on what happened, why, and what we did about it. I'll keep this short and sweet.Before I get into that, I do recommend that you change your forum password as we can't be sure if any database breach occurred (though it is unlikely in this case). If you use the same password on other websites, you should change those as well.
Thanks for reminding me to add that, amenon.What Happened
- At 10:43 AM EST on March 6, aj informed me that the CPU and network usage spiked hard on the server operating the forum and secondary Twokinds domains, and that the server had dropped offline afterward. At 8:17 PM, Tom messaged me indicating that the server was having issues. I didn't see either message until 8:19 PM.
- Shortly after, we determined that 1&1, our server host for these services, had taken the server offline. 1&1 determined that the server was infected with malware which caused it to become a DoS/spam attack node. This explained the high CPU usage and network activity that aj pointed out. At this point, 1&1 had suspended our access to the system until they received a confirmation from us.
The issue was caused by an attack which exploited unknown software on the server to plant a backdoor called "Mumblehard." This software is designed to act as a spam bot. More info on the exploit is available here
. The infection seems to have appeared on the server on March 3.What We Did
While the attack vector of the infection is still unknown, we took a number of precautions in an attempt to prevent the issue from occurring again.
- On March 7, we regained access to the old server and spun up a new server with fully updated software.
- We verified that the contents of the websites, including the forum, had not been tampered with, then restored them to the new server.
- We installed new security measures and removed all unnecessary access vectors to the server.
- We prevented the web server from accessing necessary files required to enable this particular exploit.
- On March 9, the websites were re-enabled for public access.
- We set up automated backups with staggered replication to a separate disk.
Overall, I am confident that we've prevented this particular issue from occurring in the future. In the process, we were able to set up a much faster, more secure, and cheaper server for our community to live on. We appreciate your patience during the outage!