Forum Outage Postmortem

Announcements about forum and website maintenance

Moderator: Moderators

Locked
Message
Author
User avatar
Turaiel
Administrator
Posts: 347
Joined: Wed Nov 10, 2010 3:32 am
Location: Ann Arbor, MI, USA
Fav. Twokinds Character: Natani
Contact:

Forum Outage Postmortem

#1 Post by Turaiel »

Hey all,

Since the forums were down for nearly a week, I thought it would be appropriate to fill you in on what happened, why, and what we did about it. I'll keep this short and sweet.

Before I get into that, I do recommend that you change your forum password as we can't be sure if any database breach occurred (though it is unlikely in this case). If you use the same password on other websites, you should change those as well. Thanks for reminding me to add that, amenon.

What Happened
  • At 10:43 AM EST on March 6, aj informed me that the CPU and network usage spiked hard on the server operating the forum and secondary Twokinds domains, and that the server had dropped offline afterward. At 8:17 PM, Tom messaged me indicating that the server was having issues. I didn't see either message until 8:19 PM.
  • Shortly after, we determined that 1&1, our server host for these services, had taken the server offline. 1&1 determined that the server was infected with malware which caused it to become a DoS/spam attack node. This explained the high CPU usage and network activity that aj pointed out. At this point, 1&1 had suspended our access to the system until they received a confirmation from us.
Why
The issue was caused by an attack which exploited unknown software on the server to plant a backdoor called "Mumblehard." This software is designed to act as a spam bot. More info on the exploit is available here. The infection seems to have appeared on the server on March 3.

What We Did
While the attack vector of the infection is still unknown, we took a number of precautions in an attempt to prevent the issue from occurring again.
  • On March 7, we regained access to the old server and spun up a new server with fully updated software.
  • We verified that the contents of the websites, including the forum, had not been tampered with, then restored them to the new server.
  • We installed new security measures and removed all unnecessary access vectors to the server.
  • We prevented the web server from accessing necessary files required to enable this particular exploit.
  • On March 9, the websites were re-enabled for public access.
  • We set up automated backups with staggered replication to a separate disk.
Overall, I am confident that we've prevented this particular issue from occurring in the future. In the process, we were able to set up a much faster, more secure, and cheaper server for our community to live on. We appreciate your patience during the outage!
Website Administrator

User avatar
Vintage
Certified Fool
Posts: 1213
Joined: Mon May 26, 2014 3:32 pm
Location: Planet Zambodia
Fav. Twokinds Character: Natani

Re: Forum Outage Postmortem

#2 Post by Vintage »

Jeez, that sounds kinda serious

Glad everything is back up, though! Thanks for working on it
Image Image
*pssst* Want'a see what happens when I attempt art? (Avatar made by WoofSenpai & NowandLater)

User avatar
Hayate
The Hidden User
Posts: 1447
Joined: Mon Feb 22, 2016 4:06 am

Re: Forum Outage Postmortem

#3 Post by Hayate »

I was wondering what happened, but I knew you'd tell us eventually. That's some scary stuff, I'm glad you were able to fix it all. Hopefully nobody tries to take advantage of us like that again. Thanks for you hard work getting everything running again! I missed the forum a lot in the days it was down...
~Hayate~
--Moderator/Global Moderator
--

--Forum Rules--
--Moderators --
--Administrators
--

User avatar
amenon
Grand Templar
Posts: 1693
Joined: Thu May 15, 2014 4:11 pm

Re: Forum Outage Postmortem

#4 Post by amenon »

Considering the nature of the issue, I would recommend people change their passwords. Both on the forums, and if you reuse passwords, anywhere else you were using the same one.

(Also, a friendly reminder that it's not a good idea to reuse passwords, and certainly not for any accounts you particularly care about, since getting compromised in one place can then easily lead to getting compromised in other places.)
]]> Twokinds search (search the comic based on art or text!)
     
My most recent Twokinds smutfics, newest to oldest [NSFW]:

User avatar
judah4
The Cookie Dragon
Posts: 1979
Joined: Sat Jun 04, 2011 7:09 am
Location: Sunny California
Fav. Twokinds Character: Nora
Contact:

Re: Forum Outage Postmortem

#5 Post by judah4 »

Thanks for the update on what happened. Wow.

Edit: Is it just me or are the Video boxes broken now?

User avatar
Turaiel
Administrator
Posts: 347
Joined: Wed Nov 10, 2010 3:32 am
Location: Ann Arbor, MI, USA
Fav. Twokinds Character: Natani
Contact:

Re: Forum Outage Postmortem

#6 Post by Turaiel »

Can you PM me links to where things are broken?
Website Administrator

User avatar
Turaiel
Administrator
Posts: 347
Joined: Wed Nov 10, 2010 3:32 am
Location: Ann Arbor, MI, USA
Fav. Twokinds Character: Natani
Contact:

Re: Forum Outage Postmortem

#7 Post by Turaiel »

The issue was that the site was missing a necessary Javascript for the forum's media BBcode. I restored the file from backup and everything appears to be working again.
ThunderVolt wrote:I was poking around in my chrome://flags to fix a lag issue with one of my favorite Flash games, and I inadvertently destroyed the game, but it also fixed the image and video problem. I suppose the forum outage interfered with some of my active flags, but I disabled them. Computer runs not as I like it, but my problem is fixed. No need to worry!
Nope, nothing you did affected your forum experience, or at least the items you mentioned.
Website Administrator

User avatar
Jonesy
Templar Master
Posts: 428
Joined: Tue Jun 03, 2014 8:33 am
Location: Australia
Fav. Twokinds Character: Natani

Re: Forum Outage Postmortem

#8 Post by Jonesy »

Totally misread 'Outage' as 'Outrage', which had me rather confused. Still, good to know things are back to normal.

Locked