Forum Outage Postmortem

And now, a word from our moderators

Moderators: Yash, Tom

Post Reply
Message
Author
User avatar
Site Admin
Posts: 104
Joined: Wed Nov 10, 2010 3:32 am
Location: Michigan, USA
Contact:

Forum Outage Postmortem

#1 Post by Turaiel » Sat Mar 11, 2017 7:15 pm

Hey all,

Since the forums were down for nearly a week, I thought it would be appropriate to fill you in on what happened, why, and what we did about it. I'll keep this short and sweet.

Before I get into that, I do recommend that you change your forum password as we can't be sure if any database breach occurred (though it is unlikely in this case). If you use the same password on other websites, you should change those as well. Thanks for reminding me to add that, amenon.

What Happened
  • At 10:43 AM EST on March 6, aj informed me that the CPU and network usage spiked hard on the server operating the forum and secondary Twokinds domains, and that the server had dropped offline afterward. At 8:17 PM, Tom messaged me indicating that the server was having issues. I didn't see either message until 8:19 PM.
  • Shortly after, we determined that 1&1, our server host for these services, had taken the server offline. 1&1 determined that the server was infected with malware which caused it to become a DoS/spam attack node. This explained the high CPU usage and network activity that aj pointed out. At this point, 1&1 had suspended our access to the system until they received a confirmation from us.
Why
The issue was caused by an attack which exploited unknown software on the server to plant a backdoor called "Mumblehard." This software is designed to act as a spam bot. More info on the exploit is available here. The infection seems to have appeared on the server on March 3.

What We Did
While the attack vector of the infection is still unknown, we took a number of precautions in an attempt to prevent the issue from occurring again.
  • On March 7, we regained access to the old server and spun up a new server with fully updated software.
  • We verified that the contents of the websites, including the forum, had not been tampered with, then restored them to the new server.
  • We installed new security measures and removed all unnecessary access vectors to the server.
  • We prevented the web server from accessing necessary files required to enable this particular exploit.
  • On March 9, the websites were re-enabled for public access.
  • We set up automated backups with staggered replication to a separate disk.
Overall, I am confident that we've prevented this particular issue from occurring in the future. In the process, we were able to set up a much faster, more secure, and cheaper server for our community to live on. We appreciate your patience during the outage!
Website and Facebook Page Administrator
Links // TK on Facebook // TK extras archive // deviantART

User avatar
Certified Fool
Posts: 1186
Joined: Mon May 26, 2014 3:32 pm
Location: Planet Zambodia

Re: Forum Outage Postmortem

#2 Post by Vintage » Sat Mar 11, 2017 7:42 pm

Jeez, that sounds kinda serious

Glad everything is back up, though! Thanks for working on it
Image Image
*pssst* Want'a see what happens when I attempt art? (Avatar made by WoofSenpai)

User avatar
The Hidden User
Posts: 1156
Joined: Mon Feb 22, 2016 4:06 am
Location: Canada

Re: Forum Outage Postmortem

#3 Post by Hayate » Sat Mar 11, 2017 7:45 pm

I was wondering what happened, but I knew you'd tell us eventually. That's some scary stuff, I'm glad you were able to fix it all. Hopefully nobody tries to take advantage of us like that again. Thanks for you hard work getting everything running again! I missed the forum a lot in the days it was down...
This post fills you with Determination :squirrel:
Image
Image

User avatar
Grand Templar
Posts: 1483
Joined: Thu May 15, 2014 4:11 pm

Re: Forum Outage Postmortem

#4 Post by amenon » Sat Mar 11, 2017 8:17 pm

Considering the nature of the issue, I would recommend people change their passwords. Both on the forums, and if you reuse passwords, anywhere else you were using the same one.

(Also, a friendly reminder that it's not a good idea to reuse passwords, and certainly not for any accounts you particularly care about, since getting compromised in one place can then easily lead to getting compromised in other places.)

User avatar
The Cookie Dragon
Posts: 1951
Joined: Sat Jun 04, 2011 7:09 am
Location: Sunny California
Contact:

Re: Forum Outage Postmortem

#5 Post by judah4 » Sat Mar 11, 2017 9:10 pm

Thanks for the update on what happened. Wow.

Edit: Is it just me or are the Video boxes broken now?

User avatar
Templar GrandMaster
Posts: 998
Joined: Fri Jul 29, 2016 2:15 pm
Location: The world between the molecules.

Re: Forum Outage Postmortem

#6 Post by ThunderVolt » Wed Mar 15, 2017 4:09 am

judah4 wrote:Thanks for the update on what happened. Wow.

Edit: Is it just me or are the Video boxes broken now?
You're not unique in that respect. Some, not all of my vids have destroyed themselves, as well as a few images.

User avatar
Site Admin
Posts: 104
Joined: Wed Nov 10, 2010 3:32 am
Location: Michigan, USA
Contact:

Re: Forum Outage Postmortem

#7 Post by Turaiel » Wed Mar 15, 2017 4:14 am

Can you PM me links to where things are broken?
Website and Facebook Page Administrator
Links // TK on Facebook // TK extras archive // deviantART

User avatar
Templar GrandMaster
Posts: 998
Joined: Fri Jul 29, 2016 2:15 pm
Location: The world between the molecules.

Re: Forum Outage Postmortem

#8 Post by ThunderVolt » Thu Mar 16, 2017 1:40 am

I was poking around in my chrome://flags to fix a lag issue with one of my favorite Flash games, and I inadvertently destroyed the game, but it also fixed the image and video problem. I suppose the forum outage interfered with some of my active flags, but I disabled them. Computer runs not as I like it, but my problem is fixed. No need to worry!

User avatar
Site Admin
Posts: 104
Joined: Wed Nov 10, 2010 3:32 am
Location: Michigan, USA
Contact:

Re: Forum Outage Postmortem

#9 Post by Turaiel » Thu Mar 16, 2017 2:15 am

The issue was that the site was missing a necessary Javascript for the forum's media BBcode. I restored the file from backup and everything appears to be working again.
ThunderVolt wrote:I was poking around in my chrome://flags to fix a lag issue with one of my favorite Flash games, and I inadvertently destroyed the game, but it also fixed the image and video problem. I suppose the forum outage interfered with some of my active flags, but I disabled them. Computer runs not as I like it, but my problem is fixed. No need to worry!
Nope, nothing you did affected your forum experience, or at least the items you mentioned.
Website and Facebook Page Administrator
Links // TK on Facebook // TK extras archive // deviantART

User avatar
Templar GrandMaster
Posts: 998
Joined: Fri Jul 29, 2016 2:15 pm
Location: The world between the molecules.

Re: Forum Outage Postmortem

#10 Post by ThunderVolt » Thu Mar 16, 2017 2:22 am

Turaiel wrote:The issue was that the site was missing a necessary Javascript for the forum's media BBcode. I restored the file from backup and everything appears to be working again.
ThunderVolt wrote:I was poking around in my chrome://flags to fix a lag issue with one of my favorite Flash games, and I inadvertently destroyed the game, but it also fixed the image and video problem. I suppose the forum outage interfered with some of my active flags, but I disabled them. Computer runs not as I like it, but my problem is fixed. No need to worry!
Nope, nothing you did affected your forum experience, or at least the items you mentioned.
Good to know everything's fixed again.

On the other hand, that means I have to go back and fix my flags again...

User avatar
Templar
Posts: 354
Joined: Tue Jun 03, 2014 8:33 am
Location: Australia

Re: Forum Outage Postmortem

#11 Post by Jonesy » Tue Mar 28, 2017 10:17 am

Totally misread 'Outage' as 'Outrage', which had me rather confused. Still, good to know things are back to normal.
Human & Lupusad - Of Men and Wolves
A human saves a lupine girl from the streets, forming a close bond.
Latest Chapter: 32 (Posted 7 April, 2017)

The Desulti Chronicles: Stranded
A starship of kangaroos gets stranded in deep space.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest