Twokinds Forums

A place to sit around and talk about the comic.
It is currently Mon Mar 27, 2017 12:35 pm

All times are UTC




Post new topic  Reply to topic  [ 10 posts ] 
Author Message
 Post subject: Forum Outage Postmortem
PostPosted: Sat Mar 11, 2017 7:15 pm 
Offline
Site Admin
User avatar

Joined: Wed Nov 10, 2010 3:32 am
Posts: 85
Location: Michigan, USA
Hey all,

Since the forums were down for nearly a week, I thought it would be appropriate to fill you in on what happened, why, and what we did about it. I'll keep this short and sweet.

Before I get into that, I do recommend that you change your forum password as we can't be sure if any database breach occurred (though it is unlikely in this case). If you use the same password on other websites, you should change those as well. Thanks for reminding me to add that, amenon.

What Happened
  • At 10:43 AM EST on March 6, aj informed me that the CPU and network usage spiked hard on the server operating the forum and secondary Twokinds domains, and that the server had dropped offline afterward. At 8:17 PM, Tom messaged me indicating that the server was having issues. I didn't see either message until 8:19 PM.
  • Shortly after, we determined that 1&1, our server host for these services, had taken the server offline. 1&1 determined that the server was infected with malware which caused it to become a DoS/spam attack node. This explained the high CPU usage and network activity that aj pointed out. At this point, 1&1 had suspended our access to the system until they received a confirmation from us.

Why
The issue was caused by an attack which exploited unknown software on the server to plant a backdoor called "Mumblehard." This software is designed to act as a spam bot. More info on the exploit is available here. The infection seems to have appeared on the server on March 3.

What We Did
While the attack vector of the infection is still unknown, we took a number of precautions in an attempt to prevent the issue from occurring again.
  • On March 7, we regained access to the old server and spun up a new server with fully updated software.
  • We verified that the contents of the websites, including the forum, had not been tampered with, then restored them to the new server.
  • We installed new security measures and removed all unnecessary access vectors to the server.
  • We prevented the web server from accessing necessary files required to enable this particular exploit.
  • On March 9, the websites were re-enabled for public access.
  • We set up automated backups with staggered replication to a separate disk.

Overall, I am confident that we've prevented this particular issue from occurring in the future. In the process, we were able to set up a much faster, more secure, and cheaper server for our community to live on. We appreciate your patience during the outage!

_________________
Website and Facebook Page Administrator
Links // TK on Facebook // TK extras archive // deviantART


Top
   
PostPosted: Sat Mar 11, 2017 7:42 pm 
Offline
Certified Fool
User avatar

Joined: Mon May 26, 2014 3:32 pm
Posts: 1168
Location: Planet Zambodia
Jeez, that sounds kinda serious

Glad everything is back up, though! Thanks for working on it

_________________
Image Image
*pssst* Want'a see what happens when I attempt art? (Avatar made by Chibi-Marrow)


Top
   
PostPosted: Sat Mar 11, 2017 7:45 pm 
Offline
The Hidden User
User avatar

Joined: Mon Feb 22, 2016 4:06 am
Posts: 1053
Location: Canada
I was wondering what happened, but I knew you'd tell us eventually. That's some scary stuff, I'm glad you were able to fix it all. Hopefully nobody tries to take advantage of us like that again. Thanks for you hard work getting everything running again! I missed the forum a lot in the days it was down...

_________________
This post fills you with Determination :squirrel:
Image
Image


Top
   
PostPosted: Sat Mar 11, 2017 8:17 pm 
Offline
Grand Templar
User avatar

Joined: Thu May 15, 2014 4:11 pm
Posts: 1418
Considering the nature of the issue, I would recommend people change their passwords. Both on the forums, and if you reuse passwords, anywhere else you were using the same one.

(Also, a friendly reminder that it's not a good idea to reuse passwords, and certainly not for any accounts you particularly care about, since getting compromised in one place can then easily lead to getting compromised in other places.)

_________________


Top
   
PostPosted: Sat Mar 11, 2017 9:10 pm 
Offline
The Cookie Dragon
User avatar

Joined: Sat Jun 04, 2011 7:09 am
Posts: 1949
Location: Sunny California
Thanks for the update on what happened. Wow.

Edit: Is it just me or are the Video boxes broken now?

_________________
Image For the laughs
Image Twokinds Online *version 0.32ish in progress


Top
   
PostPosted: Wed Mar 15, 2017 4:09 am 
Offline
Templar GrandMaster
User avatar

Joined: Fri Jul 29, 2016 2:15 pm
Posts: 790
Location: Between the molecules.
judah4 wrote:
Thanks for the update on what happened. Wow.

Edit: Is it just me or are the Video boxes broken now?
You're not unique in that respect. Some, not all of my vids have destroyed themselves, as well as a few images.

_________________
A Kiwi's Journey! (Updates everyday)
Confused on what a Bio-Bot is?
It's the cast of A Kiwi's Journey!
A Kiwi's Journey: The Card Game!
I take custom writing requests!
Image


Top
   
PostPosted: Wed Mar 15, 2017 4:14 am 
Offline
Site Admin
User avatar

Joined: Wed Nov 10, 2010 3:32 am
Posts: 85
Location: Michigan, USA
Can you PM me links to where things are broken?

_________________
Website and Facebook Page Administrator
Links // TK on Facebook // TK extras archive // deviantART


Top
   
PostPosted: Thu Mar 16, 2017 1:40 am 
Offline
Templar GrandMaster
User avatar

Joined: Fri Jul 29, 2016 2:15 pm
Posts: 790
Location: Between the molecules.
I was poking around in my chrome://flags to fix a lag issue with one of my favorite Flash games, and I inadvertently destroyed the game, but it also fixed the image and video problem. I suppose the forum outage interfered with some of my active flags, but I disabled them. Computer runs not as I like it, but my problem is fixed. No need to worry!

_________________
A Kiwi's Journey! (Updates everyday)
Confused on what a Bio-Bot is?
It's the cast of A Kiwi's Journey!
A Kiwi's Journey: The Card Game!
I take custom writing requests!
Image


Top
   
PostPosted: Thu Mar 16, 2017 2:15 am 
Offline
Site Admin
User avatar

Joined: Wed Nov 10, 2010 3:32 am
Posts: 85
Location: Michigan, USA
The issue was that the site was missing a necessary Javascript for the forum's media BBcode. I restored the file from backup and everything appears to be working again.

ThunderVolt wrote:
I was poking around in my chrome://flags to fix a lag issue with one of my favorite Flash games, and I inadvertently destroyed the game, but it also fixed the image and video problem. I suppose the forum outage interfered with some of my active flags, but I disabled them. Computer runs not as I like it, but my problem is fixed. No need to worry!

Nope, nothing you did affected your forum experience, or at least the items you mentioned.

_________________
Website and Facebook Page Administrator
Links // TK on Facebook // TK extras archive // deviantART


Top
   
PostPosted: Thu Mar 16, 2017 2:22 am 
Offline
Templar GrandMaster
User avatar

Joined: Fri Jul 29, 2016 2:15 pm
Posts: 790
Location: Between the molecules.
Turaiel wrote:
The issue was that the site was missing a necessary Javascript for the forum's media BBcode. I restored the file from backup and everything appears to be working again.

ThunderVolt wrote:
I was poking around in my chrome://flags to fix a lag issue with one of my favorite Flash games, and I inadvertently destroyed the game, but it also fixed the image and video problem. I suppose the forum outage interfered with some of my active flags, but I disabled them. Computer runs not as I like it, but my problem is fixed. No need to worry!

Nope, nothing you did affected your forum experience, or at least the items you mentioned.
Good to know everything's fixed again.

On the other hand, that means I have to go back and fix my flags again...

_________________
A Kiwi's Journey! (Updates everyday)
Confused on what a Bio-Bot is?
It's the cast of A Kiwi's Journey!
A Kiwi's Journey: The Card Game!
I take custom writing requests!
Image


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 10 posts ] 

All times are UTC


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Limited