Discussion: Server Failure on Sunday, September 17, 2023

[TGO David]

Hey gang.  Welcome back to TGO.  There is a lot to cover in this post, so I am going to try to hit things in the order of most important or impacting, to least.

 

Where is my TGO user account?

If you created an account on TGO after August 17th you will most likely need to create your account again.  We lost almost 30-days worth of data in this crash.

 

What Happened To My Benefactor Account?

A lot of my work today has been around connecting Benefactor memberships with those who purchased them.  I was successful with most of them, but there are a few of you (less than 10) that I will be reaching out to in email so that we can work together to get you squared away.  I am prioritizing this near the top of importance because you guys keep TGO afloat and I will take care of you.

 

I posted something and now it's gone!

Anything posted after August 17th is lost.  I am very sorry.  I suppose the good news is that if you posted something and regretted it, you now have a second chance. 

This part really sucks and I am absolutely not trying to down-play it.  Trust me, however upset you are, I am infinitely more upset.  I hope that you'll dive back in and recreate whatever we lost.

 

What happened?!?

This is the million dollar question.  Our server suffered a catastrophic hardware failure that could not be recovered from.  Thankfully we had backups, but I had to go back 30-days in time to find a backup that was viable.  This problem had unfortunately been brewing for a while and I did not notice it in time.

Prior to today (9/18/23) we were using a different web hosting company.  We aren't using them now because I am greatly disappointed in their ability to assist and am livid over the outcome.  The best and only way for me to respond was to move TGO to another hosting company who I believe will be much more responsive and responsible.

That said, I cannot place full blame on them.  Sadly, I missed some signs which were obvious in hindsight so some of this is absolutely my fault.  Especially related to backups.  I am now making backups of my backups and storing them in several places (online, offline, and in the cloud) so that we have more recovery options if something like this ever happens again.

 

How can we help??

Post, post, post.  Dive back in and start using TGO.  This will be a scar for a while, but like all scars it will eventually fade.  Other than that, there really isn't anything to do.

 

I just noticed [insert problem here].

If you notice anything weird with your account, especially with payments for Benefactor memberships, let me know.  I'll sort it out and get you taken care of.

 

 

That's about it for now.  Thanks for your patience while we were offline.  I suppose if there is a silver lining to all of this, it has provided a chance to do some things different and better behind the scenes.  Hopefully those differences and improvements will equate to odds heavily in our favor for the future.

 

--David

 

[FUJIMO]

Well I for one am glad to be back  

Damn crazy how you get use to something  

[Luckyforward]

I went in and tried to look around and when I couldn't get on, I thought . . .

 

[Luckyforward]

[TripleGGG]

Whew, glad that's over 

[Luckyforward]

[Grand Torino]

[Snaveba]

Thanks for all you do to keep this community (and it IS as community) up and running. 

[TripleGGG]

[res308]

I, for one, will have to seek counseling for my withdrawal symptoms. 

[TripleGGG]

11 minutes ago, res308 said:

I, for one, will have to seek counseling for my withdrawal symptoms. 

[Moped]

Sure am glad we are back up and running! Thank you David for being on top of it! One of the best run pages on the web! 

[No_0ne]

Edited September 18, 2023 by No_0ne

[DJTC45]

Glad you got it worked out, I can only imagine what a pain in the @$$ it was to get it back up. Thanks for your time and support to this site.

[krunchnik]

I thought we broke the internet

[Chucktshoes]

Thank you sir for all of your hard work getting us back up and running. 

[No_0ne]

2 minutes ago, krunchnik said:

I thought we broke the internet

[Sleep profit]

Many thanks    I appreciate the site.  and you  I thought the Feds took it due to some of our comments lol 

Edited September 18, 2023 by Sleep profit

[bigun]

Glad I didn’t break it after all and the ATF isn’t coming after us yet. 

[bubbadavis]

Thanks for all the effort David! Not too many people know how stressful catastrophic software failures are for the guys who have to fix them. I really, really, really hate it when I find out my backups have issues to boot.

I've already updated a couple of Trading Post topics and left feedback for a couple of members.

\Part of me does feel like I looked at the Men In Black memory eraser... 

[AuEagle]

Thanks for fixing it!!

[ken_mays]

Nice job sir!  As someone who has been in IT awhile, I have noticed that the number of organizations who have backups at all, let alone viable backups, is certainly a minority.

[TGO David]

5 minutes ago, ken_mays said:

Nice job sir!  As someone who has been in IT awhile, I have noticed that the number of organizations who have backups at all, let alone viable backups, is certainly a minority.

The shameful part is that we had about 30-days worth of backups that would have prevented us from losing all but maybe 8-hours worth of activity, but they were destroyed when the hardware failure occurred.

Normally those would have been replicated offsite and to the cloud, but part of the automation that I wrote wasn't running and I just didn't notice it. 

 

[bubbadavis]

I really want to hear the nitty-gritty details of the old hosting company - after you get some rest! I'm used to servers having at least RAID arrays; while not foolproof, they usually help with data loss. 

[Grayfox54]

Having spent my life fixing things, I understand that machines do break down. Stuff happens. 

Glad you got it fixed.