24hourcampfire down.

[gregintenn]

I haven’t been able to access it for a couple of days now. Anybody know what’s going on?

I sure hope TGO isn’t next!

[RED333]

I was just there last night, now it appears to be down.

Got this

Error 522

Ray ID: 63b69dad5ead1c79 • 2021-04-05 23:39:28 UTC

Connection timed out

[Guest]

Yep, still down hard. Makes you wonder what was said...

[RED333]

As of this morning it is still down, now has an instant window pop up with this message.

We’ll be back soon!

Sorry for the inconvenience but we are working as hard as possible to get things back online.

[No_0ne]

[MacGyver]

They were posting 200 errors, then 503, then the 522 errors.  Their host machine is having issues - and I wouldn’t wish that on anyone.

One of the realities of a small forum - or even a big one - is that while redundancy is not a particularly hard problem. It does require a specific set of skills, and is quite expensive.

Backups are your friend.

I hope they get it squared away. 

[TGO David]

19 hours ago, gregintenn said:

I haven’t been able to access it for a couple of days now. Anybody know what’s going on?

I sure hope TGO isn’t next!

We're not immune to problems but we do a LOT of backups, all "off-site", several times a day.

[gregintenn]

I’m glad to hear you guys don’t think it is a deplatforming issue.

[btq96r]

Hate to think about the issues TGO would be subject to if our Admin team wasn't heavy on IT experience. A lot of things can creep up, and troubleshooting/resolution can be a beotch.

[gregintenn]

9 minutes ago, btq96r said:

Hate to think about the issues TGO would be subject to if our Admin team wasn't heavy on IT experience. A lot of things can creep up, and troubleshooting/resolution can be a beotch.

We’d be screwed were I heading this thing up.

[MacGyver]

As those Bartles & Jaymes guys used to say, “...and thank you for your support.” (I wonder if they’re still a thing?)

The infrastructure to do something like this is way less expensive than it used to be - but it’s still not particularly cheap when you get the traffic TGO does.

Benefactors literally keep the lights on around here.  

[gregintenn]

They're back!

Quote

On Easter Sunday, the data center where our server is located suffered:

A power surge, which caused
A power outage, which caused
Backup generators to deploy, which caused
A fire in one of the backup generators, which caused
Fire sprinklers to come on.
News from the datacenter was slim, except to tell us that they were awaiting a city inspector on Monday to approve turning the power back on.

Unfortunately, that bit of info caused us to stand by and wait for our current server to get turned back on, which we hoped would be imminently.

As it turns out, there was more water damage at the data center than originally let on, and this morning we received the message:

"Unfortunately, this server appears to be in the section with a high probability of water damage. So it will need to be physically inspected before it can be powered on.

I'm afraid that we do not have an ETA for repair at this time."

I sure wish I would have known that Sunday or Monday!

Anyway, with that news, we decided to put our contingency plans into effect immediately. The good news is that we have a safe backup. Since this morning's news, we have deployed a new dedicated server, and we are in the process of configuring it and loading the backup. That should take ~24hours. We will lose a few hours worth of data ... the period of time between the backup and the server crashing ... but at this point I have made the decision that we cannot wait any longer to be back online, especially in light of the not-so-encouraging ETA.

We thank you for your patience. The clock is now ticking on a (belated) quick return.

Rick Bin

I'll bet that was frustrating.

[TGO David]

5 hours ago, gregintenn said:

They're back!

I'll bet that was frustrating.

Wow.  The description of what happened to their server falls into what I think most folks would just call an "Act of God".  Glad they got it running again!

[gregintenn]

It definitely brings a new appreciation for what you guys provide and what is involved in providing it.

[TGO David]

Just now, gregintenn said:

It definitely brings a new appreciation for what you guys provide and what is involved in providing it.

Truthfully, what protects us (and them) in cases like this is just simple ol' planning ahead and having contingencies lined up.  I appreciate the kind word and confidence, though. 

With our backups, I can re-home TGO elsewhere pretty quick if needed.  The delay is in getting DNS servers to update, and there are some tricks that can make that give us a little extra control over that.

Without backups, we'd be sunk.  They would have been too.

 

[TGO David]

I'll add:   I test our contingency plan every single time I update the software by virtue of doing the software update itself.  I have, on more than one occasion, had the software update go poorly and rolled us back to the previous state, instantly, with a backup.

I know that TGO's plan works because I test it regularly and update it as things change and new options become available or old options phase out.  It's a good habit that working in enterprise healthcare IT has ingrained for the past 20 or so years.

 

[No_0ne]

7 hours ago, gregintenn said:

They're back!

I'll bet that was frustrating.

And exciting ...