DawgTalkers.net Forums DawgTalk Everything Else... Server Outage and More

For starters, sorry about today's outage, if anyone noticed (lol, right?).

It turns out that the data center where our server lives was doing some re-organizing. Fortunately, things came back up relatively painlessly, unfortunately, however, during the process of getting in touch with the datacenter I learned that they are closing shop, and I've got roughly until next Friday to figure out what I'm doing.

So, there WILL be some outages in the coming week. Things this weekend will be unaffected, but I cannot guarantee things will be functioning for the following Gameday.

If I stay with this company, which I probably cannot do, things will get limited quickly in terms of available bandwidth, etc...
This nearly guarantees that I will be switching to a new datacenter and new IP Addresses, which means, at the least, a day or two of outages will happen at some point while things get physically relocated and DNS gets updated.

I don't, yet, have an idea of what I'm going to do or how or when as I've only had this knowledge for about 45 minutes as of this writing, but this is all happening REALLY FAST, whether I'm ready for it or not.

I will do my best to keep everyone informed and to plan outages to minimize impact as much as possible as this moves along.

Enough folks have seen this that it doesn't need to be stickied at the top of every forum any longer.


I've located a datacenter near my home that will take individual servers, however there are a couple of sticking points I have to work past with them and I'm currently waiting for them to call me back so that I can get some more information. One possible/probable solution is actually going to involve a new server, and I may have a suitable one getting donated. I'll be looking at his servers tomorrow. If one is suitable, I will be bringing it home and immediately begin building it out

I'm currently working on ensuring there are good backups of all the data on the server; copies of everything are being made to multiple locations, including being pulled back to a storage server here at my house.


I will continue to share as things progress.

Hurry up, eh?

PayPal - let us help.

Isn't the current site in Chicago? I thought so but perhaps I'm wrong.

It was moved from there in 2013 and has been in Mentor since.
Now there is a pretty good likelihood that it will live in Medina County.

I picked up a new server today. I've had to order some new drives for it and they will be here by Friday. The datacenter owner has told me there is a decent chance that I'm good to leave things as-is through the following week as he's decided to keep the building, so as long as his fiber isn't cut off on Friday, we'll be up.

As soon as the new drives arrive, I'll get the new server built up. I'll be making arrangements with the place that should be our new home tomorrow and by Wednesday or Thursday I should know what our new IP address will be and I can start planning to make the DNS changes.

I can see that there are going to be some hiccups with this, it isn't going to be seamless of super-smooth, but hopefully I can keep the amount of pain to a minimum.

I have faith in you... Call me if you need me

If there is anything I can help with from here, let me know.

I've pretty much secured a spot in the new datacenter (haven't turned over my card for billing, yet, but I've got the sales guy putting everything in motion). I'm hoping to have my new IP addresses tomorrow.

The new physical server is a SERIOUS upgrade in horsepower from our old one (24 cores @ 3.07GHz, 64GB ram). It probably doesn't really matter much for a website, but it's nice to know we can handle traffic a bit better than before. This was needed, of all things, to keep datacenter rack space leasing costs down; the new server is 1U in height, the old one was 2U, and datacenters charge by the 'U' as well as bandwidth. I'm currently doing the initial build-out of the new machine and copying files over to it as I write this.
I've purchased an upgrade license for the forum software, too, and will attempt to get things upgraded as part of the move. The upgrade should make posting from mobile devices a LOT nicer, among other things.

I do not currently have a guesstimate as to when the move will happen, nor do I know when we will go dark. After meeting with the old datacenter owner this past Sunday, he told me that he informed his landlord that he intends to keep the space, so as long as his fiber connection doesn't get turned off, I actually have a little extra time to make this work, but I'm going to ballpark guesstimate that at some point between this Saturday the 16th and Tuesday the 19th, the site will get closed so I can do one last backup and get it restored on the new server. My hope is that I'm able to have a situation where both servers are online and live at the same time so that I can try to make things relatively seamless, but I cannot guarantee that will happen; there is still the chance that his fiber gets cut off Friday, or any time after that before the end of the month at which point my only access to it will be to make the hour+ drive to go fetch it.

So, yeah, an update that isn't much of an update and still leaves a lot of ambiguity out there, lol ... but, at least now you know as much/little as I know.

I will update as I know more.

There is a forum software upgrade coming tonight. I will be closing the forums at 11PM Eastern/8pm Left Coast. We'll be moving onto the latest & greatest version of this forum software. The entire process should take less than 30 minutes, but plan on at least an hour of downtime.

First and foremost, things will look mostly the same as they do now. The forum styling will not change, but some things will be in new locations.
A couple of things will be very different... the post 'Like' functionality is probably the most noticeable. I don't like it, and I will try to get it corrected later, but it is definitely on the back burner. Another thing that is likely to happen (not tonight, but in the server move to come) is that some special characters on existing posts may get messed up... ex: an apostrophe may get turned into a series of weird looking characters. I know why and can explain the technical details (SQL character sets when we move to a new version of MySQL), but I haven't yet found a way of preventing or correcting for it. Any new posts going forward will be unaffected; it will only affect data already in the system when the transition happens.


More later.

I will panic when I want to panic, thank you very much.

At least an hour of downtime? At least? At least???????

What do we pay all that money each week for????

Only because you refuse to carry a towel.


p.s. I know that Thursday Night Football will still be going when the upgrade happens. That's life, lol... I wanna sleep, too.

I've already done two test upgrades, and even the "slow" one took at most 20 minutes, so hopefully the real upgrade is just as fast and the site is back up before the end of the game.

I'll post to Facespace before things go offline and when it is back up.

I do think we'll survive, whenever you want to do the upgrade. If you wanted to go at 9 or 10, I am sure nom one would complain.

oh, totally, but I do try to be accommodating. 11pm is late enough that most of the traffic has slowed and the impact will be minimized.

Upgrade is done. I'll start a thread in Fan Feedback for any bug/error reports; post anything that needs attention in there.

Post Bugs/Issues here:
http://www.dawgtalkers.net/ubbthreads.php/topics/1893065/board-upgrade-bug-reports#Post1893065

There will be one more brief period of downtime later today. I'll be upgrading the database server.

This will be happening within the hour, maybe a bit sooner. Board will close for the upgrade.

well, it didn't happen as soon as I expected, and it didn't go as simply as expected, but this step is done.
I *think* that should do it for us until the actual move.

I am slated to take the new server into the new datacenter on Monday. Since we are still up now, I am hopeful that the old datacenter will keep kicking until then, at least.... based on what the owner told me, it bodes well.

Thanks for all of you guys hard work

The new server is getting racked in the new datacenter around 3pm EDT/Noon Pacific today (in about two hours).
I just got the new IP addresses 30 minutes ago.

There will be website downtime this evening, but once I open it back up, that should be it for the most part. The goal is to shut down the website, back up the database on the old server, restore the backup on the new server, then point the web stuff on the old server to use the database on the new server. If I can succeed in getting that configured, then the DNS changeover will be nearly invisible as you'll simply go from loading a page on one server to loading it from another, but they'll both be loading the same data. I don't have an exact time as to when to expect the site to get closed, but probably before 5pm EDT/1PM Pacific/9PM GMT is my guess, and hopefully I'll have it back open by 8pm EDT/5PM Pacific/Midnight GMT.

Thanks for the updates and the hard work, Purp & Co. May need to double your overtime bonus! LOL

Hope it goes smoothly.

heh, thanks


The expected downtime won't be happening until after midnight EDT, now. The server is racked, but they won't be provisioning the circuit to give us our bandwidth until 10pm EDT, so it will be 11pm at the earliest, but most likely midnight, before I can get to it.

the good part of this is that most everything will happen while everyone is sleeping.

Still waiting for the datacenter to activate our circuit so that the new server is accessible. It was supposed to happen around 10pm last night, yet I'm still waiting. I'm just hoping that this isn't a red flag I'll later wish I'd paid greater attention to.... the datacenter looks nice, and the guys seem plenty skilled & knowledgeable, they're just unreliable so far when it comes to getting things done on a timetable.
Side note: we are apparently now in the same datacenter as Lebron's website and a couple of local high-end CGI Rendering shops.

FINALLY!!

Server is up and accessible, IPs are working. I have two separate test sites that are working, so we're all set to start the move.

Site will be getting closed at 2PM EDT and I'll begin the process of backing up the data and doing a final move to the new server.

With this last bit of maintenance, the database was migrated to the new server and the web pages on the old server and the new server are now running off the new database server. DNS has been updated to point to the new server.

So, now when your DNS finally gets the new info, it will be a fairly seamless transition. Until your DNS updates, however, pages may be loading a bit slower than normal. This is expected since you would still be getting pages from the old server, but it has to get all of its data from the new server... it just adds time to the loading of each page. Once your DNS updates, things should be plenty fast.


That's it, and thanks for flying with DawgTalkers Airlines!

Thanks for all you do!

Happy to

For the record, I will be turning off the web server on the old server today at noon. Enough time will have passed for like 95% of everyone's DNS to have updated, but if anyone has issues starting around noon today, this is why and you should just close the browser, re-open it, then periodically try to reload the page. Eventually your machine will get the updated DNS and it will work for you. I'd like to be able to leave the old server up longer, but I need to get it out of the old datacenter and I can see by the amount of traffic between the old and new server that most everyone is already up-to-date. The traffic levels are WAY lower between the two servers than it was just 12 hours ago.