Server Outage and More

[PrplPplEater]

For starters, sorry about today's outage, if anyone noticed (lol, right?).

It turns out that the data center where our server lives was doing some re-organizing. Fortunately, things came back up relatively painlessly, unfortunately, however, during the process of getting in touch with the datacenter I learned that they are closing shop, and I've got roughly until next Friday to figure out what I'm doing.

So, there WILL be some outages in the coming week. Things this weekend will be unaffected, but I cannot guarantee things will be functioning for the following weekend. I will do my best to keep downtime to a minimum.

If I stay with this company, which I probably cannot do, things will get limited quickly in terms of available bandwidth, etc...
This nearly guarantees that I will be switching to a new datacenter and new IP Addresses, which means, at the least, a day or two of outages will happen at some point while things get physically relocated and DNS gets updated.

I don't, yet, have an idea of what I'm going to do or how or when as I've only had this knowledge for about an hour as of this writing, but this is all happening REALLY FAST, whether I'm ready for it or not.

I will do my best to keep everyone informed and to plan outages to minimize impact as much as possible as this moves along.

[Seanette]

Will updates be posted on the Facebook group?

[PrplPplEater]

Will updates be posted on the Facebook group?

I will post updates and plans here prior to any downtimes, along with expected durations. Unless it is absolutely unavoidable, I will not take things offline without communicating it.

[EricKei]

Seanette -- I'll repost any new info over on the FB group.

[Nunavut Pants]

Good Gord! They didn't bother to tell their customers more than a week ahead of time??? That's some MiM right there!!!

[MadMike]

Will you also be updating the board software? I believe you had mentioned that awhile back.

[PrplPplEater]

Will you also be updating the board software? I believe you had mentioned that awhile back.

I did mention that, and I still fully intend to, but life got derailed and all "extra" things got put on indefinite hold when I decided to sell my house and move last year (which happened this past February). All of 2021, thus far, has entirely been about getting settled into the new place and getting it livable. No apologies there, that stuff absolutely comes first.

I'd LOVE to get the site upgraded and be a slightly better steward than I've been, and since you brought it up, I'm going to go look at it, but no promises... it's a bit of a task right now just to get all of this happening in the next week, as-is. As it stands, the move is likely going to involve moving to new hardware, which means building out a new server this week before the move and I don't yet know what, if any, cost outlays I will have to get this all done.


As for them not telling customers.... apparently, I'm the only one they "forgot" to tell. I just spoke with all of them a month ago and nobody said a peep, so I have to assume they just don't give a damn if I'm there or not. If not for this unintended outage, I might not have found out until they shut the place down next week. To say that I'm less than pleased with them would be an understatement, and yes, if you know me, you know that I am DEFINITELY tempering my language here and there is a dramatic and noticeable underutilization of the letter F right now.


I have a lead on a new datacenter and it is near my home which makes things WAY more convenient than it has been (15 minute vs 80 minute drive), and I will find out tomorrow if some potential new hardware that has been offered to me will be suitable (possibly a server upgrade happening). I have a call in with the prospective datacenter and I'm just waiting for them to get back to me, but not learning about any of this until the weekend had effectively begun has kinda left me in limbo as far as getting humans to talk to me goes.


More soon.

[Kit-Ginevra]

Can we give a to PrplPplEater for all the week he(she?it?)does tucked ahead behind the scenes in making sure we have a site that 99% of the site is cheerfully fully functioning without any problems or issues,and when we do have a stray blip/error code/annoying ad to be nuked,dealing with it quickly and keeping us informed what's going on?

[PrplPplEater]

Will you also be updating the board software? I believe you had mentioned that awhile back.

upgrade license purchased for vBulletin Connect 5 + Mobile Bundle.

I'll *try* to get this done as part of the move, but I don't know how the timing will work.
Getting the mobile apps ready will take a bit. Android will be easy, but iOS may take a bit longer because Apple has that annoying way of requiring you to pay them for the privilege of having an Apple Developer ID.



and, thanks Kit (and I'm a He)

[EricKei]

Hey, no rush. We all agree that real life comes first - no questions asked. We appreciate you helping us all out and keeping this bird in the air. So just keep flyin'!

[Ironclad Alibi]

Not mention that we are all your customers, in a way, and we know how not to be sucky.

[PrplPplEater]

Update:

I've pretty much secured a spot in a new datacenter (haven't turned over my card for billing, yet, but I've got the sales guy putting everything in motion). I'm hoping to have my new IP addresses tomorrow.

I am in possession of a new physical server as well that is a SERIOUS upgrade in horsepower from our old one (24 cores @ 3.07GHz, 64GB ram). It probably doesn't really matter much for a website, but it's nice to know we can handle traffic a bit better than before. This was needed, of all things, to keep datacenter rack space leasing costs down; the new server is 1U in height, the old one was 2U, and datacenters charge by the 'U' as well as bandwidth. I'm currently doing the initial build-out of the new machine and copying files over to it as I write this. My intention is to get everything running on there as-is and then attempt the upgrade against a test site. One way or another, my hope is to have the site upgraded to v5 within a week of the move if I can't get it done right away (it should go smoothly ...that's what Murphy told me).

I do not currently have a guesstimate as to when the move will happen, nor do I know when we will go dark. After meeting with the old datacenter owner this past Sunday, he told me that he informed his landlord that he intends to keep the space, so as long as his fiber connection doesn't get turned off, I actually have a little extra time to make this work, but I'm going to ballpark guesstimate that at some point between this Saturday the 16th and Tuesday the 19th, the site will get closed so I can do one last backup and get it restored on the new server. My hope is that I'm able to have a situation where both servers are online and live at the same time so that I can try to make things relatively seamless, but I cannot guarantee that will happen; there is still the chance that his fiber gets cut off Friday, or any time after that before the end of the month at which point my only access to it will be to make the hour+ drive to go fetch it.

So, yeah, an update that isn't much of an update and still leaves a lot of ambiguity out there, lol ... but, at least now you know as much/little as I know.

I will update as I know more.

[PrplPplEater]

I think I'm going to do the site upgrade tonight. It's multiple steps as I cannot just upgrade directly to version 5.0, and I have to upgrade versions of MySQL & PHP along the way (and, yay vBulletin, you have no overlap in PHP so the jump from v4 to v5 is a leap of faith), so it is actually going to be a lengthy process.

I am currently in the process of doing a test upgrade and it is going generally smooth, which is nice, but I am not testing the board plugins. I am just assuming that the existing one will not work (ForumRunner, etc..) and will also require upgrading. I have not pursued that, yet, but I will chase that down after the board is upgraded. The most important thing is to get this on the newest version so that when it moves to the new server it can be running on the latest versions of PHP & MySQL, and ideally, the process of moving is then just a matter of restoring the last backup and updating IP addresses.



So, long story short: Site upgrade tonight at 5PM Eastern Daylight time. Please allow several hours of downtime. That is 9pm GMT, 2PM on the Left Coast.

Rap told me once how/where to put announcements for an upgrade, I'll look around for that.

[PrplPplEater]

It looks like this will end up being much more than just 'a couple of hours', and will be at least an overnight process and more than likely will go well into tomorrow afternoon.

The initial upgrade, which is the first increment and takes us to version 4.2.5 apparently has a TON of post-upgrade processing that has to run in addition to there being like 89 template files that have to be reviewed/corrected, among other things. This post-upgrade processing alone is going to take several hours (it looks like it processes 50 posts every couple of seconds.... and we have a million-some posts).

So, just a heads up.

[PrplPplEater]

Board closing in 10 minutes.

Fingers crossed, this all goes smoothly (and, yes, there will be backups of the backups before I begin).