Site Migration Plan / Status

[Pedersen]

We finally have the pieces in place, and were going to hold off until the end of month. However, after yesterday's extended outage, we're forced to move things forward a bit.

We have four separate forums being hosted on the server. Each one will be moved during the week, one per night. I'll be posting the announcements to each of the forums today.

The schedule is as follows:


Tues, Jan 18: Testing forum
Wed, Jan 19: Devils Panties Forum
Thur, Jan 20: Fratching Forum
Fri, Jan 21: CS Forum


The moves will be happening overnight, and as late as possible. I am working a new job, and my sleep schedules are affected, so it won't be able to be any later than midnight, Eastern time, when the move occurs.

Expected downtime is about 30 minutes. During this time, the current server and the new server will display a "maintenance mode" message. You are also invited to join us on cschatter.com to get status updates as they are ready to be announced.

Once the migration is complete, DNS changes will be made, the forums reopened, and life can return to normal.

Note that cschatter.com is restricted to using CS logins. If you are using one of the other forums, and wish to join us on cschatter, please create a CS login immediately, and post a reply to this linked thread (http://www.customerssuck.com/board/showthread.php?t=73918) to have your account set up for chat. This step is important, as new users are on moderation, and moderated users (by default) are not allowed in CS Chat at all.

I apologize for the short notice. I really did want to wait, but the current server is getting worse by the day. We have to make these changes as rapidly as possible.

If you have any questions about this, please either post here or contact Pedersen. The mods might have the answers, but they might have to ask me. Better to go right to the source, I think.

[Der Cute]

I hear about server migration a lot @ owrk, and I keep seeing these little 1s and 0s marching to a new home. LOL

curious. is the reason for this migration a SQL db getting corrupted..or OS upgrade...or drive space being eaten by logs??

[Rapscallion]

Several issues, the main one being that the server hardware is no longer suitable for the amount we're asking of it. Every so often, it decides to belch.

Rapscallion

[Pedersen]

The long and the short of it is that we've outgrown this server, and even outgrown the capabilities of most single server setups. The machine only has 2G of RAM, and almost all of that memory is in use all the time. Once the machine starts hitting swap, or doing other disk intensive work, everything comes to a halt.

The new setup will see us on two separate servers, and give us better control of everything in the event of failure. For instance, part of the problem yesterday was due to having to ask webfusion to reboot. With the new setup, we're always going to be able to login. We can even remotely access the console, so one way or another, we can log in and reboot the server.

Much better. Surprisingly enough, cheaper, too.

[Rapscallion]

Much better. Surprisingly enough, cheaper, too.

Me likey everything. Me really likey this bit :D

All done and said, though, it's a far better system. We've got the new servers up and running, and they worked well under a good slapping.

Rapscallion

[Dave1982]

Also, lest anyone forget, there's the issue of the current server having been hacked, so even if the current server was adequate, it would've needed a reformat to wipe out whatever was done during the hack.

[Pedersen]

The first forum has been moved successfully. Slight glitch with the database configuration, but that is done and gone, and won't come back for the others. The next ones will be even smoother.

Tomorrow night, the Devil's Panties forum will be moved. If you want to keep tabs on things while it's moving, please read these directions (http://www.customerssuck.com/board/showthread.php?t=73918) if you're not on CS.

Once you're on CS, you can visit us at CS Chatter (http://www.cschatter.com/), and follow the links for jwchat, to be kept in the loop while the move occurs.

[Pedersen]

So, the Devil's Panties forums got moved tonight. vBulletin, though, has kind of irked me.

Tech Talk: With version 4, they enabled protection against cross-site scripting attacks. This is a good thing. However, by enabling this protection, they've also disabled an easy migration to a new server. The only way for people to access the server is to have their DNS servers update.

I've done the best that I can to mitigate the effects, but the forums will (effectively) be read only until DNS changes go all around the net. If everybody else is configured properly, this should be cleared up within the hour.

To any DP posters, I apologize for the inconvenience. I just don't have any way to fix this.

[Rapscallion]

For those interested in a more layman's version of VB4, it's a bag of wank.

Zero instructions. Had Pedersen not been online when I updated the DP forums, we'd have been screwed as to what was going wrong. Current skins won't work on it, and they released something as pretty much a beta test on their customers. 3.8 is a stable release and we're planning on sticking with it for some time.

If it's any guide, I went looking for skins for VB4, and most suppliers out there started to work on this and then refused to go further.

Rapscallion

[Pedersen]

Fratching is now on the new server. You'll all be amused to know that it took me an extra five minutes because I forgot to restore the database on the new server. Copied the files over, but forgot to actually load them.

Anyway, it's done, happy, and things are looking good.

[Pedersen]

And now CS is running on the new server.

There are still some back end details to cleanup, but outages should disappear now.

Thank you for your patience.

[Peppergirl]

Peds,

Thank you for all your hard work on this. I'm sure I speak for all of the members here when I say how much we appreciate all you do for CS and the other forums.

Everything looks great!

[JustaCashier]

Yes!

Huge thanks to Pedersen, and the Admins and Mods for this transition and upgrade, and for keeping us "in the loop" as it progressed! :worship:


Mike

[Pedersen]

I said it in chat, and I think I said it best there.


[00:00:47] <pedersen> It's strange. Everybody else thinks I did great, kept people informed, etc. I think I did only so-so, rushed too much, and didn't inform enough.


I appreciate the compliments, I do. I just feel like I should done something better. Given exact times when the different forums would be down (I couldn't, because the new job and new sleep schedule kept me from knowing how late I could stay awake), for instance.

There are still a few rough edges that I know about. Only two of them should be visible to people, though: The main page for the site is has some issues with Firefox 3.5 ( http://www.customerssuck.com/ to see), and the card list manager is currently down (something I'll be fixing Monday or Tuesday).

For everybody out there: If you have an issue, either PM me or attach a post to the Bug Reports Forum (http://www.customerssuck.com/board/forumdisplay.php?f=30). I'll resolve it ASAP.

And thank you again for the patience. Hopefully, we won't need to do this again for a very long time.

Side note: For those of you who might be wondering why we needed to move? The backup file for just the CS database (which doesn't include avatars, photos, etc) was 778M. For reference, if you were to print that out on paper, 66 lines per page, 80 characters per line, you would need 154,402 sheets of paper. Yeah, we're an active bunch :)

[JustaCashier]

I appreciate the compliments, I do. I just feel like I should done something better. Given exact times when the different forums would be down (I couldn't, because the new job and new sleep schedule kept me from knowing how late I could stay awake), for instance.


Okay, suit yourself.... :shrug:

:soapbox:


New job?!?

Who gives a rip?!? New job, Old job, B...umm anyway, what the heck do you think Raps pays you nothing but, most certainly, his undying gratitude :worship: and respect :respect: for? To go work some silly, paying job and put the maintenance and upgrading of the Boards on the back burner?!? Sheesh!!:rolleyes:

And Sleep?!?

I thought you Techy/Geeky types thrived on lack of sleep, and guzzling gallons of Red Bull ® :drink: Especially when working on an important, challenging, project!!
/:soapbox:






Okay, how's that? :lol:


Mike

[Der Cute]

We do thrive on those things.
But we normally expect migrations to go well...as I said before, little 1s and 0s marching across to the new server.

Almost always when changing code/platforms/updates/hardware there's little itty bitty annoying things that happen. Rarely is it ever a crapalanche.

Pedersen, I bet you were expecting roses and only got daisies. You still did good..short bit of down time, (and this is not a money making site!) and getting it handled. All of us members are all over the globe, so there is really no way to take it down/off without hitting someone.

Same @ my job. Part of the company is West Coast, Conneticut, East Coast, London, Shanghai...
Any reboots I do can make hiccups. So I schedule them for the slow spots, pray and it works. Shoot, the worst that happened is that I had to make a ticket for DC to manually boot a server that didn't want to come back up.

See? Small burps. No crapalanche. You did great!!!!

If you ever do a migrate/code infusion job perfectly, let me know. I think fairies bring those to us.

[Rapscallion]

See? Small burps. No crapalanche. You did great!!!!

Seconded!

Rapscallion

[wolfie]

Is it possible to make a skin for the new server that has black on white text like the old one? I took a quick look at the new server, and the text was black-on-black (Firefox 2.0.0.20 running under Win 98 SE), so I had to highlight things to see anything.

[Rapscallion]

Are wea talking about the front pages of the site?

Rapscallion

[Betweenshades]

I just feel like I should done something better.

I think that's the endless mantra of a geek. Lord knows I said this a LOT when I was setting up the work database and server.

You done good! :)

[Pedersen]

Okay, suit yourself.... :shrug:

I know it's weird, it's just how I am. I feel like I should have had everything, backend, frontend, everything, done and prepared. Instead, it's only now that I think I've finished all of it. At least, all that I have left on my list is documentation issues. Those I can work on this week.

See? Small burps. No crapalanche. You did great!!!!

If you ever do a migrate/code infusion job perfectly, let me know. I think fairies bring those to us.

I've had them. They take a lot of planning, a lot of preparatory work, and are a bitch to pull off. But they can be done. I wasn't aiming for perfect, just for marginally good enough.

I got that. But it meant leaving a few of the old server pieces not running right away, and leaving some of the known (and open) issues until after the move. For instance: the "Meet the Mods" page needs an update. Ree has those notes, just needs to know what to do. The XMas Card List manager tool was down until last night. A couple of behind the scenes tools weren't prepped. Backups weren't running properly. Lots of little pieces that I just didn't have time to get done, tested, and working in advance that should have been.

The old server was falling apart too rapidly, and I was out of time to do it slowly. That's what I mean by having done better.

Is it possible to make a skin for the new server that has black on white text like the old one? I took a quick look at the new server, and the text was black-on-black (Firefox 2.0.0.20 running under Win 98 SE), so I had to highlight things to see anything.

This problem was one of the known issues. It should now be fixed properly. Turns out that the site was using a feature that was only available in FF3.6 and up. I've implemented a very small change that worked for me when I just tried it with Firefox 3.5.

I think that's the endless mantra of a geek. Lord knows I said this a LOT when I was setting up the work database and server.

You done good! :)

It probably is. I can't count the number of things where I've said "I could do this better."

Thank you.

[Flying Grype]

Ya done good! I never noticed a thing. :P

[Mikkel]

Ya done good! I never noticed a thing. :P

Yes, the only thing I have noticed is that I don't get the "server busy" error any more.
I can't see how the move could have been done better. There will always be loose ends to take care of.

[Betweenshades]

I can't count the number of things where I've said "I could do this better."

It's how we become better. It's a good thing until you beat yourself up. ;)

[JustaCashier]

I know it's weird, it's just how I am. I feel like I should have had everything, backend, frontend, everything, done and prepared.



Yeah, I understand.

I'm not particularly OCD about most things, but just a very few. (And OCD is probably too strong of a description, perfectionist probably describes it better).

But, if I were in your shoes, I probably would have been the same way, since other people are relying upon me. If it's something just for myself, and wouldn't much of an effect on anyone else, I'd probably let little stuff slide, or just give up altogether for awhile, and go back to it later. (Most likely much later! :lol: )

I think that might be why, even after 30+ years of jobs with one on one customer contact, I still have trouble dealing with unhappy customers. Even if I realize it's not my fault, and that they are just being an EW, or if it's some corporate/management policy/decision I have no control over. Someone is unhappy, and in their eyes, (at least in my mind) I've failed. :ashamed:

Hell, I'm like that in non-work situations too! :runaway:

But yeah, just remember you're hard work is appreciated, :salute: probably dozens of times more than what has been said on this thread, as not everyone is going to comment here.

Mike

[wolfie]

The XMas Card List manager tool was down until last night.

Considering it's late January, that sounds like something that's appropriate to leave until everything else is fixed.:roll:

This problem was one of the known issues. It should now be fixed properly. Turns out that the site was using a feature that was only available in FF3.6 and up. I've implemented a very small change that worked for me when I just tried it with Firefox 3.5.

Just tried going in to the top-level page (do most of my access from the "board" page), and the "must highlight to see the thread" issue is fixed. Thanks.