Go Back   Customers Suck! > The Heart of the Site > Unsupportable

Closed Thread
 
Thread Tools Display Modes

IT problem apparently caused by IT department
  #1  
Old 02-05-2018, 10:49 AM
SpyOne SpyOne is offline
Deluxe Lackey
 
Join Date: Jul 2007
Location: Virginia Beach, VA, USA, Sol3
Posts: 269
Default IT problem apparently caused by IT department

I work in a convenience store. We are part of a large national (actually international) chain.

A few years back we upgraded to a system where everything that gets delivered has to be scanned, and then the computer prints paperwork for both the store and the delivery driver.

Within 6 months we developed a major issue:
Twice a week we get in a large delivery, and scanning the items that must be verified before the drivers leave takes about 25 minutes. The drivers are only required to wait 35 minutes from the time they start to unload. But when you finished scanning the delivery and sent the report to be printed, it would be 12 minutes before it would print.

After some experimentation, I determined that this would only happen if the delivery arrived (and was completed) between midnight and 1:30am. The back office computer starts crunching the numbers for the day at about 11:59pm, and if someone tries to pay by credit card between then and 12:02 it can take nearly a minute to process. And many reports are unavailable between midnight and 1:30 with some sort of "processing that data, try again later" message.
I thought I'd found a bug in the software that meant you couldn't scan deliveries while the system was processing numbers.

So it falls to me to be the guy on the phone with Tech Support trying to nail down this issue.

We run down the troubleshooting checklist with the Tech Support system remotely accessing our computer, and then there was a long pause.
"Um, according to this that computer hasn't been rebooted in ... 28 months?"
"That sounds right, since we only reboot it when specifically directed to do so by you." (It even says that on the screen saver. Not just "Do Not Turn Off" but "Do Not Reboot Unless Directed To Do So By The Help Desk".)
"Um, ... okay, well I'm going to reboot it and we'll see if that helps on your next delivery."

It has worked fine ever since. And about once every six months it reboots as part of one of the periodic software updates.

Timed Backup
  #2  
Old 02-05-2018, 09:52 PM
earl colby pottinger earl colby pottinger is offline
Area Manager
 
Join Date: Feb 2007
Posts: 1,659
Default Timed Backup

As I was read your problem I wondered if a backup is taking place at that time.

Depending how backups are done on your server, individual files may be locked so any program that writes to them is frozen while the file is being backed-up. If a number of files are doing this and the program needs them it will lock-up/slow-down until the backup is done.

On the other-hand, your idea of restarting the machine may work too, memory could be very fragmented which will slow down the machine if it uses a lot of virtual memory. Also the log files may be very long and depending on how they are appended to the machine can slow down, they need to be trimmed or deleted . And worse but most silly, the hard drive could be fragmented and just running Defrag will give you a performance boost.

Last but not least - replacing an old and maybe failing hard drive with a SSD will make a real speed difference.

  #3  
Old 02-08-2018, 04:53 PM
protege's Avatar
protege protege is offline
Super Moderator
 
Join Date: Jul 2006
Posts: 6,108
Default

Quote:
Quoth earl colby pottinger View Post
Depending how backups are done on your server, individual files may be locked so any program that writes to them is frozen while the file is being backed-up. If a number of files are doing this and the program needs them it will lock-up/slow-down until the backup is done.
That's what happens with the server at my place. If you're doing anything involving the system databases, they slow down to a crawl. Most of us know to get things done before we hear what sounds like jets taking off in the server room. The fans get unbelievably loud during backups, then suddenly return to their usual only-slightly-annoying noise levels.
__________________
Aerodynamics are for people who can't build engines. --Enzo Ferrari

  #4  
Old 02-17-2018, 06:03 PM
SpyOne SpyOne is offline
Deluxe Lackey
 
Join Date: Jul 2007
Location: Virginia Beach, VA, USA, Sol3
Posts: 269
Default

It is certainly possible that some form of backup is taking place at midnight.
Or more accurately, that it is sending huge amounts of data over the internet to Corporate.

It might also have something to do with the fact that: the server is at least 10 years old, and is a model that doesn't even show up under "discontinued models" in the manufacturer's webpage.
About 6 years ago they upgraded from an operating system that stopped getting service updates 15 years ago to one that stopped getting service updates 3 years ago.

As I said, rebooting it seemed to fix it completely. It just seemed funny to me that the Help Desk was surprised we had let it run for so long without rebooting it when we specifically aren't supposed to reboot it unless they tell us to.

  #5  
Old 02-22-2018, 07:03 PM
Tanasi's Avatar
Tanasi Tanasi is offline
Alpha Geek
 
Join Date: Jul 2006
Posts: 363
Default

I have a couple of customers whose servers haven't been booted in over two years out of fear that they won't reboot. We've been telling them to up grade but to no avail. Both are old CenTOS servers with one being a document server. If they loose it, it would take several weeks to recover from they would have to recreate lots of work. Unknown to them I've been backing up their doc server onto ours every few hours.
Way back when a customer had an old HOBIC costing pc that interfaced with their front desk software via RS-232. A cow-irker and I went to replace it. I copied over the data files to the new pc and my hardware guy hooked it up. It wouldn't interface/talk to save our lives, we tried everything we could think of and others could thing of without any luck. So we put the older PC back in place. It wouldn't restart, just gave HD errors. My HW guy picked it up about six inches and dropped it, wham-o bam-o it started fired up and sat there and worked for another year. Never did get the new pc to work. To by-pass it all we instead replaced the phone switch with one that did the costing we could support remotely.
Oh we also got a fast-driving award coming home.
__________________
Bow down before me for I am ROOT

Preserving precious bodily fluids sine 1952
Closed Thread

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 10:38 PM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2018, vBulletin Solutions, Inc.