Please, people, pay attention to your computers once in a while. (Long)

[ArcticChicken]

The funniest part of all this is, this isn't what they pay me for. I'm a pharmacist, not an IT guy; I'm there to fill prescriptions and counsel patients, not fix the damn computers, but somehow wherever I work I wind up doing this stuff . . .
.

Is there someone they pay for it?

[protege]

You know how to do it, and apparently have a very strong work ethic. That would be why.

In my case, I got stuck doing IT...simply because I replaced one hard drive. Seriously though, I work for a financial services company. One thing I take very seriously, is backing up our data. If we lose it, we're royally screwed. That's why when we bought the new server last year, it has multiple TB drives in a RAID. Plus, the server backs itself up onto a small unit at my boss' place. As if that wasn't enough, I back everything up onto discs a few times a week, and take said discs off site.

I'd rather do the IT stuff myself. Mainly because I know my company's needs better than our 3rd party guy...plus I can get it done faster than he can. I'm on site, he's across town. He knows what he's doing...but if the shit hits the fan (or in my case, the server), we can't have the down time. We're basically out of business until the server is back online.

[Pedersen]

This problem can be solved using the right encryption set-up... Needs to be double-checked, to make sure nothing's leaking, but it's doable.

Double-checked is not sufficient.

Here's what it would take before encrypted medical data could be safely backed up to the cloud somewhere:

• We must have a perfect encryption algorithm.
• That algorithm cannot have any weakness now, nor can it have any weaknesses for the next 150 years (long enough to ensure that anybody mentioned in it will be dead).
• We have to know it will not have any weaknesses during that 150 years before we deploy it.
• The algorithm must be implemented perfectly.
• The operating system running the encryption must not have any holes that would expose the data inadvertently (for instance, swap space is normally unencrypted).
• The people using the backup system must be able to be trusted to run the backups perfectly every time (this can be mitigated by making the entire system automatic).
• The people configuring the system must get the configuration done perfectly.

And that's just what I have so far, and I'm being nice and not attacking the operating system issues, or anti-virus issues, or ... you get the idea.

Medical data, in the cloud, in unknown/unknowable hands, is a bad idea. It's also illegal, at least here in the US. Personally, I'd like to keep it that way for all the failure points I listed above, and the many others I hinted at.

[suburbandecay]

Once I removed the bezel, it was obvious which drive had failed; it was the only one whose I/O light wasn't flashing. Now you mention it, the power light might have been orange as well, I don't recall exactly. Problem is, unless you take off the bezel, you can't see the drives; out of sight, out of mind.

Normally on a dell system, the power button will flash orange/amber when there is an issue and will continue to do so until you fix it. Generally, I look at my servers with the lights out to check the lights on systems/drives. It makes the errors stand out more.

[Krysalis]

Being nerdy, I'd love to just have a RAID 1 or 5 setup on my home machine, along with weekly backups. Unfortunately, I lack the internal configurations for such a whimsy (my motherboard only has 5 SATA slots, and one of them has to be used for Optical), and 2 PATA. Yeah, not doable. Also lack the physical space to setup a rack and a few drives. I can dream, though, right?

I'm also fascinated by POST messages at boot, so I know if something decides to try blowing up. Once had a hard drive fail on my old XP machine, took a look at it, and the PATA cable had disintegrated.

Also, I think I have the cousin to your old Dell there. Picked it up at a thrift shop for 20 bucks, looks like it stopped use after the dotcom boom. Had lots of virii on it. Need to clean it up sometime...

[wolfie]

I think that's how it's supposed to work here. We have four tapes, labeled Daily-1, Daily-2, Weekly and Monthly. All four of these are supposed to have the same stuff on them, more or less. It's not an incremental backup, if I understand it correctly, each tape is an independent full backup. We're supposed to change the Daily tapes daily, once a week put the Weekly one in there and then stick it in the safe, and once a month put the Monthly one in, and then take it home and put it in your house, which I guess is to protect it in the event of what you showed above. Unfortunately our safe isn't very, so all the tapes not currently in use are stored in a consumer-grade fireproof box on the shelf next to the computer. Including the monthly one, until I explained to my boss its purpose. I think he's got it in the business office, which is in a different building on the other side of the village; just have to remind him to bring it back to the store once in a while...

You need a 5th tape (instead of "Monthly", have "Monthly 1" and "Monthly 2") to protect yourself from a complete wipeout. What happens if the place burns down while the monthly tape is on the premises? That's why you only bring one of the Monthly tapes out of the off-site storage at any one time (alternate which one you use). Worst case, you still have the other Monthly tape off-site to recover from. Do you also have off-site a duplicate tape drive (tech items have a habit of getting discontinued), along with disks to install your O/S and the backup/restore software on a "virgin" machine? NASA has a lot of unreadable data from early unmanned missions - it's stored on (what was at the time) industry standard tapes, but they don't have any of the (now obsolete) drives left in order to read the tapes.

[hjaye]

If it's a vendor owned and supported server, they should have monitoring software in place that will alert them when these things happen. It sounds to me like your vendor is only doing half the job they're supposed to be doing.

[earl colby pottinger]

Then

According to Shalom, the hardware/service is the responsibility of the 3rd party tech company. Personally I'd write a note to DM and say, dude, they're not pulling their weight. I had to CALL for help, and that's to get the updates that THEY didn't get.

I read the original post again. The computer is leased, the software is leased, nowhere does it say a maintenance contract is also included. Those can be included in a lease but not always. Consider, if you lease a car who pays for insurance?

The leasing company will probably write back to point out that the OP's company did not want to pay the small monthly fee to monitor their system saying that they have their own tech.

Of-course they (OP's company) probably laid off the tech a year later since it was clear that they did not need him since nothing was ever going wrong. And of-course since the system has now ran for three years with no problems the monitoring fee is clearly just a scam to make money off them.

Yes, I have seen it in real life. More than one company in-fact.



Read the other posts, notice that a number of people who are not trained to be techs, not paid to be techs and above are the only people who seem to know what they are doing are running the backups, fixes and updates in their companies?

If anything happens to these people (lay-offs, sickness, accidents, better paying jobs), these companies are not even aware how much they are needed, other-wise they would had hired a tech already.

[Chromatix]

After about 2.5 years of service, my workstation's hard disk decided to fail several weeks ago.

In theory, there is SMART monitoring that sends alerts to IT. In practice, the first sign that anything was wrong was that everything slowed down and became vaguely unstable - and the following *week* it suddenly failed to boot at all. Even after that week, IT had no idea that anything was wrong until I poked my head in the door (now being unable to send e-mail).

Now I have a fresh hard disk, but nothing was (easily) recoverable from the old one. If it had been detected earlier, there's a good chance that everything could have been retrieved.

The lesson for IT is to make sure that SMART monitoring actually *works*. Have the monitoring tool send a heartbeat message every week saying that everything is fine, and compare it to the list of active machines. If one is missing for a week or two, it's probably someone off on holiday, but if it misses all beats for a month, it's worth investigating.

[RealUnimportant]

I have SMART monitoring on my work machine. However, if the machine's not rebooted while under observation (which rarely happens due to time constraints) then no-one can see the status... So when the drive failed out of the blue the other month, it caught everyone by surprise. Rebuilding the machine to a functional level took 2 weeks of reinstalling & catch-up configuring of stuff that had changed since the one-size-fits-none image was taken, and even now it's still not back to normal. It took a long time to get management to realise that in essence it is a "new" computer now, even though only one part was changed...

[Shalom]

I read the original post again. The computer is leased, the software is leased, nowhere does it say a maintenance contract is also included.

I should have said it explicitly, you're right. Maintenance is included, both hardware and software.

Read the other posts, notice that a number of people who are not trained to be techs, not paid to be techs and above are the only people who seem to know what they are doing are running the backups, fixes and updates in their companies?

If anything happens to these people (lay-offs, sickness, accidents, better paying jobs), these companies are not even aware how much they are needed, other-wise they would had hired a tech already.

This company (or at least the pharmacy department) employs six people, four of whom are pharmacists, and only one of the latter is full time (and it's not me). We've never had a computer tech, because the vendor is supposed to be taking care of that for us. Unfortunately they aren't proactive (hate that word, damn it), in effect waiting for things to break. Preventive maintenance doesn't seem to be their big point.'

I have a mega-rant coming up on this, which I'm gonna post in a new thread once I get a chance to commit the electrons to paper, but the tl;dr version is, the stuff they fixed in April broke again, and the monitoring system, such as it is, is so obscure and cryptic that even I, who have been working with computers since 1980, found it too much of a PITA to be of any use.. I am not happy with them at the moment.

[dalesys]

...the monitoring system, such as it is, is so obscure and cryptic ...

I take it as it isn't written "...using standard Intel numeronics..." ?

(From a Shadio Rack advertising blurb on an Editor/Assembler program.)

[Howler]

There are two power supplies, but they're both plugged into the same UPS, so I don't really see the point. (OK, it protects against failure of the supply itself, but if we really wanted to do it right, they'd be plugged into two different outlets on separate circuits.)

PLEASE tell me that they took your advice...

I'm in the process of spec'ing a CG rendering farm and one of the first things I mentioned was the power requirements (at least 2 circuits and 2 UPS boxes for a blade box)

At least my boss said 'Yeah, do that!'

[Shalom]

PLEASE tell me that they took your advice...

Nope, both cords were plugged into the same UPS.

Which failed catastrophically at 2:10:29 PM on Monday afternoon, without any sort of warning, dumping the server and everything else connected to it.

I unplugged everything and found enough wall outlets to plug everything into, and we were back in business. Thank G_d all the drives spun up OK, even the ones with impending failure warnings.

They overnighted us a new UPS, or at least a replacement UPS, which arrived Tuesday morning. I had the afternoon shift that day, so I didn't get in until 2:05 PM, finding the UPS in a box waiting for me to install it.

Guess what happened at 2:09:51.

Yup, the store had a brief power failure. Dumping the server and everything else connected to it, again, since we still had no UPS hooked up.

Thanks, Orange & Rockland Utilities, for your excellence in keeping the juice running; thanks, APC, for your wonderful piece of equipment that was incapable of keeping the server running even without a power loss; and especially thanks, [Vendor] Pharmacy Services, for sending me a "new" UPS that lasted not even 5 months from installation to failure.

I now have the server hooked up with one power supply plugged into the UPS, in case ORU can't keep the lights on again, and the other plugged right into the wall, in case the next UPS blows its zap again. Hopefully one or the other will keep things running.

I'll post the saga of the disk drives tomorrow; that one probably deserves its own thread.

[mhkohne]

I have SMART monitoring on my work machine. However, if the machine's not rebooted while under observation (which rarely happens due to time constraints) then no-one can see the status... So when the drive failed out of the blue the other month, it caught everyone by surprise. Rebuilding the machine to a functional level took 2 weeks of reinstalling & catch-up configuring of stuff that had changed since the one-size-fits-none image was taken, and even now it's still not back to normal. It took a long time to get management to realise that in essence it is a "new" computer now, even though only one part was changed...

Don't rely on SMART data, even if the monitoring works - Google did a paper a few years back that showed that the SMART data doesn't actually show anything before a very large number of failures. By all means, keep an eye on the SMART data, but don'tnbe surprised if the drive fails without so much as a peep before-hand.