Backing up huge amounts of data

[MadMike]

Last night, I finally finished copying all the DVDs to the server, bringing the total to 427 discs, a total of 2.68 TB.

Since it took me the better part of a month to get it all done, I'm a bit paranoid. I'm using a RAID 5, which if I understand correctly, I'm still OK if I lose a drive. Still, I don't want to have to go thru all this again. Is there any to back up this much data?

[Pedersen]

Yes. But not for anything you'd want to do on a home budget. A backup system that will handle all of that data probably costs more than that server.

For point of reference, here's what you would need:
1) Hardware. No single tape can hold that much data. So that means you need a tape library. Small ones are on the order of $5K to $10K. That cost might not include a drive, either.
2) Tapes. You'll need a fair number of them, and they will cost at least $50 each.
3) Software to handle all of the above. Software to handle a tape library exists, and open source software, to boot. I've never configured it, though. And proprietary software to handle it tends to start in the $1K range.

If you want to spend the money, it can be done. Realistically, though, it wouldn't be worth it to me. I'd just bite the bullet and spend another month working on it if that happened.

[Cutenoob]

Mike,

If you have the DVD in the disc format, the case, it IS backed up! Look at it this way- the original is in the case, put away somewhere the kids can't get to it. The electronic copy is on the server.

Now if the e-copy gets corrupted or deleted or something happens to the server, you would not actually lose the data. You'd lose time, really.

Pedersen is right tho, to RE-back up your server, tape library and tapes. I think Mammoth tapes can hold 40G in one tape...I might be wrong...you'd still need a lot of tapes.

But how static will this be?
Are you just loading movies onto the box so you can watch them w/o putting a disc in? Then the other job would be to possibly purchase movies online and save them onto RAID?

If you copied everything to now from a DVD, the only difference would be the newer data - burn those to a physical disc.

Another idea - call a Data center/ ISP and ask if they do electronic / online backups. (you'd need a hellova upload speed tho). Save data THERE instead of in house, and you've got redundancy.

BTW, how many PDU's did you put in the server? And you have it connected to a UPS/ clean power supply, eh?

Cutenoob

[Pedersen]

Actually, CuteNoob, I've had the pleasure of working with a very very nice system at work. The price, though... For home? Forget it. Go and buy a new car. And no, I'm not kidding.

Ultrium 3 (http://www.newegg.com/Product/Product.aspx?Item=N82E16840998009) tapes can hold 400G uncompressed (with DVD data, that's likely what you get). But at $86/tape, it's a tough sell. Even if you get a 40% discount, you're still at $50/tape.

The tape library (http://configure.us.dell.com/dellstore/config.aspx?c=us&cs=04&kc=6W300&l=en&oc=bvcw3k1&s=bsd) is really nice, but at $4421, another very tough sell.

And the software to manage it all? Too many options, both open source and closed source. And of them, choosing the right one is a major hassle. I could see trying to avoid spending the money on that, and going open source instead, and then spending a few months tweaking that to do what you want.

Oh, and that tape library is considered a low end tape library, believe me. It can only hold 16 tapes. I've seen shops that had a tape library that you could walk into, which hold hundreds, if not thousands, of those tapes.

[Cutenoob]

Oh, yeah, I know Pedersen....I saw an EMC tape library @ my last job..it had a cool touchscreen and it's little arms ..you could see inside the box (imagine old phone booth) and it was really neat...

But yeah, mike....If you've just pulled the movies off DVDs,....I'd only backup/copy/burn the ones you have that are NOT on a disk. Let's say you purchase 10 movies from Wally..and 3 from Amazon.com (download)

Well, the 10 are hardcopies, and the 3 are not...those 3 would be the ones I'd backup somewhere else.

P:
you use a file server to backup data, and then backup your backup...where does it end!!! backup your backup'd backup! *aeeieeee*

I guess it depends on how much money you're going to LOSE when it dies. Think of it this way....on 9/11 the stock market was abruptly shut down. But I could still log into my brokerage account. This is due to the data being farmed on multiple datacenters all over america.....and that the corporation was rolling in the BILLIONS.

If they had LOST all that data.....*shudder* Not just me....all the big businesses that bank with them.....

So Mike - time = money ? or data = money ?

Cutenoob

[Pedersen]

Hey, you wanna know a cool tape library to see? I was lucky and contracted at StorageTek for a while. Now, I'll tell you how you can see one of their libraries, and see it in action.

Rent the movie "Eraser", and watch the scene with Vanessa Williams' character copying the data. That big old robotic arm that you see through the window? Yeah, that entire room through the window is one of their tape libraries. They removed the top half of the cylinder which held the arm, too, to make it look better. Oh, and slowed it down. Seems the real deal has those arms running at 60mph in full production mode.

BTW, for anybody who wants to know: These things are genuinely dangerous. 60mph robotic arms will kill. The safeguards they have to prevent it are huge. It's very difficult to close the door with somebody inside. The arms will not move if the door is open. You can push them, but that's it. The motor is disconnected while the door is open. Finally, if you do get stuck inside, the arm makes one slow sweep before going online. Tapping the arm as it makes that first revolution stops it, instantly.

Serious fricking tape library. Also not what I'd get for home :)

[digilight]

Just an idea, but if you are only looking at a little over 2 terrabytes have you thought about a low budget NAS. I've seen enclosures starting around 150 bucks so even if you drop for the 750 gb SATA drives at 179 bucks each you are looking at around 900 bucks for 3TB of backup.

Or you could begin picking up a set of external hard drives as cost allows. Since the drives are smaller then what the total would need to be you would then just backup a section of the alphabet to each drive.

But either way you are looking at around a grand or so give or take for your current backup needs.

The final option while I don't know if its the most cost effeciant or not is to "recycle the same externail enclosure over and over again. Pickup a stack of normal hard drives and just pop them one at a time into the enclosure. Backup that amount of data to the drive (and note whats on the drive). Then pop the drive out of the enclosure and put a new one in. Put all the drives in a safe place and hope that you never have to spend a week to reload the whole thing.

But if cost isn't that much of an issue a 3+TB NAS (Network Attached Storage) is you're best bet for ease of use.

[Shabo]

As of right now, with RAID 5, you are fine if you lose ONE hd. If you lose two, well... things will not be so good for you. I agree with what others have said. Keep the DVDs stored somewhere safe, keep the RAID system, and only back up movies that you've purchased off the internet. Any sort of storage system that you are going to need to back all of that up is going to cost you more than your server did. It's going to suck big time if you lose more than one hd, but it's going to save you a considerable amount of money.

It's a big debate in the tech world as to whether it is more important to create redundancy or to streamline data. There's no happy answer, just that you have to find one that works best for you.

[Geek King]

This may be a little silly to mention, and if so, please feel free to ignore.

If a drive in your RAID 5 goes down, replace it ASAP. Don't wait for a week saying, "I'll get a new one this weekend." I had a boss do that once, and I was on pins and needles all week for fear of total drive failure and over a weeks time to restore a drive that was considered vital to the works. I couldn't say much about it at the time though (I did mention that it would be bad if a second drive happened to fail before the weekend). I was just a lowly peon.

[Shabo]

Excellent point, Geek King. Not at all silly to mention. Sometimes people don't realize the importance of those things, so even if MadMike knew that, other people reading this may not have. So :cheers: to you. :D

[MadMike]

Rent the movie "Eraser", and watch the scene with Vanessa Williams' character copying the data.

I have that one, and it's copied to the server along with the rest of them. ;)

Now if you two are done hijacking my thread... :p

I don't download DVDs from the internet. All the ones on the server are copies of store-bought discs. I'd just rather not have to recopy them all if something goes down. It would be nice if I could just run some copy utility from a backup device and just leave it, but until they start making petabyte drives, I guess that's not a possibility.

Now, for a possibly dumb question -- if one of my drives dies on me, how do I tell? :confused:

[Pedersen]

Now if you two are done hijacking my thread... :p

Never! Death to the infidels! This thread will be held until... Wait, sorry, wrong forum :whistle:
It would be nice if I could just run some copy utility from a backup device and just leave it, but until they start making petabyte drives, I guess that's not a possibility.
Not without spending a lot more money than you'd like to, I'm sure. Sorry to have to say that. Unless, of course, you'd like to backup to DVD. That's a cheap option. Best estimate is that it should only take about 427 of them, too ;)
Now, for a possibly dumb question -- if one of my drives dies on me, how do I tell? :confused:
If this card is at all like the one I used way back when for my old job, you'll know. The thing will start making a loud howling noise that will wake the dead. And since it does use the same software, it's likely a lot like the card I used way back when.

When it does, log in to the machine, and run megamgr From there, you've got a menu driven program which will let you turn off the alarm (on the adapter, btw, so that will help you find it), rebuild the array, etc. And it will tell you which drive is failed (when you look at the physical drives from the main menu).

Hope that helps a bit!

[Fashion Lad!]

This would be a long and tedious thing to do...

Why not go and pick up three 1TB external hard drives with firewire capabilities and back them up that way? Am I missing something otherwise??

[MadMike]

Why not go and pick up three 1TB external hard drives with firewire capabilities and back them up that way? Am I missing something otherwise??

That might be a possibility, although I'd probably have to get USB ones. I don't think the board I put in the server has firewire capability. Unless I get a card that would give me those connections. I think I have a few PCI slots free in the server.

[digilight]

Mike, just for comparison, I just finished transfering about 200-250 gigs of music from over to my new mybook drive earlier this week. It was extremely time consuming going that route. I would say that it took probably close to 10-15 hours to complete (give or take). But I picked the drive up for around a hundred bucks it seems to be built like a tank, great ventilation and all.

[Fashion Lad!]

That might be a possibility, although I'd probably have to get USB ones. I don't think the board I put in the server has firewire capability. Unless I get a card that would give me those connections. I think I have a few PCI slots free in the server.

There are cards that you can get that can give you firewire capabilities. Yeah, it'll still be time consuming, but it's cheaper than a lot of other options and very secure.

Apparently, HP is coming out with a home use server with 2TB storage capacity.We're stoked for it at work.

[Shabo]

Actually, USB 2.0 is faster than the first firewire, so if you can get your hands on the newer firewire technology, go for it.

[Fashion Lad!]

Actually, USB 2.0 is faster than the first firewire, so if you can get your hands on the newer firewire technology, go for it.

Well considering Firewire 800 (IEEE 1394b) has been out for about 4 years now and still quite a bit faster than High-speed USB, I don't think it'd be a problem.

Besides, Firewire was basically made for that type of data backup.

[Shabo]

Well considering Firewire 800 (IEEE 1394b) has been out for about 4 years now and still quite a bit faster than High-speed USB, I don't think it'd be a problem.

Besides, Firewire was basically made for that type of data backup.

As told by Wikipedia:

Firewire 800 allows a transfer rate of nearly 800 Mbit/s, BUT Windows XP limits the transmission speed to 100 Mbit/s (until you run the manual patch), whereas USB 2.0 has a transfer speed of nearly 480 Mbit/s. Therefore, IF you have the ports for Firewire on the computer, it's still going to be slower than USB 2.0 if you are running WXP unless you know about the patch. Even Vista does not support the full speed of Firewire 800 yet (when will Microsoft learn to play nice with Apple? *le sigh*). If I remember correctly, MadMike is using a Microsoft OS, so he will need to get the patch if he wants to use Firewire. Either way, it's going to be tremendously expensive for him to back up his entire collection using an external (or 3 or more) so he might want to back up his favorite movies on an external, and then if he ends up needing to reload the movies on his computer, it won't take as long as if he didn't back any up, and as he gets additional funds to purchase external drives, he can back up more movies. That would probably be his best bet.

[Pedersen]

Actually, he's using Ubuntu Linux right now, so that shouldn't be an issue.

[Fashion Lad!]

Therefore, IF you have the ports for Firewire on the computer, it's still going to be slower than USB 2.0 if you are running WXP unless you know about the patch. Even Vista does not support the full speed of Firewire 800 yet (when will Microsoft learn to play nice with Apple? *le sigh*). If I remember correctly, MadMike is using a Microsoft OS, so he will need to get the patch if he wants to use Firewire.

There you go MadMike, you now know about the patch. ;)

[tollbaby]

Mike, hindsight is 20/20, but here's what I plan on doing when I get my brand-spanking new laptop (I currently don't have a DVDRom drive or a writer): I want to make a disc copy of every one of my DVDs.

While you were copying them all onto the hard drive, I'd have made a backup DVD copy as well (granted, too late for that now, but I doubt you're done acquiring DVDs... right?). I know it kind of isn't what you were looking for... just decided to share anyway in case anybody else is playing around with ideas.

[MadMike]

While you were copying them all onto the hard drive, I'd have made a backup DVD copy as well (granted, too late for that now, but I doubt you're done acquiring DVDs... right?). I know it kind of isn't what you were looking for... just decided to share anyway in case anybody else is playing around with ideas.

I'm not sure you understand why I was copying them. It wasn't to have a backup copy of them, it was to have them on a server that is accessible to the media PC I built (I just finished the second one, I'm hoping to finish the third one this weeked) so that I can have all my movies in one easy to access location, and I'm not constantly swapping discs out.

There's a couple threads about my project in here, but I'm too lazy to look for them right now. :p

[tollbaby]

No, I understood why you were backing them up.... it just seems silly to me to make yet another backup of THAT (but then, I'm the girl who's had two computers die on me in the last year).... just ignore me ;) I'm not a techie type person anyway. LOL