An Article from Aaron's Article ArchiveEvil MP3s, RAID Crash Recovery, and Upcoming VOIP Fun
Photo: BeedaisyIPv4You are not logged in. Click here to log in.
Use Google to search aarongifford.com:
Evil MP3s, RAID Crash Recovery, and Upcoming VOIP Fun
Friday, 19 September 2003 8:06 PM MDT
Well, Horsey did it again! In one of his regular columns, he talked about an issue that's all the rage in the online culture, namely MP3 file trading and the RIAAs recent actions to attempt to curb the spread of copyrighted material shared freely online.
Oh, the article! Here it is!
His well-thought-out article even managed to rate a front page appearance on the geek site, Slashdot this week. Not bad.
Maybe someday the recording industry will get a clue and make true CD-quality downloads available legally, unencumbered by encryption, at a reasonable price. Perhaps they'll be smart enough to include a catalog vast in both depth and width, extending back to the earliest days of audio recording, and covering every possible artist, composer, performer, musician, etc. I'd gladly pay!
After having my file server, web server, mail server, and Jabber server offline since the 11th, I'm happy to report that both web and email services are back online. I haven't yet got Jabber running again, nor have I set up Samba again so my files will be shared on my home network. Throughout this last week, it's been awkward knowing my web sites were down. I was unable to vent steam on this Yakkity Yak log. All this because of some disk corruption in my RAID-5 array.
It started a week ago Thursday, on the 2-year anniversary of the terrible tradgedy of 9/11. I was moving the four drives that form the RAID array into a new computer, an Athlon 2600+. Unfortunately, I made a poor choice. Instead of attaching them in exactly the same way they had been attached in the old Athlon 700 box, I instead put each drive on its own separate ATA cable. In the old box, there were two drives each on two cables.
When I booted FreeBSD, things looked normal, except instead of the drives being listed as ad0, ad1, ad2, and ad3, they came up ad0, ad2, ad4, and ad6. The Vinum software system that makes the drives into a RAID-5 array loaded, but then put the RAID-5 volume into a crashed or down state, because only the first drive was correctly recognized. That meant my attempt to boot failed, and I was dumped to a single-user-mode shell.
"Not a problem!" I thought. Well it was a bit of a problem, since my entire /usr partition was on that RAID-5 volume, and thus inaccessable. Only my root partition and backup root partition were online, since they're ordinarly disk partitions. (I periodically manually backup the root partition to the backup partition, so a disk failure won't leave me totally without a bootable disk.) That meant I couldn't use my favorite editor,
I wished it were, so I could whip up a corrected vinum configuration in a file, check for typos, then load it. Instead, I ended up teaching myself
It loaded fine, but it thought this was a brand-new plex and volume, instead of an existing one. Oops! I still don't know how to reconfigure vinum on-the-fly to use a different device for a subdisk (e-mail me if you do!). Quickly, I forced the volume and plex into the down state by hand. Whew! Then working carefully, I put the subdisks back into up state, then the plex, then the volume. Things looked good again.
Just when things looked good, I made a blunder. I ran
I back-tracked at this point, and ran vinum's
Because the Athlon 700 box (the one I gutted to migrate stuff to the new Athlon 2600+ box) was my Internet router and gateway in addition to acting as my file server, web server, mail server, and jabber server, I was disconnected from the world. I couldn't let that last long, now could I? With a quick re-route of some ethernet cables, and a reconfiguration of my laptop's IP address, I had my laptop back online so I could be useful while the RAID-5 parity rebuild executed on my livingroom floor.
I'm not sure if I've got a hardware issue, or if there's a FreeBSD 5.1R driver issue with the Highpoint ATA RAID controller, the 372 I think. Since I use vinum for my RAID, I was using the Highpoint just as an ATA controller, not for any RAID services. In the process of recovering my vinum RAID volume, I noticed some odd errors, and even a system crash that may have been due to either FreeBSD's ATA drivers for the Highpoint 370 in non-RAID mode, or due to the hardware itsself.
In the end, I put the four drives back into their old configuration, two per cable, and connected them to the motherboard's onboard ATA controller, and disabled the Highpoint controller in the BIOS. After a second parity rebuild (the crash and driver/hardware issues caused some more corruption), multiple fsck runs, I was able to get my /usr filesystem back online. Unfortunately, the corruption meant there was lots of stuff in lost+found to go through. My MySQL database that contains my web site blog posts, as well as PHP sessions, and the forums for SU3G was gone. At least until I noticed it buried in lost+found.
Earlier this week (on Tuesday), after having whittled the 200 GB down to only 90 GB (the core essentials, you might say), I broke down and bought a 160 GB drive from the local Staples. I plugged it into the Highpoint ATA controller, and used
Then on reboot, the Highpoint/FreeBSD 5.1 data corruption issue hit again. (Yeah, it might just be my motherboard...) Now the 160GB drive had a messed-up filesystem on it. There's no way I was going to use it this way to backup my data.
I yanked the drive out of the system and put it in an old Pentium II 350MHz system that I by now had up and running, functioning as my home's network gateway router, NAT, and DNS server. With the drive repartitioned, and a new filesystem created, still connected to the P2-350, I started copying data to it across the network.
Grrr! It was s...l...o...w, even on a 100mbps ethernet. About a third of the way through the copy, I realized how stupid I was to do it how I was doing it. I used
This odd way of transferring the data meant I didn't have to create a tarball on the source machine, then transfer it. I was doing it on-the-fly. But using SSH meant an encrypt/decrypt on all 90 gigabytes of data, and the P2-350, also busy as a router/gateway, in addition to untarring it, had to decrypt it. I should have used
Once the backup was successfully completed, I rebuilt the vinum volume from scratch, and put a new filesystem on it. After carefully reverifying parity, and making sure the new UFS2 filesystem was happy, I tricked the existing FreeBSD installation into compiling a fresh FreeBSD RELENG_5_1 system from sources downloaded using
I started the copy of the data from my backup drive to the fresh vinum volume at 1:00AM so the 7-hour copy could occupy the wee hours of the morning while I slept. Then, at last, I had a good system installation, and my data was ready for reassembly.
Today I got my web server and mail server back up and running. The Pentium II 350 will stay doing duty as firewall/gateway/router, and the Athlon 2600+ will sit on the inside, the mail and web services network address translated to the outside world. I still need to get Jabber working, then install Samba, finish cleaning up my data, then things will be back to normal.
Oh, I still need to figure out where I'm gonna stash the new Athlon system. The livingroom floor just isn't a good permanent place for it.
What about the Athlon 700? It now has Gentoo Linux on it, in preparation for an install of Asterisk, an open-source software PBX system. It's got a couple of Digium cards plugged in, awaiting some voice-over-IP fun. Eventually it will become my home answering machine (voice mail system with interactive menu), as well as gatewaying my analong telephone handsets to Internet VOIP telephony. Fun, I tell ya!