Well, Horsey did it again! In one of his regular columns, he talked about an issue that's all the rage in the online culture, namely MP3 file trading and the RIAAs recent actions to attempt to curb the spread of copyrighted material shared freely online.

Oh, the article! Here it is!

His well-thought-out article even managed to rate a front page appearance on the geek site, Slashdot this week. Not bad.

Maybe someday the recording industry will get a clue and make true CD-quality downloads available legally, unencumbered by encryption, at a reasonable price. Perhaps they'll be smart enough to include a catalog vast in both depth and width, extending back to the earliest days of audio recording, and covering every possible artist, composer, performer, musician, etc. I'd gladly pay!

Here's hoping!

After having my file server, web server, mail server, and Jabber server offline since the 11th, I'm happy to report that both web and email services are back online. I haven't yet got Jabber running again, nor have I set up Samba again so my files will be shared on my home network. Throughout this last week, it's been awkward knowing my web sites were down. I was unable to vent steam on this Yakkity Yak log. All this because of some disk corruption in my RAID-5 array.

It started a week ago Thursday, on the 2-year anniversary of the terrible tradgedy of 9/11. I was moving the four drives that form the RAID array into a new computer, an Athlon 2600+. Unfortunately, I made a poor choice. Instead of attaching them in exactly the same way they had been attached in the old Athlon 700 box, I instead put each drive on its own separate ATA cable. In the old box, there were two drives each on two cables.

When I booted FreeBSD, things looked normal, except instead of the drives being listed as ad0, ad1, ad2, and ad3, they came up ad0, ad2, ad4, and ad6. The Vinum software system that makes the drives into a RAID-5 array loaded, but then put the RAID-5 volume into a crashed or down state, because only the first drive was correctly recognized. That meant my attempt to boot failed, and I was dumped to a single-user-mode shell.

"Not a problem!" I thought. Well it was a bit of a problem, since my entire /usr partition was on that RAID-5 volume, and thus inaccessable. Only my root partition and backup root partition were online, since they're ordinarly disk partitions. (I periodically manually backup the root partition to the backup partition, so a disk failure won't leave me totally without a bootable disk.) That meant I couldn't use my favorite editor, vi was unavailable.

I wished it were, so I could whip up a corrected vinum configuration in a file, check for typos, then load it. Instead, I ended up teaching myself ed the hard way, by trial and error. Using my old vinum configuration as a template, I made the changes so that the subdisks would use the correct devices. Once the file looked good, I popped over to the vinum interactive command-line utility, and loaded the new configuration file.

It loaded fine, but it thought this was a brand-new plex and volume, instead of an existing one. Oops! I still don't know how to reconfigure vinum on-the-fly to use a different device for a subdisk (e-mail me if you do!). Quickly, I forced the volume and plex into the down state by hand. Whew! Then working carefully, I put the subdisks back into up state, then the plex, then the volume. Things looked good again.

Just when things looked good, I made a blunder. I ran fsck (the file-system checker) on the UFS2 partition. Oops! I should have made sure the RAID-5 parity was okay first. There were lots of errors!

I back-tracked at this point, and ran vinum's checkparity command. Ick, there were parity errors! So I had to run rebuildparity. Let me tell you, that's not exactly a speedy process. I think it took 5-6 hours to complete, all this while the computer lay in the middle of my livingroom floor, case open, wires snaking across the carpet.

Because the Athlon 700 box (the one I gutted to migrate stuff to the new Athlon 2600+ box) was my Internet router and gateway in addition to acting as my file server, web server, mail server, and jabber server, I was disconnected from the world. I couldn't let that last long, now could I? With a quick re-route of some ethernet cables, and a reconfiguration of my laptop's IP address, I had my laptop back online so I could be useful while the RAID-5 parity rebuild executed on my livingroom floor.

I'm not sure if I've got a hardware issue, or if there's a FreeBSD 5.1R driver issue with the Highpoint ATA RAID controller, the 372 I think. Since I use vinum for my RAID, I was using the Highpoint just as an ATA controller, not for any RAID services. In the process of recovering my vinum RAID volume, I noticed some odd errors, and even a system crash that may have been due to either FreeBSD's ATA drivers for the Highpoint 370 in non-RAID mode, or due to the hardware itsself.

In the end, I put the four drives back into their old configuration, two per cable, and connected them to the motherboard's onboard ATA controller, and disabled the Highpoint controller in the BIOS. After a second parity rebuild (the crash and driver/hardware issues caused some more corruption), multiple fsck runs, I was able to get my /usr filesystem back online. Unfortunately, the corruption meant there was lots of stuff in lost+found to go through. My MySQL database that contains my web site blog posts, as well as PHP sessions, and the forums for SU³G was gone. At least until I noticed it buried in lost+found.

Even though fsck thought the UFS2 filesystem was now consistent, there was still something wrong. When I tried to remove certain files, FreeBSD 5.1 would panic. I ended up just renaming the directories containing the bad files something like bogusstuff or the like, and working around them. Since I needed a consistent, working filesystem, I needed to move my important data off the RAID volume. That's not an easy thing to do when you have over 200 gigabytes of data, and no other drive or array handy to store it on temporarily.

Earlier this week (on Tuesday), after having whittled the 200 GB down to only 90 GB (the core essentials, you might say), I broke down and bought a 160 GB drive from the local Staples. I plugged it into the Highpoint ATA controller, and used fdisk and disklabel, and newfs on it, then installed FreeBSD 5.1-RELEASE on it since my existing FreeBSD installation was severely crippled by a barely-working /usr partition. Things went quite smoothly.

Then on reboot, the Highpoint/FreeBSD 5.1 data corruption issue hit again. (Yeah, it might just be my motherboard...) Now the 160GB drive had a messed-up filesystem on it. There's no way I was going to use it this way to backup my data.

I yanked the drive out of the system and put it in an old Pentium II 350MHz system that I by now had up and running, functioning as my home's network gateway router, NAT, and DNS server. With the drive repartitioned, and a new filesystem created, still connected to the P2-350, I started copying data to it across the network.

Grrr! It was s...l...o...w, even on a 100mbps ethernet. About a third of the way through the copy, I realized how stupid I was to do it how I was doing it. I used tar -cf - to roll up the data and shoot it to standard output, which I piped into ssh -e none -c arcfour adg@pentium2-350.host. SSH shot it across the network to the P2-350, and there I executed cd /backup ; tar -xf - to unroll the data on my backup drive.

This odd way of transferring the data meant I didn't have to create a tarball on the source machine, then transfer it. I was doing it on-the-fly. But using SSH meant an encrypt/decrypt on all 90 gigabytes of data, and the P2-350, also busy as a router/gateway, in addition to untarring it, had to decrypt it. I should have used netcat or something to shoot it across the ethernet in-the-clear. It took seven excruciatingly long hours to complete the transfer.

Once the backup was successfully completed, I rebuilt the vinum volume from scratch, and put a new filesystem on it. After carefully reverifying parity, and making sure the new UFS2 filesystem was happy, I tricked the existing FreeBSD installation into compiling a fresh FreeBSD RELENG_5_1 system from sources downloaded using cvsup. I say "tricked" because I moved things around on my existing install to fit the base stuff that was on [/i]/usr[/i] on the small root partition, and used plentiful symbolic links to put the source, object, include, and other /usr subdirectories on the fresh vinum volume.

I started the copy of the data from my backup drive to the fresh vinum volume at 1:00AM so the 7-hour copy could occupy the wee hours of the morning while I slept. Then, at last, I had a good system installation, and my data was ready for reassembly.

Today I got my web server and mail server back up and running. The Pentium II 350 will stay doing duty as firewall/gateway/router, and the Athlon 2600+ will sit on the inside, the mail and web services network address translated to the outside world. I still need to get Jabber working, then install Samba, finish cleaning up my data, then things will be back to normal.

Oh, I still need to figure out where I'm gonna stash the new Athlon system. The livingroom floor just isn't a good permanent place for it.

What about the Athlon 700? It now has Gentoo Linux on it, in preparation for an install of Asterisk, an open-source software PBX system. It's got a couple of Digium cards plugged in, awaiting some voice-over-IP fun. Eventually it will become my home answering machine (voice mail system with interactive menu), as well as gatewaying my analong telephone handsets to Internet VOIP telephony. Fun, I tell ya!