ZFS In Action

SysOp — Thu, 02 Oct 2008 23:05:07 +0000

Well we’ve now had a few months of full production ZFS usage. We’ve had our first drive failure which exposed the oddities of drive failures under ZFS. It does work REALLY hard to cover them up, so much so that it never really quite gave up on the dead drive until I ran zpool offline on the drive. That said there was NO effect to users at all as far as I can tell, despite the drive producing errors and just generally not responding the only commands suffering were zpool related commands that actually went to access the affected drive directly, overall performance and function of ZFS didn’t degrade. Once I told ZFS to offline the offending drive the zpool commands didn’t hang anymore. The never hung indefinitely, just for a timeout period. Later we actually replaced the physical drive, part of which was figuring out how to get the new drive visible since we’re using a RAID card, but aren’t doing any RAID with it, and finally running zpool replace. Side rant – LSI‘s MegaCli is…Sometimes hard to understand. Forget about the [] means optional convention. When it’s talking about [E0:S0] it means literally that. except E0 and S0 are just numeric so enclosure 2525 slot 1 in say the -pdInfo command is expressed as MegaCli -pdInfo -PhysDrv[252:1] -a0 so in BASH and BASH-like shells you need to actually type MegaCli -pdInfo -PhysDrv\[252:1\] -a0 in order to make sure you don’t run afoul of the fact that [ introduces a filename expansion in bash – why can’t they just follow the normal man page conventions on SYNOPSIS sections and plain old good practice with regards to command line arguments?

The quirkiest bit is that a simple zpool status, would indicate everything was A/OK and healthy. You really do have to have “something” monitor the READ/WRITE/CSUM columns because ZFS is only going to go into DEGRADED states if the device becomes completely absent, and even then due to it’s fault tolerance that might not stop the filesystem. Depending on the filesystem settings, it’ll either try to keep working from cache, block, return i/o errors, or panic the system. This is actually a Good Thing. Because say you’re moving a fibre channel loop or an iSCSI cable? Well ZFS will handle that gracefully even if ALL the drives disappear. Folks, don’t try that with EXT2/3 or ReiserFS. Trust me, just Don’t Ask How I Know.

Goodbye Areca, Hello LSI

SysOp — Fri, 25 Jul 2008 17:26:31 +0000

While it’s not clear exactly who-is-causing what, what is clear is the areca driver tries to de-reference a NULL pointer, this is either because the adapter screws up, or the driver screws up somewhere. The result is a Solaris kernel fault, pointing at the arcmsr driver, and apparently an adapter lockup. It’s not 100% clear what causes this condition. It could be the driver not handling some buffer appropriately, it could be the card sending an error that the driver doesn’t handle. It’s pretty likely though the issue is completely inside of the arcmsr driver and Areca hardware. One thing we did discover that means we HAVE to replace the Areca hardware is that in JBOD mode (which is how we use it, since we’re using Solaris’ ZFS superset of RAID functionality), any disk failure seizes the whole card up until the failure clears, or maybe until some apparently long timer clears. SATA and SAS have ethernet-like link failure detection. You know within milliseconds when the cable is pulled. The Areca’s in JBOD mode seem unable to handle hot-swap of any type, or even failures of any type. When we tried to get them to address it all we received was vague “you must have a failing drive” answers, which for a RAID card is a bad answer. Even in JBOD mode the controller should signal/propogate an error. Solaris’ would handle this condition.

Then there’s the boot selection. All logical drives appear in the boot selection. Either the list fills up or the Areca’s only show drives on the first controller. That’s a problem if you want to be able to boot an alternate drive on a second controller.

So, Saturday, I get to backup the entire user data. Blow the whole damn thing away. And start over. *sigh*

I know I posted about this before, but we were avoiding the whole rebuild thing, turns out I’m going to have to do that anyway. Argh. The biggest reason is the LSI cards use an on-disk metadata format (apparently). They’re kinda quiet about it all so I’m not sure how big it is or where on the disk. I’m betting it’s the last N megs of the drive. 8, 16, 20 something. Most people won’t notice, but if you’re migrating, it becomes noticeable.

DotBlag.Com » lsi

ZFS In Action

Goodbye Areca, Hello LSI