CloudEngines saw fit to include relatively robust filesystem support sporting HFS+, NTFS as well as the usual ext2/3, FAT/FAT32/VFAT.  HFS+ is provided by custom kernel modules, that work better than their cousins integrated into the Linux kernel (I’ve had some experience with them).  I haven’t yet tested the NTFS support.

The really unique thing about the Pogoplug is that it is completely integrated with the Internet.  The API system allows you to write your own scripts, or use others.  You can even cross-compile and run binaries on the Pogoplug itself.  Setup was easier than anything I’ve ever used of this nature. I plugged my device in, and went to the Pogoplug site.  I created a login, it quickly identified my Pogoplug device (I assume the Pogoplug called home and it saw us both coming from the same IP) and I was able to immediately use the WebUI to upload and download files.  They don’t support SMB/CIFS, but they have OS level drivers for Windows, OS/X, and Linux readily available.  I’ve only tried the OS/X and Win64 (Windows 7) drivers and they work very well.  Honestly since they seem to be fully supporting multiple OSes, and SMB/CIFS is so complicated and slow I don’t feel like this is much of a minus.  It does limit native support to “Supported” OSes for now unless the WebUI/API access fits for you.  I don’t know what their product roadmap is but I did find a (broken) symlink/mention of Samba within the device itself, and for those users that are on other OSes and *really* want CIFS they can cross-compile and install/run their own Samba binaries.

The native clients I’ve tested under Windows 7 and OS/X 10.6 (Snow Leopard) seem to perform well and bug free.  I’ll be pushing them a bit harder in the coming days to see what happens.  So far though I’ve had no issues.  The Native clients can be set to multiple drive or single drive mode.  The Windows client defaults to single drive mode with all of your connected drives showing up as P:\<Device Name>.  The OS/X (and I assume Linux clients) default to multiple drive mode with all of your connected drives showing up as separately mounted volumes.

You also can not initialize (format) a drive from the Pogoplug.  So you have to format your removable devices with a PC/Mac first.  This rather minor since if you have this device, then you have a machine, and the drives are removable by nature.

Performance is also very good, thanks in no small part to the speedy embedded Orion SoC, the Ethernet controller also has TSO, Receive and Transmit Checksum Offloading (part of TOE) which helps keep the CPU free from a lot of overhead.  These offload features are common in higher end servers and many ‘gaming’ Ethernet adapters.  Having a 1.2GHz CPU and these helpful hardware offload engines means that the CPU doesn’t work too hard and the performance will generally be limited by the RAM speeds.  To get the full LAN performance you do need to install the native clients.  The unit may be a little slow when you first start it as it indexes your files for searching and generates thumbnails and video previews.  This latter bit enables one of the more interesting features, search.

You can search all of your Pogoplug drives relatively easily from the WebUI, I haven’t toyed with this much yet but on the drive after you mount it the Pogoplug software creates a .ceid file that includes the name of the device and the version of the metadata, and a .cedata directory holding an SQLite 3 database file for indexed information and directories for the generated thumbnails and video previews.

The Pogoplug also ‘integrates’ with Facebook, MySpace and Twitter.  The Twitter support is definitely buggy, I was able to authenticate to it for one drive but not for another, and after signing out I have been unable to authenticate again.  Once setup you can ‘share’ a folder to these services and the unit will post updates whenever the folders are changed.  The update includes a (public) link to the folder’s contents.  Users can then download the data.  However the data is pushed directly from your Pogoplug so you must be connected via broadband.

On the hardware side inside the case there’s documented JTAG and Serial Port.  What does this mean?  Well if you’re asking then it won’t matter to you.    Basically it means that with a JTAG dongle and a 3.3v FTDI to USB serial adapter you have a $130 ARM9 dev kit, not bad at all.

I haven’t (yet) opened mine…I may yet buy another to do just that.  The Orion/Feroceon has a SATA controller that’s turned off (and quite possibly not even pinned out) on this board.  It also has a second Gig-E MAC thats likewise not available.  The CloudEngines/Pogoplug Engineers do read their forums, and seem to be (refreshingly!) helpful to those people who ask specific questions about the hardware and essentially how to use it as a dev platform, de-brick it, etc.  Being so helpful as to even link to DigiKey Parts for the mating connectors to the JTAG/Serial ports.

There are some chinks.  It has a NEON PINK “foot”.  The device has no ‘shutdown’ command (either via WebUI or SSH, or anything) so you can’t cleanly shut down the unit, you have to manually eject via the WebUI.  Unplugging the device, at least with HFS+, can cause the filesystem to come up ReadOnly with no way to fix it from the Pogoplug short of ejecting the device and manually running the included chkhfs utility.  Even that may not work since the utility is based off hfsprogs, which aren’t very good.  It will claim errors, not tell you what they are, and refuse to fix them.  Morale, either eject before you unplug the Pogoplug, or use other, better supported, filesystems.  I also have no clue what happens to the device when it loses Internet connectivity.  It may turn into a pretty pink and white brick, I don’t know, that’s one of the only things that actually worry me so far. I’ll be toying with that in coming days and make an updated post time permitting.

What problem is this that I am speaking of? LACK OF INDEXES. Seriously. If you have a sessions table, and you’re searching for old sessions to expire YOU NEED AN INDEX ON THE TIME COLUMN.

Another fun one is seeing ‘delete’ functions that don’t take care of the extra data referencing the main table, this is largely MySQL’s fault for not having foreign key support until lately. It really is the database’ job to deal with that.

1.8 million sessions occupying about 500MB or so, bringing a server down to it’s knees. CREATE INDEX sessions_last_active_idx ON sessions (last_active); and suddenly everything is 100% kosher.

What I am saying is, either find someone who has a damned clue to design your DB Schema INCLUDING INDEXES with you, and review your EVERY SELECT, DELETE AND UPDATE, or learn how to do it yourself, or don’t write the app.

[root@vps1 private]# vztop

vztop - 18:02:45 up 4 days, 17:30,  2 users,  load average: 1.29, 1.36, 1.28
Tasks: 372 total,   2 running, 370 sleeping,   0 stopped,   0 zombie
Cpu(s): 13.3% us,  1.1% sy,  0.0% ni, 85.2% id,  0.1% wa,  0.1% hi,  0.2% si
Mem:  65881472k total, 22104056k used, 43777416k free,  1624648k buffers
Swap:  4192924k total,        0k used,  4192924k free, 17053276k cached

 CTID   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    0 27254 root      25   0  308m 299m 4504 R  100  0.5  46:03.17 yum
    0 31593 root      15   0 89524 4672 2568 S    8  0.0   0:35.59 sshd

Seriously guys? 299MB?  SERIOUSLY?

Not that it really matters on this machine.  For those of you having trouble with the math, thats 64GB w/ around 42GB *free*, yes, same machine from the Serverpr0n article earlier.  It’s still annoying.  And, 100% CPU.  Doing WHAT?!  Hell if I know.  YES THATS 45 MINUTES!  At 2GHZ!

I’ll try to keep this updated.

But Mr. SysOp, how lame are you that it’s going to take months to setup a machine?!  I have a day job folks.  A BUSY day job.  Doing SA work in my VERY limited spare time is hard, and not very attractive.  So yes it could take a while.  I might throw up Nexenta or the free ESX and find out I hate it.  I want to have time to play with those options before I commit to moving live services to it.  Once I do that I’ll be totally stuck with my choice, warts and all.

Output from one of our mail processing frontends.  The script runs mailq and collates the output, sorted by recipient domain.  Redacted of course to protect the people with the bad idea.

mfe3:~# /opt/bin/top-mailq-to-domains
   2947 *REDACTED*.com
    158 *REDACTED*.co.uk
    117 *REDACTED*.org
    116 mailbox.com

This is why sending yourself an email for every error on your site is A BAD IDEA.  Make a log.  Process the log.

Kthnxbai.

Please, don’t start on NFSv4.  Linux can’t even support NFSv3 hardly, and the 2.6 kernel is worse than 2.4.  Atleast 2.4 doesn’t completely lose track of locks and locking the way 2.6 clients do.  They completely forget about cleaning up after NFS locks somewhat randomly.  Don’t believe me?

web4:~# uname -an
Linux web4 2.6.18-5-amd64 #1 SMP Tue Oct 2 20:37:02 UTC 2007 x86_64 GNU/Linux
web4:~# cat /proc/locks
1: FLOCK  ADVISORY  WRITE 25347 00:33:6591864 0 EOF
2: POSIX  ADVISORY  READ  21417 00:23:3323129 0 426
3: POSIX  ADVISORY  READ  12981 00:23:3323129 0 426
4: POSIX  ADVISORY  READ  14067 00:23:3323129 0 426
5: POSIX  ADVISORY  WRITE 15061 08:09:48938 0 9223372036854775806
6: POSIX  ADVISORY  WRITE 15061 08:09:48936 0 9223372036854775806
7: POSIX  ADVISORY  WRITE 15061 08:09:48934 0 9223372036854775806
8: POSIX  ADVISORY  WRITE 15061 08:09:48932 0 9223372036854775806
9: POSIX  ADVISORY  WRITE 15061 08:09:48929 0 9223372036854775806
10: POSIX  ADVISORY  WRITE 15061 08:09:48895 0 9223372036854775806
11: POSIX  ADVISORY  WRITE 15061 08:09:48893 0 9223372036854775806
12: FLOCK  ADVISORY  WRITE 14354 08:08:402423 0 EOF
13: FLOCK  ADVISORY  WRITE 3263 08:08:96619 0 EOF
14: POSIX  ADVISORY  WRITE 2628 08:08:322011 1024 2047
web4:~# stat /proc/14354
stat: cannot stat `/proc/14354': No such file or directory
web4:~# cat /proc/locks
1: FLOCK  ADVISORY  WRITE 25464 00:24:1986016 0 EOF
2: FLOCK  ADVISORY  WRITE 25463 00:23:106101 0 EOF
3: POSIX  ADVISORY  READ  21417 00:23:3323129 0 426
4: POSIX  ADVISORY  READ  12981 00:23:3323129 0 426
5: POSIX  ADVISORY  READ  14067 00:23:3323129 0 426
6: POSIX  ADVISORY  WRITE 15061 08:09:48938 0 9223372036854775806
7: POSIX  ADVISORY  WRITE 15061 08:09:48936 0 9223372036854775806
8: POSIX  ADVISORY  WRITE 15061 08:09:48934 0 9223372036854775806
9: POSIX  ADVISORY  WRITE 15061 08:09:48932 0 9223372036854775806
10: POSIX  ADVISORY  WRITE 15061 08:09:48929 0 9223372036854775806
11: POSIX  ADVISORY  WRITE 15061 08:09:48895 0 9223372036854775806
12: POSIX  ADVISORY  WRITE 15061 08:09:48893 0 9223372036854775806
13: FLOCK  ADVISORY  WRITE 14354 08:08:402423 0 EOF
14: FLOCK  ADVISORY  WRITE 3263 08:08:96619 0 EOF
15: POSIX  ADVISORY  WRITE 2628 08:08:322011 1024 2047
web4:~# 

Yeah, PID 14354 is LONG GONE but still holding a lock. Yes I know they’re advisory, but 2.4 doesn’t have a problem cleaning up and getting rid of locks when processes die.  Back with 2.6.8 we tried to convert our NFS Server from 2.4->2.6, and that was a complete disaster.  The 2.6.8 NFS Server would just randomly spit out permission denied errors to clients under load.  We never tried again.  NFS isn’t where it’s at for us.  We’re moving to AFS.  AFS isn’t without pitfalls either mind you.  And we’ll have to do a LOT of work to get there including some custom code, most notably to support device inodes.  There’s also a pretty bad bug in the Linux AFS client that we’re hoping to see fixed soon.