Planning for my ZFS fileserver

I’m looking to consolidate my storage. As seen is my previous post machine setup and disk/net benchmarking 2009-04-16 my disks are spread out a lot between my machines. There are several issues that I would like to address and improve upon: speed, data security, backup, maintenance, locating data and scaling.

I’ve decided to build a dedicated fileserver. Its name shall be… hmm… FILEZ. Afterwards I’ll move my stuff around and somehow get rid of PLUMBUM or GATES.

This is what my intranet looks like at the moment:


(Yes, it’s an actual old-style drawing, on real paper!)

I’ll be running OpenSolaris and ZFS.  ZFS because it’s the most secure filesystem in existance, and also the fastest, most flexible and most scalable filesystem in existance. OpenSolaris because it’s the “mother” operating system of ZFS.

Here’s my chassis:


Can you dig it? CAN YOU DIG IT?  With 6 internal disks and 6 of those 2×5.25–>3×3.5 drive bay converters and two 3×5.25–>5×3.5 drive bay converters you could fit 34 (!!)  disks in there without having any loose disks dangling in their molex cables inside the chassi. But I won’t be doing that, me thinks overheating drives would be a problem. Anyway, this should take care of my storage needs for a looooooong time. The problem isn’t the space for disks but the SATA connections. ZFS isn’t a clustered filesystem, you need all the disks connected to the same machine. A standard server mobo has six SATA ports, with a extension card like SuperMicro’s AOC-SAT2-MV8 (for PCI-X = 1.5 GB/s) or the AOC-SASLP-MV8 (for PCI-E = 500 MB/s) you get 8 more SATA ports… if you have those slots on the mobo. If you’ve just got PCI slots there are cards with 4 SATA ports, but PCI is 133 MB/s (split between four disks) so the performance will suffer.

I’m looking at mobo’s from Intel and SuperMicro, must have one, preferably two, PCI-X slots and ECC memory and as many SATA ports as possible. The X7SBA is an option, S3210SHLX another (but where’s its page at I’ll get any cheap CPU that fits the mobo and 2 x 2GB of dual-channel RAM. I will need a really good PSU for the mobo with lots of SATA connectors, not sure which one yet, possibly one more PSU in there to manage the power for all the disks. Worst case scenario: have another PSU outside the case, ugh…

The disks.. now there’s where the real fun starts! 🙂

This is my little TODO-list, including a lot of moving files around… No need for you to read this, it’s just to assist me…

  • Must move data to free the three minor disks in CONAN and the 160GB in GATES
  • I will by re-install TORPED2 on the two (striped) 160GB disks
  • Move database (MySQL) from PLUMBUM to CONAN (or FILEZ (AMP)?)
  • Shutdown PLUMBUM for good
  • Swap mobo + CPU (but keep memory!) of PLUMBUM and GATES.
  • Will try to setup GATES to boot off FILEZ, over ethernet.
  • If that doesn’t work I will move GATES to CONAN’s 200GB disk.
  • I will make a ZFS pool using 7 x 250GB disks in a raidz config (ZFS’s equivalent of RAID-5). The disks will come from the old TORPED2 (4 x 250GB), CONAN (1 x 250GB), GATES (2 x 320GB).
  • Will now have 1.5TB of usable space in FILEZ
  • Move all data from CONAN to FILEZ
  • Try to make CONAN boot off FILEZ. If that doesn’t work to my satisfaction then setup a mirror config using the internal RAID (?) and two disks (which ones?)
  • Make a new raidz (RAID-5) using 7 x 500GB disks. The four from old CONAN, the one from PLUMBUM and the two from PLUMBUM’s backup. Add the raidz to the pool to give me a total of 4.5 TB continous diskspace. Yes, indeed.
  • We have a risky situation now. No backup. TORPED2 and GATES do not have redundant storage but that’s fine as they’re backuping daily to FILEZ. A disk breaks, no problem at all. But no backup of NAS or FILEZ? Scary. Even though I know ZFS is safe as heck, it’s still scary…
  • Move all data from NAS to FILEZ.
  • Make a new raidz using the four 1TB disks from NAS. That’s 3 TB of usable space. Add to the ZFS pool to make a total of 7.5 TB continous usable diskspace.
  • Make something of old PLUMBUM (which now has the mobo from GATES)… If I got GATES and CONAN to boot off FILEZ I should have some spare disks to put in it.
  • Addendum: One of the 250GB disks in TORPED2’s current RAID-5 keeps failing. Intel’s controller seems unable to identify which one, hopefully ZFS will identify it so I can remove it, making the first vdev 6 disks, not 7.
  • Addendum 2: Disk 3 in my NAS is failing the daily SMART test, replace it with new disk and use the old one for backup.
  • Get one more disk for the (JBOD) external backup enclosure. Attach to GATES and config backup of vital data from FILEZ.
  • Setup off-site backup (a deal with a friend is in progress)
  • Sell the old NAS 🙂

Something like that. Should work. Phew.

That’s 18 disks plus 2 mirrored ones for the root (on separate pool). 20 disks. Will need the six internal SATA ports and two 8-port PCI-X cards. Or maybe go with 18 disks and one PCI-X card and use my old the Adaptec 2410SA 4-port PCI-card for now?

Possibly I’ll run AMP (LAMP without the L) on FILEZ. Would take a lot of stress off CONAN. I will still need to run Apache on CONAN though, to handle all my CGI’s. It will take some time to transfer to FILEZ as I will have to start handling directory/file permissions… I imagine I’ll be chmod’ing for days…

I am running tests of OpenSolaris/ZFS right now, using GATES and two old 120GB-disks. I am having problems sharing folders using SMB/CIFS, it’s very annoying. Hopefully it will work out in the end, will post details here if/when it does. (Edit 090503: I just solved this problem!)

I need to check for bottlenecks:
– cables,  they’re all CAT 5e or better?
– routers/switches etc: all up to speed?
– disks: all disks have similar speeds?
– memory, cpu, mb: should be no problem there
– using more than one ethernet port, would it make a difference?

What a beast of a blog post. Took half the day including research. Posting it now, possibly making changes to it later.

Comments are disabled as usual. Will create a forum thread for discussions regarding this project.