Open-source storage that doesn't suck? Our man tries to break TrueNAS
FreeNAS's 2003-looking grown-up sibling examined
Review Data storage is difficult, and ZFS-based storage doubly so. There's a lot of money to be made if you can do storage right, so it's uncommon to see a storage company with an open-source model deliver storage that doesn't suck.
I looked at TrueNAS from iXsystems, which, importantly, targets the SMB and midmarket with something that is theoretically more resilient than a Synology. That's really odd. Not a lot of companies do that, so it intrigued me.
I'd also had a few interesting conversations with some Reg readers about the dearth of storage offerings for the "small, but not Synology small" business space.
So, iXsystems. The firm is best known for the open source version of their storage distribution, FreeNAS. FreeNAS is powerful, feature-rich and one of the most popular open source storage solutions currently available. TrueNAS is essentially the long term support (LTS) version of FreeNAS.
To become TrueNAS, FreeNAS's code is feature-frozen and tested rigorously. Bleeding-edge development continues with FreeNAS, and FreeNAS comes with far fewer guarantees than does TrueNAS.
So far, so normal. Commercial open source often follows such a model. Especially for distributions as mature as FreeNAS.
Of course, storage is different. If your website's CMS crashes every now and again, it isn't the end of the world. Storage has to Just Frakking Work. No ifs, ands or buts.
CPU in the TrueNAS UI
For this reason, TrueNAS ships only on specific appliances. The hardware/software combo is tested to the point that iXsystems feels they can sell these units to enterprises without getting sued into a crater because they missed some critical bug.
Never underestimate how seriously storage companies take this.
I have crashed EMC and NetApp units, caused data loss events in $deity only knows how many commercial hyperconvergence solutions and managed to bring my Tintri to its knees. Pushing storage beyond its design capabilities in the lab is fun! I get to talk about it at parties and have product managers freak out until they realise I broke their baby on purpose, and not in the course of normal operations.
Of course, real-life storage matters too. In the real world, most of my clients don't run EMC or Tintri or NetApp. Some use a storage appliance, but rather too many of them run DIY whitebox open source storage gear. After a couple of decades working with these sorts of systems, it's safe to say that I'm pretty familiar with the intricacies of storage.
The result is that open source storage and I are not good friends.
The difficulties around making good storage led me to approach the TrueNAS review with a healthy dose of skepticism. I know just how easy it is to make open source storage go horribly, horribly wrong. I pulled no punches during the review process. I found rough edges and errors, but overall came away far more impressed with TrueNAS than I thought I would be.
iXsystems provided a Z20 hybrid storage array. The Z20 is a dual-controller, SAS-based, high-availability, hybrid storage array. The testing unit came with a 2x 10GbE NIC per controller and retails around US$24k. The unit shipped with 10x 300GB 10k RPM magnetic hard drives, an 8GB ZIL SSD and a 200GB L2ARC SSD. 50GiB of RAM was dedicated to the ARC by the system's autotune feature.
The performance of the unit was within the range of acceptability for the hardware provided, but wasn't stellar. (There are other models available with better performance, but weren't reviewed.) Of course, judging performance on a ZFS system is like making Carl Sagan an Apple pie from scratch: first you must create the universe. When properly tuned for the relevant workload, ZFS will absolutely sing. The flip side is that ZFS cannot be made into a "one size fits all" storage solution.
In ZFS, the ZIL is a write cache and ARC/L2ARC are read caches. ZIL is usually sized based on how fast your network goes. The faster the network, the larger the ZIL. ARC is usually a RAM cache of the index of things that reside in the L2ARC SSD cache. The bigger the L2ARC you have, the more RAM you need for ARC. The bigger the L2ARC, the less your reads need to come from disk.
The amount of RAM dedicated to ARC is manually adjustable, allowing you to make the ARC bigger than required for the L2ARC index – and thus letting you cache data in RAM – but this is one way to easily wreck the system if you don't know what you're doing. If you aren't a ZFS guru, leave the autotune settings alone and pretend they aren't there.
There are many different ways to make a hybrid storage solution – the above is how ZFS (and hence TrueNAS) does it. The results are decidedly mixed.
The TrueNAS as provided was excellent at iSCSI. If what you want is a solid, decent-performing iSCSI solution, what I was provided out of the box does a great job, especially for read dominant workloads. Throw a typical OLTP workload at it (4K, 70% read, 30% write, full random) over iSCSI and it delivers a rock solid 150MiB/sec.
At $24k USD for a proper HA array, it's hard to complain about that performance. The TrueNAS, however, does a lot more than iSCSI, and here is where tuning matters.
In addition to block storage (iSCSI or FibreChannel), the TrueNAS offers file storage via SMB (which their UI incorrectly calls CIFS), NFS, FTP and AFP. Large file transfers to the TrueNAS (say a few gigs per file) perform more or less like iSCSI. Throw 10 million itty bitty files at it, however, and the need to twiddle the tunables becomes evident.
When writing, accessing or deleting large numbers of files from the TrueNAS, the ARC and L2ARC were essentially unused. Not only were the hit ratios low, but for reasons I don't fully understand the system didn't actually grow the L2ARC much. Not only were the millions of files themselves not cached, but – more critically – the file system index didn't seem to be cached.
In its shipped configuration, allowing autotune to configure ZFS, the TrueNAS array could only create or delete around 350 files/sec. This resulted in a maximum throughput of around 25MiB/sec for small files.
At first glance, the TrueNAS UI looks dated. There are icons that look like things. The colour palette contains more than just pastels. There are rounded corners. In short, iXsystems hasn't had time to update to the HTML 5 UI that every other storage company has slavishly copied. It's a little bit less cookie-cutter 2015 and a lot more 2003.
I am entirely okay with that.
The UI contains truly excellent mouseover tooltips that provide detailed information and rationale for almost every setting. An experienced sysadmin will be able to navigate the TrueNAS UI with ease. An experienced storage admin who knows what all the terms mean won't have to refer to a wiki or the more traditional help manual, but the same can't be said for the uninitiated.
Don't take your standard Windows Server admin or VMware vSphere admin and throw them at TrueNAS without time to learn, or having iXsystems tune it for you. The TrueNAS UI is written in storage-ese and if you don't grok the lingo you're going to need reference material. Fortunately, TrueNAS comes with an exhaustive and comprehensive guide, so there is nothing to fear except lack of time. And clowns. (Always fear clowns.)
Or a wizard. Yes, TrueNAS has wizards!
If you are a fan of the "next, next, next, next, done" school of systems administration then TrueNAS has a button just for you. Be prepared to connect your TrueNAS up to Active Directory, as that's the first thing on the wizard's list. Somehow, however, I expect that if you're investigating the wizard, getting TrueNAS working in your Windows environment quickly and painlessly is probably your goal.
You can tweak any variable in the UI. You can create shares, manage LUNs, add users and tinker with ACLs. There is a reasonable amount of reporting available in the UI and access to the vSphere plugin and settings for the VMware admins.
Services (such as SMB, NFS, etc) are controlled by a simple on/off toggle. There is also a shell available through the GUI for those who prefer to use the command line.
Breaking the TrueNAS
Storage reviews never quite seem done unless I break it somehow. In the case of the TrueNAS the Achilles' heel is the network. If, for whatever reason, your network cables are not connected right or you change something related to the network, the whole thing goes sideways.
This normally isn't a problem: plug things back in and otherwise make sure it is connected as per the manual, and workloads will be served. Where things go really squirrely is if you are in the middle of performing an update and, for example, your bearded dragon decides she's a chaos monkey and unplugs the fibre optics for the backup node.
This results in one node being a different version than the other and the update process being royally confused. That said, I was able to resolve this without the vendor's help by simply turning both nodes off, rebooting and rerunning the update.
With that one exception, the TrueNAS otherwise survived the sorts of things you expect it to survive. Failed disk, failed power, controller failover, switch outage and so forth.
After a lot of testing, I'd trust my data to the TrueNAS. I am convinced that it will ensure the availability of my data to within any reasonable test, and do so as a high availability solution. That's more than I can say for a lot of storage out there.
I'm not so sure about performance. I've been spoiled by the Tintri (which is way more expensive than the TrueNAS), my hybrid hyperconverged clusters and my all flash Synology. Anything as heavily reliant on magnetic media just feels frustrating by comparison.
That said, iXSystems makes an all-flash system, the Z50 TrueFlash. The list price is $Call, but the nudge-nudge, wink, wink entry price on it starts at under US$30k. That has me interested.
iXsystems makes good arrays for the SMB and midmarket. They're probably worth considering for point solutions in the enterprise. The HA TrueNAS units such as the one discussed above should be a serious consideration for any organisation looking for storage more reliable than a Synology, but aren't yet ready to go knocking on EMC or NetApp's door.
The larger players have clustering. The ability to scale up with a single management plane and a single namespace is a distinct advantage for them. For many configurations, however, iXsystems is significantly less expensive. Enough to make juggling multiple arrays worth it, for some, at least.
What's astonishing, however, is that this is where the debate is at. The TrueNAS is clearly storage that just works. It's passed the reliability barrier and we're into debates about price, performance and features. Debates that, quite frankly, iXsystems is fully aware of, and they are aggressively pursuing R&D to resolve.
iXsystems produce a storage array that is decent enough to entice away some existing users of the likes of EMC, NetApp, Dell or HP. Honestly, that's not something I thought possible going into this review. It's a nice surprise. ®