We recently transitioned from a proprietary NAS storage solution (NetApp) to an open, linux-based solution. Here I explain what brought us to that decision and an overview of the new solution.
When we originally architected our web infrastructure in 2006, we spec-ed a NetApp filer for the core of our system. We bought a FAS 3020c high-availability cluster with about 1 TB of usable space. We wanted to do offsite backups , so we also purchased a FAS270 for these backups. (Total cost: approximately 130,000 USD) We used NetApp's SnapMirror to perform the remote backups. All in all, we were very pleased with the system.
As the months went by, we found that our archives were growing faster than expected, and we were on pace to outgrow the storage sometime in early 2009. We requested a quote for an additional TB of storage for the 3020c and the 270. That quote came in around 40,000 USD.
That was outrageous to me. Considering you can get a 300GB 15K SAS drive for under $350 today, you could get 12 of them for around $4200, and at RAID5 with a spare drive, you could have around 2.7TB. Double it for the two systems, and the raw disk cost would be under $10K. Sure, there's the cost of the enclosure, and some markup, but I can't really explain the $30K differential.
We talked to Penguin Computing, our preferred Linux hardware vendor, about building an open-source NFS server. We wanted a high-performance, highly available system for our production environment, and a lower cost system for our development environment. They quoted us about $60K for a pair of Relion 2650s with a Xyratex F5412E drive enclosure and expansion chassis (total usable storage of 3TB). The quote also included a Relion 2612 for offsite backup (4TB storage), and some professional services to help us configure the system and to train our team on administration.
The Xyratex has dual controllers, and the two Relions are cross-connected to the controllers via Fibre Channel. We use RedHat Enterprise Linux 5 with RedHat Cluster to provide high availability for the NFS service.
All in all, this system performs as advertised, and it is pretty straightforward to administer. I really appreciate being able to use familiar tools like ganglia and nagios to monitor the system. Where there are problems, performing diagnostics is much easier than with a proprietary system.
That said, we do miss some things that we had in our NetApp servers:
- snapshots - if you've never experienced the convenience of NetApp's snapshots, you're missing out. Depending on your situation, you may not *need* them, but they can really be convenient.
- snapmirror - related to snapshots, snapmirror sends changed blocks over the network to another NetApp; it's about as efficient a remote backup as you could do. But rsync does a pretty decent job, too, even if it can be a bit rough on your system resources for a large file system.
Part 3: the gotchas we encountered when deploying this system
Comments