I’ve known about IBRIX for a while, but it was refreshing to talk to a decent techie that knew the product. They have improved it a lot over the past year.
For the uninitiated, IBRIX can be either
- A network-based filesystem using the IBRIX client and protocol
- Also accessible using NFS or CIFS
- SAN-based parallel filesystem
The product’s claim to fame is it’s scalability and performance (realized by adding extra nodes “hot”). Their most famous client is probably Pixar, they replaced a ton of NetApp boxes with an IBRIX cluster and realized huge performance benefits and vastly reduced costs. I always liked cool filesystem technologies and this definitely falls under the realm of “cool”. Some highlights based on notes I took on my Blackberry during the session and questions I asked:
- No limits on filesystem size (they have deployed single namespace filesystems several PB in size).
- 300mb/s read, 200mb/s write on small box per node. Bigger boxes can do 1.2GB/s per node, of course your storage needs to be able to keep up.
- No limit on the number of nodes.
- Automatic rebalancing of data over time. When you add new disk you rebalance to keep things humming.
- Dedicated ibrix backup node, works with 3rd party backup SW, can have many backup servers for backup speed.
- Has snaps now (global), this was a failing of the product before since it was lacking snapshots.
- No real limit on the number of files per FS.
- Biggest file size they have tested on production is an 8TB file, no software limit.
- Nodes use FC to access storage, clients use Ethernet.
- Client on Windows or Linux, otherwise general NFS and CIFS. Client is fastest.
- Your prod servers can be the ibrix nodes but very compute-intensive. They recommend the client (IP-based, bonded). or get an 8-core box.
- There is no single lock manager – this is the coolest thing. There is global metadata and global locking, all nodes participate equally.
- How are node failures handled? All nodes interchangeable. All see same storage. Storage allocated to remaining servers if you lose a node.
Can lose all but 1 server. - Back-end storage size per node? Unlimited.
- Multipathing per node? Powerpath works. Can do bonded GigE up to 8 ports per.
- How are files allocated? The file inode contains the info concerning which node it needs to go to. Round-robin allocation or preferred servers per file type. Also if server over 50% full then it’s skipped.
- All volumes accessible by all nodes.
- Can stripe huge files across many nodes.
I’m stoked! I can think of so many uses for this product:
- Data mining
- Digital media
- Oil and gas
- Backups
D