On deduplication and Data Domain appliances

One subject I keep hearing about is deduplication. The idea being that you save a ton of space since a lot of your computers have identical data.
One way to do it is with an appliance-based solution such as Data Domain. Effectively, they put a little server and a cheap-but-not-cheerful, non-expandable 6TB RAID together, then charge a lot for it, claiming it can hold 90TB or whatever. Use many of them to scale.

The technology chops up incoming files into pieces. Then, the server calculates a unique numeric ID using a hash algorithm.

The ID is then associated with the block and both are stored.

If the ID of another block matches one already stored, the new block is NOT stored, but it’s ID is, as is the association with the rest of the blocks in the file (so that deleting a file won’t adversely affect common blocks with other fles).

This is what allows dedup technologies to store a lot of data.

Now, why it depends how much you can store:

If you’re backing up many different unique files (like images), there will be almost no similarity, so everything will be backed up.
If you’re backing up 1000 identical windows servers (including the windows directory) then there WILL be a lot of similarity, and great efficiencies.

Now the drawbacks (and why I never bought it):

The thing relies on a weak server and a small database. As you’re backing up more and more, there will be millions (maybe billions) of IDs in the database (remember, a single file may have multiple IDs).

Imagine you have 2 billion entries.

Imagine you’re trying to back up someone’s 1GB PST, or other large file, that stays mostly the same over time (ideal dedup scenario). The file gets chopped up in, say, 100 blocks.

Each block has it’s ID calculated (CPU-intensive).

Then, EACH ID has to be compared with the ENTIRE database to determine whether there’s a match or not.

This can take a while, depending on what search/sort/store algorithms they use.

I asked data domain about this and all they kept telling me was “try it, we can’t predict your performance”. I asked them whether they had even tested the box to see what the limits were, and they hadn’t. Hmmm.

I did find out that, at best, the thing works at 50MB/s (slower than an LTO3 tape drive), unless you use tons of them.

Now, imagine you’re trying to RECOVER your 1GB PST.

Say you try to recover from a “full” backup on the data domain, but that file has been living in it for a year, with the new blocks being added to it.

When requesting the file, the data domain box has to synthesize the file (remember, even the “full” doesn’t include the whole file). It will read the IDs needed to recreate it and put the blocks together so it can present the final file, as it should have looked.

This is CPU- and disk-intensive. Takes a while.

The whole point of doing backups to disk is to back up and restore faster and more reliably. If you’re slowing things down in order to compress your disk as much as possible, you’re doing yourself a disservice.

Don’t get me wrong, dedup tech has it’s place, but I just don’t like the appliance model for performance and scalability reasons.
EMC just purchased Avamar, a dedup company that does the exact same thing but lets you install the software on whatever you want.

There are also Asigra and Evault, both great backup/dedup products that can be installed on ANY server and work with ANY disk, not just the el cheapo quasi-JBOD data domain sells.

So, you can leverage your investment in disk and load the software of a beefy box that will actually work properly.

Another tack would be to use virtual tape – doesn’t do dedup (yet, but it will since EMC bought Avamar and Adic, now Quantum, also acquired another dedup company and will put the stuff in their VTL, you can get the best of both worlds) but it does compression just like real tape.

Plus, even the cheapest EMC virtual tape box works at over 300MB/s.

I sort of detest the “drop at the customer site” model data domain (and a bunch of the smaller storage vendors) use. They expect you to put the box in and if it works OK to make it easier to keep it than send it back.

Most people will keep the first thing they try (unless it fails horrifically), since they don’t want to go through the trouble of testing 5 different products (unless we’re talking about huge companies that have dedicated testing staff).

Let me know what you think…

D

12 Replies to “On deduplication and Data Domain appliances”

  1. I’d suggest you take another look and read up a bit more on their technology, or talk to one of their technical folks. data domain’s “database” is far from weak, and not a separate database but a filesystem based approach (though one can argue a filesystem is a database). Respectfully, I don’t think you’re familiar enough with their technology to critique it, and are making some assumptions that don’t hold true in what they ship. Data domain holds patents around their dedeplucation technology, and that technology allows them to break the stream into variable size segments – and leverage CPU’s and RAM primarily so as not to have to go relatively slow disk to see if they’ve already seen a segment.

    Even on my DD200’s (now EOL, their 1st gen box). I saw more than 60MB/sec sustained, and often more (the DD200 wasn’t necessarily the bottleneck). And while the CPU’s get busy, disks don’t appear to get that busy in normal operations. You can “imagine” as you say. I’d suggest you not “imagine”, but experience it to see many assumptions you hold dissolve (around restore times, search algorithms) in reality. Then you can ask them about why your assumptions don’t necessarily hold true.

    Read performance in my experience is within a couple MB/sec of write performance. You’d find if you filled up a box (as I have), performance does not degrade (as you imagined) in a noticeable way when close to full for either reads or writes. I’d also add that their foiundation and mindset center around data protection.

    Personally I’d take an appliance over software any day (avoiding the related headaches around patch interrelations, interactions, and lots of clients to manage) but some people like to do that to stay busy. I’d rather be doing something more meaningful with my time, while my appliances work away.

  2. Thanks for your candor. However, I don’t think you got my point.

    I believe I do know how the boxes work, the algorithms are pretty simple. There aren’t too many ways to do this, BTW. You don’t need to go to disk, the best way is to keep as many hashes as possible in RAM. Almost everyone does variable block, too.

    I DID talk to their technical folks and I DID ask them how the box might behave over time given MY workload (at the time, billions of files, petabytes of storage, unsure of the commonality in the blocks).

    They never did testing on such a large scale, over years (emulated or not). They could never give me a straight answer on how it would deteriorate (or not) if I left stuff in there for a long time and tried to recover.

    They kept calling me, 3 times a week. “just try it”. I dislike guerilla sales. Most boxes will work OK for a while. My concern was more for the long-term and for high throughput for large files.

    60MB/s is nothing – one of my VTLs was giving me 400 and that was because my ISL’s were maxed out (long story). And 400MB/s was NOT ENOUGH. I bought 2 more VTLs, had tons of media servers, multiple fabrics, dozens of tape drives. Not a small environment.

    Maybe my needs were far in excess of what their technology could possibly provide.

    I can see it being OK for a small windows shop.

    However, just think of the math if you ARE loading the box with billions of files on a weekly basis. There ARE limits on their DB (they told me as much). There are limits on ANY DB.

    This is a pure mathematical problem.

    What was the workload you put on the thing, if you don’t mind me asking? How many systems? I had thousands of systems to back up.

    D

  3. How certain are you that only hashes are used in de-duplication?

    Mr. Monkey: You know very well that Avamar works perfectly in a vacuum; the real world, however, presents different challenges for the SW

    And, no, Diligent is a very bad idea.

  4. Hashes are the biggest part of it – if anyone knows different please post. Unlike other bloggers, I don’t censor people.

    Almost all technologies work well in a vacuum. If you have information as to why some technology is better than another please post why, in detail. We all can benefit from the collective wisdom.

    D

  5. You really do need to do a test for yourself. DataDomain is the first to tell you this by the way. You will find out that they are the best at what they do. There are always individuals who think they have this figured out but to that I say, “Put this to the real test in your environment.”

    When you have something real to say it will be interesting to read but your conjecture and diatribe is akin to heresay and therefore not very interesting or enlightening.

    Have a good one Dimitri

  6. Mjc0, (whatever your real name is 🙂

    ALL dedup technologies are the ULTIMATE “your mileage may vary” tools!

    Telling me I need to try before I buy is like telling me the sun is warm and bright.

    Also, “heresay” is telling me that Data Domain are the best at what they do. Your point has no more validity than my uninteresting and unenlightening diatribe and conjecture. WHY are they the best?

    If you read the entirety of my post, you’ll realize that my beef lies more with the inability (or lack of desire) of Data Domain employees to explain certain things in more detail, than with anything I have against their technology. I have no reason to have a beef with their technology, as I’ve never tried it.

    My point is that, in some cases, before even TRYING the technology, one needs to know certain things before spending the time testing the gear. For instance, how will the box deal with billions of small image files? Multi-TB databases? Certain dedup vendors will immediately admit performance will suck with very large database files.

    EMC, for instance, freely admits that the reason one needs multiple nodes with their Avamar product is to not lose performance (they even recommend that each node sees no more than 2TB of visible storage). So, as your capacity needs grow, you need to add CPU capacity, which makes sense.

    If you (or any Data Domain employee) are willing to shed some light, by all means write up a post.

    D

  7. Dear Data Domain Customer –

    The only reason Data Domain owns any patents is because they licensed them from RockSoft. RockSoft was acquired by ADIC, and ADIC was acquired by Quantum. Data Domain didn’t invent any of this technology. They just leveraged what was already out there. This is proven by the fact they were sued by Quantum (and DD lost). They got to keep their fork of the code once they paid Quantum for the technology.

    DD boxes work just fine in small sized businesses. If you are deploying these in an Enterprise environment where you are backing up 10’s or 100’s of TBs or even PBs, you would have serious scaling issues. You would have to sprinkle numerous DD units out in the environment, and this is always a challenge technically as well as financially.

    De-Dup requirements vary based upon need. Source vs target, in-band vs out-of-band. One size does not fit all; and unfortunately, DD offers only one method.

  8. Billions of small image files don’t deduplicate because JPEG is already compressed. I don’t care which dedupe provider you use the same will be true. You cannot compress compressed data.

    Multi-TB databases: With a DD690 we can achieve 330 MB/sec througput, ~1.1 TB/hour using NFS and multiple streams of data. Doing a straight RMAN dump to the NFS mount point would be what I would recommend. How big of a DB do you have? 5 TB? How fast is the server that is pushing the data? What is its Front-side bus? PCI-bus? For Oracle with enough retention we see on average 15x compression. A lot of customers find that their bottlenecks in their backups don’t reside on the storage destination (be it tape or otherwise), but with the origin or the backup server(s) themselves.

    You originally wrote the article when the models peaked out at a DD560 (max throughput 100 MB/s aggregate), most of what you said then may have been true.

    Data Domain doesn’t use a JBOD. The 3u 500 series boxes ship with 15 drives, 1 of which is a hot spare, the others are set up in a RAID-6 configuration allowing for a dual drive failure. The JBOD statement should be stricken from the original post as it is an utter fabrication.

    I also have a problem with your scenario of restoring a 1 GB PST. As a former customer and current employee of Data Domain I take exception to it. As a former customer we conducted 4 DR Tests using DDRs (DD430’s and DD560’s) We recovered 700 GB Exchange DBs in under 5 hours. We also recovered (5) 200 GB oracle DBs at the same time in 2-3 hours on average. Most of that time was spent getting the RMAN configured properly to do the restore. Your statement didn’t hold water then, and doesn’t now.

    An LTO-3 is rated at higher than 50 MB/s. How do you achieve that backup throughput in a backup environment? You have to use multi-plexing. What happens to multi-plexed restores? The fact is they stink. Fast in……slow out. One way or another tape drives are slow. The other thing with tape is if you don’t constantly stream the minimum rated throughput, it shoe-shines. Not good for the media, the throughput, or the drives themselves.

    Every year that Intel/AMD releases new CPUs with more cores the faster we get. How fast is disk access increasing? Not very, look up Moore’s Law on Wikipedia.

    Avamar is agent based. By show of hands, who loves to maintain agents on a server farm? Who loves to schedule upgrades and downtime across all of your servers? Who wants to offload CPU/RAM work onto your backup clients so that they can figure out what is duplicate data and what isn’t?

    If you have a lot of really small remote offices, Avamar or NetBackup Puredisk is a viable solution, in all seriousness. Don’t use them in your main data center though.

    Data Domain fits seamlessly into any existing backup environment. No need to put agents anywhere. It just takes the place of any existing D2D or backup to tape solution. Attach it to a private VLAN with your backup servers and dump data to it. Or you could use a direct attach method to the backup server.

    Want DR Replication? What kind of an effect would 20x compression have on WAN bandwidth utilization for DR Replication? How about a 95-98% reduction in utilization. That allows many more company’s to achieve DR replication that never could before.

    To Mr. De-Duper:

    http://www.datadomain.com/products/faq-patents.html

    If RockSoft’s dedupe algorithms are so great why is QTM’s stock $1.85? Why aren’t they the front-runner in deduplication

  9. I heard about not bad application-recover pst, It can recovering lost Microsoft Outlook data is a sequence of operations involving scanning, identifying and saving such Microsoft Outlook items as messages, contacts, notes, reminders, journals, meetings, etc.

  10. I have done a research in Deduplication and i claim my Dedupe algo works best in industry
    Can anybody here help me in posting my research paper in deduplication
    I mean where to post my paper

    Thanks in advance

Leave a comment for posterity...