The Channel logo


Chris Mellor

Contact Mail Follow Twitter RSS feed
Chris Mellor covers storage and allied technology areas for The Register. After experience working for DEC, Unisys and SCO, he became an IT journalist writing for a variety of print publications. He edited the UK's first storage print magazine and then moved into the online world writing for IDG's Techworld, then started up the Blocks & Files blog, which was bought by El Reg.

He has written many sportscar buying guides, a few mountaineering guides and drives a car that's faster than he is.

By | Chris Mellor 5th November 2009 13:11

Could a hard drive dedupe data?

Manufacturers start looking for the next big thing

Comment Hitachi GST president Steve Milligan says one of the drivers affecting the hard drive industry is the need for efficient storage with technologies like virtualisation and deduplication. What is he on about?

He presented at a recent Needham conference for HDD investors and said that the storage market was driven by three things. First was the need to store more data, which he saw continuing if not accelerating. Then he said mobility had risen strongly and was set to continue as people needed to store more data in mobile devices. Thirdly, he mentioned storage efficiency and virtualisation and deduplication.

Why would a hard drive manufacturer be bothered about these two items? All HGST and the other drive suppliers do is build a raw byte storing device and present it to drive array and compute device manufacturers as a component. They add virtualisation functions and may build the drives into deduplicating storage arrays. Deduplication simply isn't a concern of drive manufacturers, or is it?

A couple of years ago, data encryption was performed by in-band devices, such as NetApp's Decru boxes, applications running in a server, or tape drives that encrypted backup data in case the tape cartridges got lost or stolen.

As encryption hardware was then small enough to shove in a tape drive controller, it was also small enough to put inside a hard drive's case. So Seagate and others came out with self-contained hard drive encryption products.

Could Milligan be thinking that where an encryption ASIC can go then a deduplication ASIC could follow?

You would need to do HDD-level dedupe inside the HDD case, where space is restricted, and you'd need to provide CPU cycles because the HDD controller CPU is already busy doing standard HDD I/O-type tasks. Suppose there were an ASIC, an application-specific processor you could use?

Exar has bought Hifn for its hardware encryption, compression and deduplication products. Here is a description of a Hifn chip:

The 9725 provides power efficient hardware offload capabilities for data compression, encryption and deduplication for enterprise class storage applications. OEMs now have the ability to integrate these capabilities into platforms that require non-standard PCIe form factors or need to be embedded directly on a system’s motherboard.

Solid State Drive start-up Whiptail is adding Hifn dedupe to its RaceRunner SSDs, so the idea isn't entirely half-arsed.

As storage array futures look to be heading to two main disk tiers, one for performance (FC, SAS, SSD) and one for capacity (SATA) then taking a 2TB SATA drive used for virtual tape library and disk-to-disk backup applications and increasing its capacity by 20, 30 or possibly more per cent by deduplicating data stored on it, with no decrease in I/O performance, might look interesting to a hard drive manufacturer looking to gain an edge and justify a pricing differential.

By the way, for a vendor of sealed disk drive canisters, aka Integrated Storage Elements (ISEs), like Xiotech, adding this functionality at a canister level might be worth considering.

Of course Milligan could just have been talking generally. But it's enticing isn't it? Once HDDs didn't encrypt their data. Now they do because there is enough HDD controller board real estate to permit the addition of encryption hardware. What else might be added, what other functions might be considered by the HDD suppliers? Could on-board - or in-canister - HDD deduplication be a realistic prospect over the next few years? ®

comment icon Read 17 comments on this article alert Send corrections


Frank Jennings

What do you do? Use manual typwriters or live in a Scottish croft? Our man advises
A rusty petrol pump at an abandoned gas station. Pic by Silvia B. Jakiello via shutterstock

Trevor Pott

Among other things, Active Directory needs an overhaul
Baby looks taken aback/shocked/affronted. Photo by Shutterstock

Kat Hall

Plans for 2 million FTTP connections in next four years 'not enough'
Microsoft CEO Satya Nadella


League of gentlemen poster - Tubbs and Edward at the local shop. Copyright BBC
One reselling man tells his tale of woe