Storage vendor bloggers - losing data or losing the plot?
So does EMC's Center really lose data?
Comment Store data in EMC Centera and lose it - that's the claim of NetApp blogger Val Bercovici, and he cites a Symantec support document entitled 'Archiving items in Enterprise Vault may result in an extremely rare data loss situation' to prove it.
Bercovici, who works in the office of the CTO (Chief Technology Officer) at NetApp, blogged last week: "When Symantec publicly declares that EMC Centera (and only EMC Centera) is vulnerable to data loss, the entire industry - and most importantly archiving customers need to stand up and listen."
Indeed it should - except Bercovici was wrong. He was misled by a Symantec support document (tech-note) referring to its - Enterprise Vault product - which has now been updated.
The original version, up to January 26, stated there would be: "Potential data loss if archiving to Centera and running any of the known affected Enterprise Vault versions outlined below (see the What is Affected section)."
The revised version says: "There is a potential for data loss with the known affected Enterprise Vault versions outlined below when configured with an EMC Centera storage partition. The data loss occurs as part of the archiving process but before storing on Centera. (See the What is Affected section)."
Symantec spokesperson Cory Edwards said: "This tech note relates to a known issue with [Symantec's] Enterprise Vault, not EMC Centera This issue is limited to customers running Enterprise Vault and Centera together and is extremely rare. It is limited to a small percentage of archived information which is not converted into a text-based format (eg jpeg). Symantec is working with any EV/Centera customers who might have this issue.
"All customers running Enterprise Vault 8.0 or the latest service pack of older versions of Enterprise Vault will not experience any problems. The tech-note provides appropriate measures for customers to identify the issue and take corrective action."
Bercovici's blog declaring that the Centera users and supporters should consider the whistle blown sparked a vigorous, even aggressive discussion in its comments section. The back story here is that there are strong differences between NetApp and EMC concerning the two companies' approaches to storage, benchmarking and capacity usage. Both have strongly enthusiastic evangelising bloggers, with Chuck Hollis, Global Marketing CTO, being EMC's vigorous champion.
Bercovici took the stance that the Symantec tech-note was the nail Centera's coffin needed, and mentioned previous worries over the content-addressing algorithm used and the possibility of it not generating dependably unique strings to identify data absolutely, also a silent data corruption issue. He put it like this: "Evidence emerged which raised questions about the integrity of EMC's commitment to the integrity of their customers' archives... NetApp has never knowingly sacrificed the integrity of data archived by SnapLock and consequently enjoys a reputation of responsibility in this regard."
The comments took exception to this moral high-grounding and to the pinning of data loss fault to EMC when it was in fact down to a Symantec software bug.
Hollis commented: "Val, you don't seem to be able to write anything unless you're attempting to beat up someone, usually EMC. You've gotten in trouble before with this sort of behavior. Looks like you're heading down the exact same road again."
Bercovici replied to Hollis, saying: "It seems the only trouble I get into is with you and your selectively delicate sensibilities, forcing you to autocratically censor comments from me (with condescending reminders no less) which you don't want appearing on your blog. Regardless, I consider it a compliment either way. I look forward to observing how much EMC is willing to sacrifice your relationship with Symantec here (by throwing them under the bus) in order to cover up the truth about Centera's obvious data porosity."
He confirmed his belief about the supposed data porosity in a later comment: "I can assure you we did a 360 on this Symantec Alert with many experts on all sides (including EMC customers). All of us came to the same conclusion - under certain circumstances (such as recovery from disk or node failure) the Centera will drop archived objects without notification or the ability to retrieve them. This seems to be precisely the use-case highlighted by the Symantec Alert."
(They were wrong, all of them.)
Later in the comment sequence he states: "There is zero uncertainty or doubt in my claims. You see a few years ago I executed a script... and was able to repeatedly and intentionally make the Centera lose an object silently with no notification back to the script or user.
"I have spoken to dozens of EMC Centera partners and customers who have experienced the same symptoms due to various product deficiencies not necessarily related to hash algorithms."
Bercovici has updated his blog entry with score-throughs thus: "
In case you're wondering, this is NOT an issue with Symantec software. The root cause is the complex EMC Centera API and brittle internal Centera architecture which Symantec Enterprise Vault and other similar applications are forced to utilize for proper archiving functionality on that platform."
He also posted a later blog entry on Whistleblowing which revisited the topic and concluded: "It's unfortunate that EMC product complexity and uncertainty is increasing the risk of the very data it is entrusted to protect. It's time to stabilize and simplify your archiving environment."
With Symantec's Enterprise Vault and Centera, complex software is writing to a complex storage product. But that is a side issue, irrelevant in my own view. NetApp's ONTAP is a highly complex storage product and he's not suggesting we abandon its use because of its complexity. Simply put, Bercovici claimed the Symantec tech-note indicated Centera lost archived data when it did not. He was wrong.
Storage blogging by professionals well-informed about their own company's products, generally knowledgable about storage products, practices and processes, and also pre-disposed to criticising competing vendors' products, can be a great generator of light. But sometimes the heat and smoke flowing from a blog controversy completely obscures the light.
As far as I can ascertain EMC's Centera, like NetApp's SnapLock, does not lose stored data. ®