Intel Atom chips have been dying for at least 18 months – only now is truth coming to light
Vendors clocked C2xxx flaw when returns spiked
Exclusive The flaw in Intel's Atom C2000 family of chips has been vexing Intel's hardware customers for at least a year and a half, according to a source at one affected supplier, but it wasn't immediately obvious that Intel's silicon was to blame.
The well-placed insider, who spoke to The Register on condition of anonymity, said the problem – which results in bricked systems – became apparent to engineers at product makers when the return rate on gear spiked about 18 months ago.
It took additional time to figure out that Intel's hardware was responsible. The vendor in question worked with Intel engineers during this period to replace affected units under manufacturer warranty. Then about two months ago, Intel acknowledged the issue was with its chip blueprints, the source said.
In January, with the publication of errata documenting the flawed LPC clock outputs in its Atom C2000 system-on-chips, Intel acknowledged the issue publicly. Around the same time, chief financial officer Robert Swan said profitability during Intel's fourth quarter last year had been constrained by an unspecified product issue, prompting the company to establish a cash reserve to deal with the blunder.
Cisco's subsequent publication of an advisory warning that a variety of its products could fail after 18 months of use because of a faulty clock component appeared to The Register to be related to Intel's disclosure, in no small part because Cisco's devices contained Intel Atom C2000 line chips.
Both companies have repeatedly refused to acknowledge the linkage, despite repeated hectoring from The Register. At the same time, they have refused to offer any correction or claim that their respective issues are unrelated.
"We are working with all customers to implement the fix we have available now," an Intel spokesperson told The Register on Tuesday. "We aren't providing additional details on which companies."
Companies that appear to be affected by the Atom C2000 flaw include not only those mentioned in our previous report – Aaeon, Dell, HP, Infortrend, Lanner, NEC, Newisys, Netgate, Netgear, Quanta, Supermicro, Synology, and ZNYX Networks – but also ASRock Rack (C2550D4I and C2750D4I), iXsystems (FreeNAS Mini), Seagate (NAS Pro), and Sophos (UTM firewall SG 125).
Our industry source suggested that non-disclosure agreements with Intel prevent companies from releasing or discussing information about technical issues related to Intel products.
Some Intel customers, however, have trouble maintaining the code of silence. A spokesperson for Synology contacted The Register on Tuesday in response to a languishing query, saying that "Synology is investigating this issue together with Intel."
The spokesperson subsequently asked to revise the company's statement with alternative facts that omitted any mention of Intel.
This code of silence, in the opinion of our source, ends up hurting Intel's hardware customers because they get blamed for products that fail, even when the fault is Intel's.
Public's right to know
Had Intel spoken up sooner and named customers with affected products, there might not be so many forum posts and online product reviews lamenting unexplained failures of devices running Atom C2000 chips.
"I think it's better for the public to know [the failure is] not because of our product," the source said.
Intel's hardware partners and their customers are still trying to sort things out. A Reg reader reported that when he asked about his Seagate NAS Pro, which contains an Intel Atom C2338, Seagate technical support offered a response suggesting disinterest in any potential problem:
"Regarding the Intel Specifications Sheet on the Atom C2000 series in reference to item AVR54, Seagate does not maintain any information on device failure rates or failure rates that may exist due to the above related bulletin."
Synology said it is still investigating the issue and that while it "has not seen any indication that this issue has caused an increase in failure rates for DiskStation or RackStation models equipped with Intel Atom C2000 series processors," the company intends to post an advisory when additional information becomes known.
Others report more positive interaction with affected companies. "I had an ASRock C2750D4I (Intel Atom C2750-based Mini ITX board) that died after 26 months of use!" a Reg reader told us. "Yes, 2 months out of its warranty! Thankfully ASRock replaced the board with no fuss at all ... ASRock just said that the board wasn't repairable and sent a replacement, nothing was ever said regarding why it died." ®
Updated to add
In a statement today, pfSense-maker Netgate said: "Netgate has become aware of an issue related to a component manufactured by one supplier that affects some of our products. This is a widely used component that is used by many companies around the world ... if the component fails, the system will not successfully reboot. We are working with the component supplier and our manufacturing partner to resolve this issue as quickly as possible."