FYI: Ticking time-bomb fault will brick Cisco gear after 18 months
Replacement units will be on their way – just don't call it a recall
Updated Cisco has issued a warning that an electronic component used in versions of its routing, optical networking, security and switch products prior to November 16, 2016 is unreliable – and may fail in the next year and a half, rendering affected hardware permanently inoperable.
"Although the Cisco products with this component are currently performing normally, we expect product failures to increase over the years, beginning after the unit has been in operation for approximately 18 months," Cisco said in its advisory.
"Once the component has failed, the system will stop functioning, will not boot, and is not recoverable."
And without naming names, Cisco said that the clock-signal-generating component is also used by other companies. Expect further notices of this sort from other vendors shortly.
Cisco said it learned about the issue in late November and has worked with the component supplier to fix the faulty part. As a result, currently shipping products are not affected.
For customers with affected products under warranty or covered by service contracts through November 16, 2016, Cisco intends to provide replacement products. It is prioritizing replacements for those who have been operating the affected products the longest, because of the correlation between operation time and component failure.
Cisco insists this isn't a recall; rather it's a proactive replacement. The company also said that while the component maker indicated failures will become more likely after 18 months, it expects it will take three years of runtime before its products show a spike in failures.
In its advisory, Cisco specifically declines to identify the supplier that made the faulty part, or other affected vendors. And company declined to do so in response to a request from The Register for further information.
"Cisco strives to deliver technologies and services that exceed customers' expectations, and meet rigorous quality and customer experience standards," a company spokesperson told The Register in an email. "We became aware of an issue related to a clock signal component manufactured by one supplier. We have worked with the supplier to resolve the issue, and we're providing information and support for our customers."
Cisco's advisory affects the following products:
|Routing||FN-64231||NCS 5500 Line Cards|
|FN-64252||IR809/IR829 Industrial Integrated Services Routers|
|FN-64253||ISR4331, ISR4321, ISR4351 and UCS-E120|
|Security||FN-64228||ASA 5506, ASA 5506W, ASA 5506H, ASA 5508, and ASA 5516|
|FN-64250||Cisco ISA 3000 Industrial Security Appliance|
|Meraki Notification||MX 84|
|Switches||FN-64251||Nexus 9000 Series N9K-C9504-FM-E/N9K-C9508-FM-E/N9K-X9732C-EX|
|Meraki Notification||MS350 Series|
Network engineer Tony Mattke in a blog post said rumors about ISR 4331 router problems and a possible recall have been circulating for several months.
"Many of us have received phone calls at this time from our account managers ... while some of us, myself included, have been left in the dark," he wrote. "This is troubling considering how many products are out in the field." ®
Updated to add
It looks like the dodgy electronic component at the heart of the matter is an Intel Atom C2000 system-on-chip Cisco uses in its gear. The processor can fail to produce a clock signal required to drive the whole device – see this rather worrying errata, specifically AVR54, on this Intel data sheet, dated January 2017: