Should SANs be patched to fix the Spectre and Meltdown bugs? Er ... yes and no
General assumption is yes. But five suppliers say no
Analysis Is the performance sapping spectre of the X86 Spectre/Meltdown bug fixes hanging over SAN storage arrays? The general assumption is "yes" but five suppliers say not.
You would expect SANs to need patching; they run their controller software on X86 servers after all.
UK storage architect Chris Evans writes: “Patching against Meltdown has resulted in performance degradation and increased resource usage, as reported for public cloud-based workloads.”
His understanding is that “the overhead for I/O is due to the context switching that occurs reading and writing data from an external device. I/O gets processed by the O/S kernel and the extra work involved in isolating kernel memory introduces an extra burden on each I/O. I expect both traditional (SAS/SATA) and NVMe drives would be affected because all of these protocols are managed by the kernel.`'
He wonders if there’s a difference between SAS/SATA and NVMe, simply because NVMe is more efficient.
Specifically, for traditional storage arrays: “The additional work being performed with the KAISER patch appears to be introducing extra CPU load in the feedback reported so far. This means it also must affect latency. … The impact to traditional storage is two-fold. First, there’s extra system load, second potentially higher latency for application I/O."
Customers implementing this patch need to know if the increased array CPU levels will have an impact on their systems. A very busy array could have serious problems.
"The second issue of latency is more concerning. That’s because like most performance-related problems, quantifying the impact is really hard. Mixed workload profiles that exist on today’s shared arrays mean that predicting the impact of code change is hard. Hopefully, storage vendors are going to be up-front here and provide customers with some benchmark figures before they apply any patches.“
Nothing to see here - carry on...
But five suppliers say no, their SAN systems will not be affected.
In a blog post IBM says its storage appliances will emerge unscathed.
Here is a statement from Netapp: “Unlike a general-purpose operating system, Element OS is a closed system that does not provide mechanisms for running third-party code. Due to this behaviour, Element OS running on SolidFire or NetApp HCI Storage nodes is not affected by either the Spectre or Meltdown attacks as they depend on the ability to run malicious code directly on the target system.“
On this basis we would expect the same to be true for its DataONTAP FAS arrays as well.
Tintri founder and CTO Dr Kieran Harty tells us: “We are not vulnerable because we only run our own software on our appliances,” adding, “We’re not planning on patching the software that runs on our appliances.”
In effect, Tintri says it doesn’t have to make the performance or security choice - because its dedicated engineered appliance systems don’t run anybody else’s code except Tintri’s and so are secure already.
Meanwhile, DataCore told us that “once a [DataCore SAN] target request has been received by the kernel, whether from a SAN, a Hyper-Converged environment, or MaxParallel, there are no additional transitions to user space involved.
"As a result, based on the information currently available about the proposed mitigation strategies, it seems unlikely that there will be any performance impact on the storage presented by DataCore. Tests are still in progress in the lab to verify that this is the case.”
It doesn’t believe its SAN software needs patching: “DataCore has a close connection to the operation of the Windows kernel, however, it is currently believed that no software changes will be required to protect against the vulnerabilities or as a result of the mitigations. [Again] tests are still in progress in the lab to verify that this is the case.”
What is the rationale for this stance? DataCore says that in the event that a DataCore installation has been compromised, the risk of data under management being exposed currently appears to be almost zero.
A Datacore document that The Register has seen makes these claims:
In order for Meltdown to gain unauthorised access, the memory needs to have a virtual address assigned to it which is not the case for the DataCore cache. A virtual address will be assigned temporarily to individual cache buffers when performing specific operations on a snapshot, replicating data, or allocating storage to a thinly provisioned volume, but this will be released as soon as the operation is complete.
Given that the reported data access rate using Meltdown is up to 503KB/s it is implausible that an attacker would be able to identify a temporary mapping and extract data in the time available.
The DcsAddMem processes have access to user virtual addresses for the cache contents which would potentially open up an attack route using Spectre. However, Spectre requires that the application under attack be executing in order to be vulnerable and this is not the case for DcsAddMem. The processes are blocked within the kernel until virtualisation is stopped at which point the memory is released.
Infinidat CTO Brian Carmody was asked if Infinidat arrays would be affected, and told us: "Not affected. The design of InfiniBox provides no facility for non-privileged users to run 3rd party code locally on the system."
He's repeating the message put out by the other suppliers.
The consequences of some malware-toting person gaining access to mission-critical data could be severe so you really would not want your shared external storage arrays compromised. The Spectre and Meltdown bugs increase X86 servers' vulnerability attack surface, and SANs and filers are controlled by X86 servers, ergo … except not ergo, according to IBM, NetApp, Tintri, Infinidat and DataCore.
It seems this is a judgement call in a way. The suppliers are saying their customers do not have to make a choice between performance and security, because their systems are secure enough already. Are they? It’s not even your call as these suppliers are proposing not to patch their systems. ®