HTTP/2, Brute! Then fall, server. Admin! Ops! The server is dead
Beware the denials of service: Netflix warns of eight networking bugs
On Tuesday, Netflix, working in conjunction with Google and CERT/CC, published a security advisory covering a series of vulnerabilities that enable denial of service attacks against servers running HTTP/2 services.
HTTP/2, like earlier versions, governs the application layer of the internet stack; it runs atop the transport layer (TCP), the network layer (IP), and data link layer of the internet. The eight CVEs disclosed do not allow information disclosure or modification, but they could be employed to overload servers.
"Today, a number of vendors have announced patches to correct this suboptimal behavior," the media streaming biz said in its post. "While we haven’t detected these vulnerabilities in our open source packages, we are issuing this security advisory to document our findings and to further assist the Internet security community in remediating these issues."
Seven of the flaws were identified by Jonathan Looney of Netflix, and the eighth (CVE-2019-9518) which was found by Piotr Sikora of Google.
Netflix, which characterized the severity of the flaws as "high," did not name the vendors affected by vulnerable HTTP/2 implementations but CERT/CC has.
Microsoft is one of them: It patched five of the eight CVEs, each rated "important" in terms of severity, in its security fix today.
"If any of our customers host web services over HTTP/2 on an alternative, publicly accessible path that is not behind Cloudflare, we recommend you apply the latest security updates to your origin servers in order to protect yourselves from these HTTP/2 vulnerabilities," said Nafeez Ahamed, a security engineer at Cloudflare, in a blog post.
Other affected vendors and products include: F5's NGINX, Go 1.12, Netty, Apache, node.js 11.11.0 + libnghttp2 1.35.1, gRPC C 1.21.0, gRPC Java 1.21.0 (Netty), gRPC Go 1.21.0, Twisted 16.3.0-19.7.0, Envoy prior to 1.11.1, proxygen, H2O project and Istio.
Amazon, Facebook and Ubuntu also appear to be affected, according to CERT/CC, with about two hundred other vendors listed as status unknown.
The vulnerabilities have to do with the design of HTTP/2, which includes parameters that can be abused. That possibility is touched on in the Security Considerations section of the HTTP/2 spec, RFC 7540: "An endpoint that doesn't monitor this behavior exposes itself to a risk of denial-of-service attack."
As Netflix explains, "Many of the attack vectors we found (and which were fixed today) are variants on a theme: a malicious client asks the server to do something which generates a response, but the client refuses to read the response. This exercises the server’s queue management code. Depending on how the server handles its queues, the client can force it to consume excess memory and CPU while processing its requests."
The CVE summaries provide further detail:
- CVE-2019-9511 “Data Dribble”: The attacker requests a large amount of data from a specified resource over multiple streams. They manipulate window size and stream priority to force the server to queue the data in 1-byte chunks. Depending on how efficiently this data is queued, this can consume excess CPU, memory, or both, potentially leading to a denial of service.
- CVE-2019-9512 “Ping Flood”: The attacker sends continual pings to an HTTP/2 peer, causing the peer to build an internal queue of responses. Depending on how efficiently this data is queued, this can consume excess CPU, memory, or both, potentially leading to a denial of service.
- CVE-2019-9513 “Resource Loop”: The attacker creates multiple request streams and continually shuffles the priority of the streams in a way that causes substantial churn to the priority tree. This can consume excess CPU, potentially leading to a denial of service.
- CVE-2019-9514 “Reset Flood”: The attacker opens a number of streams and sends an invalid request over each stream that should solicit a stream of RST_STREAM frames from the peer. Depending on how the peer queues the RST_STREAM frames, this can consume excess memory, CPU, or both, potentially leading to a denial of service.
- CVE-2019-9515 “Settings Flood”: The attacker sends a stream of SETTINGS frames to the peer. Since the RFC requires that the peer reply with one acknowledgement per SETTINGS frame, an empty SETTINGS frame is almost equivalent in behavior to a ping. Depending on how efficiently this data is queued, this can consume excess CPU, memory, or both, potentially leading to a denial of service.
- CVE-2019-9516 “0-Length Headers Leak”: The attacker sends a stream of headers with a 0-length header name and 0-length header value, optionally Huffman encoded into 1-byte or greater headers. Some implementations allocate memory for these headers and keep the allocation alive until the session dies. This can consume excess memory, potentially leading to a denial of service.
- CVE-2019-9517 “Internal Data Buffering”: The attacker opens the HTTP/2 window so the peer can send without constraint; however, they leave the TCP window closed so the peer cannot actually write (many of) the bytes on the wire. The attacker then sends a stream of requests for a large response object. Depending on how the servers queue the responses, this can consume excess memory, CPU, or both, potentially leading to a denial of service.
- CVE-2019-9518 “Empty Frames Flood”: The attacker sends a stream of frames with an empty payload and without the end-of-stream flag. These frames can be DATA, HEADERS, CONTINUATION and/or PUSH_PROMISE. The peer spends time processing each frame disproportionate to attack bandwidth. This can consume excess CPU, potentially leading to a denial of service.
In the absence of a patch, Netflix suggests disabling HTTP/2 services, though that may not be practical in some cases. ®