Is Windows 10 ignoring sysadmins' network QoS settings?
Sysadmin fingers auto-update mechanism as source of accidental-self-DoS-hell
An Australian sysadmin frustrated with his business' sudden loss of performance has sparked a conversation about whether Windows 10 is behaving badly on network connections.
To jump well into the discussion thread that points the finger at Microsoft: “We have had reports now from several people, not all our clients, reporting that their Internet connection is brought to a standstill and the common thread is that they all have Windows 10 machines recently installed.”
The issue arose on broadband news and tech chat site Whirlpool here, when a longtime inbound QoS user asked “why are my clients' network connections getting soaked?” Rather than a denial-of-service, it seemed to stem from connections initiated from inside the network:
“ I noticed a couple of internal hosts accessing a couple of IPs 22.214.171.124 and 126.96.36.199, making many connections. As soon as I blocked these addresses the data dropped away.
“A reverse DNS lookup shows that the addresses are Akamai addresses and looking up the source of the addresses shows them to be Zettanet owned addresses. So I am assuming that the issue comes from an Akamai distribution point on the Zettanet network.”
So far so good: a rate limiter stopping too many connections to a single IP address, but what happens next is where Windows 10 turns up in the discussion: “This issue popped up on some other sites and we quickly realised it was Windows 10 updates”.
As anybody on a constrained network will know, Microsoft updates can hammer a network – for example, if everybody switches on their desktops at roughly the same time and clicks “yes”.
Rate-limiting is an obvious defence, but for this: “What seems to be happening is that instead of the sending server reducing its window size when packets are dropped, it just keeps re-sending large windows, which are obviously being dropped at my end. The queue algorithm has no idea of this and it will be letting packets through at a rate it thinks is correct, so the flow continues even though much of the traffic is dropped. However as the traffic keeps coming, the link is totally saturated.”
Even worse, the issue seen by the Whirlpool user (ChopsyWA) is intermittent.
ChopsyWA then went from speculation to certainty:
“It has happened again. The problem is definitely looking to be Windows 10, or the new Microsoft update service as used on Office 2016. We had the issue in our own office with the latter and yesterday the culprit was a Surface Pro on the network. Last week the issue happened also with a Surface Pro.”
Wireshark revealed more information: “There are 20 concurrent connections and stacks of retransmissions and TCP Out of order frames. Just a total mess. The problem is amplified by the multiple concurrent connections issue.
“Comparing this to a ftp conversation that was coming in to us from an Imagemanager client (multiple FTP streams), the FTP conversation has far fewer retransmits and no out of order frames.”
The sysadmin says for his own networks he uses a patch management system to block auto-update, but others aren't so lucky.
What do our sysadmin readers think? Is ChopsyWA ringing the right bell in blaming Windows 10 updates? Share your own experiences in the comments. ®