• Latest
  • Trending
  • All
Answer card stating that GitHub.com was degraded for 7 hours and 47 minutes on 17 August 2026, from 13:28 to 21:15 UTC, with peak error rates around 20 percent on web and API traffic, and that the immediate cause was an Istio sidecar pod reaching its concurrency limit while the autoscaling policy watched the host service instead.

GitHub’s autoscaler watched the wrong service for 7h47m

3 September 2026
Answer card stating that Mullvad announced on 3 September 2026 that it is shutting down its public encrypted domain name system servers on 2 November 2026 and sponsoring the Quad9 Foundation instead, with 194.242.2.2 and its five sibling addresses all going away, and virtual private network customers unaffected.

Mullvad’s DNS servers go dark on 2 November, and Quad9 blocks no ads

5 September 2026
OpenAI announcement image for GPT-6 Astra, a spiral galaxy of white, blue and amber points of light curling around a bright core on a near black star field.

GPT-6 Astra lists at $10 and $50, 2.5x what GPT-5.6 Sol costs

6 September 2026
Google's official announcement image for the release, reading Introducing Gemini 3.8 Flash and 3.8 Flash Cyber in black type over a pale blue background with a blurred white chevron and the four colour Gemini spark below.

Gemini 3.8 Flash keeps the price and the 1 January cliff

3 September 2026
Answer card stating that Anthropic announced Enterprise Frontier Safeguards on 1 September 2026, that activity data used for misuse monitoring moves into cloud storage the customer controls under the customer own encryption keys, that Anthropic charges nothing for the feature while the cloud provider bills storage and egress, and that the phased rollout starts later in autumn 2026 with interim zero data retention on Fable 5 and Fable 5.1 for eligible customers.

Anthropic moves retention into your own cloud, for 30 days

3 September 2026
Official Google diagram of a client connection in three numbered steps: a DNS lookup with a query and an address, a TLS ClientHello and ServerHello, then a content exchange with a website. A callout on the DNS step reads 25% of global web traffic is now protected by encrypted DNS, and a callout beside an Android phone on the ClientHello step reads Android 17 supports ECH GREASE by default.

Android 17 hides the SNI, not your DNS or destination

3 September 2026
Still frame from the Claude Fable 5.1 launch video showing model-designed protein binders in orange docked against twelve grey target proteins, rendered as ESMFold2 structure predictions.

Claude Fable 5.1 breaks forced tool use, cuts cache 75%

1 September 2026
Answer card stating that on 31 August 2026 the European Commission designated ChatGPT a Very Large Online Search Engine under the Digital Services Act, the first conversational AI service classified that way, because it answers user prompts and queries including by searching the web, with OpenAI having declared roughly 159.1 million average monthly users in the European Union for ChatGPT search.

The EU now calls ChatGPT a very large search engine

3 September 2026
Answer card stating that on 31 August 2026 the Department of War added OpenAI ChatGPT Mil and Starshield AI Grok for Government to the GenAI.mil portal alongside Google Gemini, all three accredited at Impact Level 5 for Controlled Unclassified Information, with 1.7 million unique users onboarded out of roughly 3 million eligible personnel, and ChatGPT Mil currently serving GPT-5.4 Terra with GPT-5.6 Terra said to be rolling out.

ChatGPT Mil and Grok reached IL5 on GenAI.mil

3 September 2026
Answer card stating that Anthropic opened a research preview of the Model Hardware Standard on 27 August 2026, standardising the driver layer between an operating system and a laboratory instrument with read and write primitives plus discovery and safety limits, reachable through MCP as well as a command line and code files, with no public specification published.

Anthropic’s Model Hardware Standard is gated, and sits under MCP

3 September 2026
Official Cohere key art for the Parse 5 launch: the Cohere mark and the wordmark Parse with a superscript 5 in white, centred on a soft out of focus gradient of deep blue, violet and amber curves.

Cohere Parse 5 is $1.50 per 1,000 pages, on three of five dimensions

3 September 2026
Title card from the OpenAI announcement video: a man sits on a blue sofa in a loft with tall windows and potted plants, a laptop open on the coffee table in front of him, with the words WebMCP in ChatGPT in large white type across the lower left.

WebMCP in ChatGPT needs GPT-5.6 Sol or Terra

3 September 2026
The Agentic Coding section of the official Hy4 preview benchmark appendix published by Tencent, a table comparing Hy3 and Hy4 preview against DeepSeek V4 Pro 0813, Qwen 3.8 Max, GLM 5.3, Kimi K3, GPT 5.6 Sol and Claude Opus 5 across SWE-bench Multilingual, SWE-bench Pro, DeepSWE, three SWE Atlas tasks, SWE-Marathon, Terminal-Bench 2.1, NL2Repo-Bench, CyberGym, ProgramBench, PostTrainBench and Harbor-Index.

Tencent’s 770B Hy4 tops one benchmark row in 46

3 September 2026
  • About
  • Contact
  • Privacy
  • Legal
Sunday, September 6, 2026
  • Login
Packet Nebula
  • Home
  • Articles
    • Security
    • Network
    • Dev
    • Sysadmin
    • SEO
    • Email & DNS
  • Tools
    • Network tools: free, fast, no signup
    • Security tools: free, fast, no signup
    • Developer tools: free, fast, no signup
    • Sysadmin tools: free, fast, no signup
    • SEO tools: free, fast, no signup
    • Email & DNS tools: free, fast, no signup
  • Download
  • About
No Result
View All Result
Packet Nebula
No Result
View All Result
Home Sysadmin

GitHub’s autoscaler watched the wrong service for 7h47m

by stephane
3 September 2026
in Sysadmin
0
Answer card stating that GitHub.com was degraded for 7 hours and 47 minutes on 17 August 2026, from 13:28 to 21:15 UTC, with peak error rates around 20 percent on web and API traffic, and that the immediate cause was an Istio sidecar pod reaching its concurrency limit while the autoscaling policy watched the host service instead.
491
SHARES
1.4k
VIEWS
Share on FacebookShare on Twitter

Monday afternoon, 13:28 UTC, and roughly one request in five to github.com came back an error. That ran for 7 hours and 47 minutes, and the incident report GitHub posted afterwards names a cause with no bad deploy in it anywhere: an Istio sidecar pod hit its concurrency limit and never scaled, because the scaling policy was watching the host service and not the sidecar. Then it spread. Four HAProxy nodes exhausted their flow limits, the gateway auth path went down with them, and a dormant retry bug in VS Code multiplied Copilot token traffic roughly tenfold on the way back up. We read both writeups GitHub published, which don't say quite the same thing, and pulled out the part that applies to anyone running a mesh.

The short answer

GitHub.com was degraded for 7 hours and 47 minutes on 17 August 2026. No code change, no config change. An Istio sidecar pod in the Central US datacenter hit its concurrency limit and failed to auto scale, because the scaling policy watched the host service and not the sidecar. Load balancers saturated, four HAProxy nodes exhausted their flow limits, and the gateway auth path took SAML and OIDC sign-in down with it. Pausing HAProxy on those nodes at once brought broad recovery. Copilot lagged another three hours because a latent VS Code retry bug pushed the token service from 7 to 9K RPS up to 70 to 100K.

7h 47mdegraded, 13:28 to 21:15 UTC
~20%peak error rate, web and API
10xCopilot token traffic, from a client retry bug
Answer card stating that GitHub.com was degraded for 7 hours and 47 minutes on 17 August 2026, from 13:28 to 21:15 UTC, with peak error rates around 20 percent on web and API traffic and 50 percent on archive and raw content downloads, and that the immediate cause was an Istio sidecar pod reaching its concurrency limit while the autoscaling policy watched the host service instead of the sidecar.
The whole thing in one card. Note what is absent: a deploy.

The chain, in order

Two documents came out of this and they’re pitched at different readers. The CTO’s blog post on 20 August is the corporate one, and it stops at “a critical infrastructure component in our Central US data center failed to scale with it.” The status page resolution note is where the engineering actually lives, and it’s unusually candid.

Here’s what it says. Traffic reached a new peak. An Istio sidecar pod reached its concurrency limits and failed to auto scale correctly, because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more, and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path. That’s the whole outage in three sentences, and the interesting word is sidecar.

Because that’s the bit that generalises. If you run a service mesh, your HPA is almost certainly scaling on the application container. CPU, memory, maybe a custom queue-depth metric from the app itself. The proxy that every one of those requests has to pass through on the way in and out has its own concurrency ceiling, and it is not the same number. GitHub’s policy was watching one and the traffic was killing the other. Nothing in the dashboards said “scale me” because the thing being asked to scale wasn’t the thing being measured.

Then the failure modes stack. Auth is the nastiest one to lose, and GitHub lost it broadly: SAML and OIDC sign-in, SCIM, Team Sync. Also, in a detail almost nobody picked up, Actions workflows in GHEC with Data Residency that depend on public workflow step definitions hosted on github.com. Your sovereign-region CI has a dependency on the public site. Worth knowing before your next audit.

Diagram of the five step cascade in the 17 August 2026 GitHub outage, from a traffic peak in Central US, to an Istio sidecar pod hitting its concurrency limit while the scaling policy watched only the host service, to four HAProxy nodes exhausting flow limits, to the gateway auth path degrading, to a VS Code retry bug driving Copilot token traffic from 7 to 9 thousand requests per second up to 70 to 100 thousand, ending with the mitigation of pausing HAProxy on the affected nodes.
Five steps. Only the second one is a mistake, and it's a configuration line.

The recovery reads like a bad night. Some failing traffic got moved from Central US to Northern Virginia, where it was served fine, and promptly started a retry storm of its own. GitHub fixed that by shipping a PR to trim gateway retry logic and then blocking inbound Copilot token requests at the load balancers with a flat 403, ramping traffic back up per site so callers could actually succeed. Deliberately serving errors to stop serving errors. It works, but you have to be willing to do it.

The Copilot tail is its own lesson. A failed token operation could generate many extra requests and enter a retry loop, so the token service went from a normal 7 to 9K RPS to 70 to 100K. Roughly ten times, from clients, during recovery. GitHub names the culprit plainly: a latent retry bug in VS Code, triggered by delayed replies to a single internal endpoint. One slow endpoint, one editor shipped to millions of desktops, and your recovery window stretches by three hours. There were also scraping attacks on codeload endpoints muddying the picture while all this ran.

The number in the blog post

The CTO post is thinner on mechanism but it carries the stat that actually explains the year. Since April, monthly commits on GitHub have grown from 1.4 billion to 2.9 billion. Doubled. In four months.

Fedorov is careful about it: “That growth explains the pressure on our systems, but it does not excuse these outages.” Fair. But it reframes what these incidents are. This wasn’t a platform slowly outgrowing its capacity plan. It’s a platform whose load curve went vertical, largely because agents now write and push code at machine pace, running into limits that were provisioned for humans typing. GitHub says it has added more than 3 million CPU cores and 120 petabytes of high-speed storage, and installed as much hardware as available power allowed in existing datacenters. Azure now carries roughly 58 percent of platform load and half of all Git operations, up from 12 percent of platform load in May.

That’s a genuinely enormous migration to be running while your commit volume doubles. I’d argue the surprise isn’t that August had two bad days. It’s that the number is 58 and not 20.

Checklist separating what GitHub has confirmed about the August 2026 outages, including two capacity failures on 6 and 17 August neither caused by a code or configuration change, monthly commits growing from 1.4 billion in April to 2.9 billion, 3 million CPU cores and 120 petabytes of storage added and Azure carrying roughly 58 percent of platform load, from what remains unanswered such as completion dates for the five follow-up actions and the root cause of the 20 August Copilot agent task incident.
On the record, and still open. The follow-ups have no dates on them.

Three things to go check this afternoon

The first one costs you ten minutes. Pull up any HPA in front of a meshed workload and ask what metric it scales on. If the answer is the app container’s CPU, you have GitHub’s bug. Istio’s sidecar has its own concurrency and connection limits, and they need to be in the scaling signal or at least in an alert. GitHub’s own follow-up list starts with exactly this, plus an audit of Istio request, concurrency and scaling limits across affected services.

Second, retries. Both of GitHub’s remediations that touch code are about retry discipline: consistent retry limits, retry budgets, variable timeouts across service-to-service calls. A retry storm doesn’t need a bug to hurt you, it just needs enough clients being optimistic at the same moment. And check the clients you ship, not only the servers you run, because the VS Code amplification came from software GitHub wrote and users installed.

Third, the boring one nobody does: know what breaks for you when github.com is 20 percent errors for eight hours. Not the drama, the actual list. Which pipelines block, which deploys can’t fetch, whether your runners can still authenticate. GitHub’s own numbers say the platform is under structural strain while a huge infrastructure migration runs underneath it, and 6 and 17 August were nine days apart. Plan like there’ll be a third.

If you’re checking whether an endpoint is genuinely back or just returning a polite error page, our HTTP headers checker shows you the raw status and response headers. And if the Actions side of this is your problem, the Bun 1.4 release quietly changed how the on: key in a workflow file parses, which is a different way to lose a morning to CI.

Sources

GitHub’s public incident record for this outage sits in two places: the resolution note on the GitHub Status history, which carries the technical detail on the Istio sidecar, the HAProxy nodes and the Copilot Token Service traffic figures, and the CTO post “The August 17 outage, and the work ahead” by Vlad Fedorov, published 20 August 2026, which is the source for the commit growth, the added capacity and the Azure share. Both were read on 23 August 2026. The 6 August Actions incident is recorded separately in the same status history.

Frequently asked questions

How long was GitHub down on 17 August 2026?

7 hours and 47 minutes, from 13:28 to 21:15 UTC. At peak, web and API error rates ran around 20 percent, while archive and raw repository content downloads hit roughly 50 percent. Most services came back as the Central US datacenter recovered at about 16:36 UTC, Actions stayed degraded until roughly 18:03 UTC, and the Copilot Token Service was the last to clear at 21:02.

What caused the GitHub outage?

GitHub's incident report puts the immediate cause as network saturation on load balancers in Central US after traffic hit a new peak. The origin was an Istio sidecar pod reaching its concurrency limits and failing to auto scale, because a misconfigured policy watched the host service rather than the sidecar's own limits. That cascaded until four HAProxy nodes exhausted their flow limits, which degraded the gateway auth path.

Was the outage caused by a bad deployment?

No. GitHub's CTO Vlad Fedorov wrote that neither the 6 August nor the 17 August incident came from a code or configuration change, and that both were capacity failures at their core. The company says it failed to scale critical components before demand exceeded their capacity.

What did GitHub do to stop it?

Pausing HAProxy on the affected nodes simultaneously produced immediate broad recovery. Before that, some failing traffic was shifted from Central US to Northern Virginia, which set off a second retry storm there. That one was handled by shipping a PR to reduce gateway retry logic and by blocking inbound Copilot Token Service token requests at the load balancers with a 403, then ramping traffic back per site.

What is GitHub changing so it does not happen again?

Five follow-ups are listed: correcting autoscaling policies to account for service-mesh sidecar concurrency and capacity, auditing Istio request, concurrency and scaling limits across affected services, reviewing retry limits and backoff behaviour across gateways and clients, addressing the VS Code retry behaviour that amplified Copilot token traffic, and improving load-balancer capacity monitoring plus regional failover safeguards. None carries a published completion date.

Tags: devopsgithubistiokubernetesnewssysadmin
Share196Tweet123
stephane

stephane

  • Trending
  • Comments
  • Latest
Answer card: Proton Lumo 2.0 is private by policy, not by locality. Saved history is locked so even Proton cannot read it, but the prompt is decrypted on a Proton EU server to answer it, then forgotten.

Proton Lumo 2.0 review: how private is it, really?

3 September 2026
Google's official announcement image for the release, reading Introducing Gemini 3.8 Flash and 3.8 Flash Cyber in black type over a pale blue background with a blurred white chevron and the four colour Gemini spark below.

Gemini 3.8 Flash keeps the price and the 1 January cliff

3 September 2026
Answer card stating that Anthropic announced Enterprise Frontier Safeguards on 1 September 2026, that activity data used for misuse monitoring moves into cloud storage the customer controls under the customer own encryption keys, that Anthropic charges nothing for the feature while the cloud provider bills storage and egress, and that the phased rollout starts later in autumn 2026 with interim zero data retention on Fable 5 and Fable 5.1 for eligible customers.

Anthropic moves retention into your own cloud, for 30 days

3 September 2026
Answer card: JWTs are not encrypted, anyone can read them; the signature proves who issued the token, not who may read it.

Are JWTs encrypted? No, and the difference will bite you

0
Answer card: a random 8 character password falls in under 2 hours offline, while 16 random characters hold for 1.4 trillion years at the same speed.

How long does it take to crack a password in 2026?

0
Answer card: three DNS records decide if your mail lands or bounces; SPF lists allowed senders, DKIM signs messages, DMARC sets the failure policy.

SPF, DKIM and DMARC explained: the records your email needs

0
Answer card stating that Mullvad announced on 3 September 2026 that it is shutting down its public encrypted domain name system servers on 2 November 2026 and sponsoring the Quad9 Foundation instead, with 194.242.2.2 and its five sibling addresses all going away, and virtual private network customers unaffected.

Mullvad’s DNS servers go dark on 2 November, and Quad9 blocks no ads

5 September 2026
OpenAI announcement image for GPT-6 Astra, a spiral galaxy of white, blue and amber points of light curling around a bright core on a near black star field.

GPT-6 Astra lists at $10 and $50, 2.5x what GPT-5.6 Sol costs

6 September 2026
Google's official announcement image for the release, reading Introducing Gemini 3.8 Flash and 3.8 Flash Cyber in black type over a pale blue background with a blurred white chevron and the four colour Gemini spark below.

Gemini 3.8 Flash keeps the price and the 1 January cliff

3 September 2026
  • About
  • Contact
  • Privacy
  • Legal

Copyright © 2026 Stephane Cardon.

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In
No Result
View All Result
  • Home
  • Articles
    • Security
    • Network
    • Dev
    • Sysadmin
    • SEO
    • Email & DNS
  • Tools
    • Network tools: free, fast, no signup
    • Security tools: free, fast, no signup
    • Developer tools: free, fast, no signup
    • Sysadmin tools: free, fast, no signup
    • SEO tools: free, fast, no signup
    • Email & DNS tools: free, fast, no signup
  • Download
  • About

Copyright © 2026 Stephane Cardon.