Microsoft blames three-month-old gateway bug for lengthy global Microsoft 365 outage
Microsoft has explained last Friday's lengthy global Microsoft 365 outage, blaming a defect that sat undetected in its networking gateway servers for more than three months.
According to Charles Lamanna, Microsoft's executive vice-president of Copilot, agents and platform, the defect was lock contention inside an internal process. This happens when many tasks queue for the same shared resource, so work stalls while they wait. Under certain load conditions it made servers unresponsive. The bug was introduced by a fix Microsoft wrote for the gateway servers, which began rolling out in July and ran worldwide for more than three months without problems. Lamanna said neither the defect nor the fix applies to broader Windows deployments.
Microsoft said its load testing missed the problem because labs focused on weaker hardware with fewer cores, while the contention was worse on newer machines with more processor cores. About 15 hours before the outage, Teams failed over from a separate gateway onto the primary one that also serves Exchange Online, adding load. Nearly 14 per cent of gateway capacity was already out of service from an unrelated issue the previous day. Servers began failing during peak US traffic, and as traffic management steered users away, the remaining machines were overloaded. In under two hours, more than 80 per cent of the gateway's servers were unresponsive.
Isolation between capacity units and regions should have contained the damage but did not, due to a separate issue Lamanna did not describe. Teams, OneDrive, SharePoint and Office apps could fail out of the faulty gateway and recovered earlier, but Exchange Online could not. Recovery was slow because restored servers were swamped by waiting traffic, and automated recovery was not built for such a broad failure. Normal rollback, flighting and feature flags could not quickly revert the change, so engineers built an update restoring the earlier code and applied it to large blocks of servers. A final post-incident review is due later this week.
Why it matters
This outage shows that a vendor's fault isolation and rollback controls can fail together, so a single cloud dependency can take out a core service like Exchange Online for every customer at once. Businesses that assume large providers are resilient by design, and have no fallback for email or identity-linked workflows, are exposed. It also forces a decision on whether continuity plans cover a provider-side failure that no customer action can fix.
What to do
- Document an out-of-band communication channel for staff and key customers to use if Exchange Online is down.
- Review the final Microsoft post-incident review when published and note any dependencies it identifies.
- Test how critical business processes continue when Microsoft 365 email is unavailable.