Skip to content

Microsoft traces global Outlook outage to a July software defect

Microsoft says a July software defect, rerouted Teams traffic and reduced server capacity caused its global outage. More than 80% of network gateway servers became unresponsive, while pre-release testing had failed to detect the defect.

Microsoft traces global Outlook outage to a July software defect

Microsoft has identified a software defect introduced months earlier as a central cause of the global outage that disrupted Outlook and other Microsoft 365 services. In a preliminary post-incident review detailed by The Guardian, the company explained how changes in network traffic and reduced server capacity exposed a defect that had remained undetected since July.

The disruption lasted approximately 24 hours, beginning on November 12 and continuing into November 13. Exchange Online suffered the most severe effects, with users unable to access mailboxes through Outlook and other email clients, while connected Microsoft 365 services experienced additional failures.

01 / 03

A July software fix passed testing despite a hidden defect

Microsoft said the underlying problem originated in a software fix developed specifically for the networking gateways that process connections to its cloud services. Deployment began in July, and the updated software operated across the company's infrastructure for more than three months without triggering a major incident.

The defect caused servers to stop responding under particular traffic conditions. According to BleepingComputer's examination of the preliminary report, Microsoft's pre-release testing failed to reproduce those conditions because its laboratories used a different combination of traffic patterns and server hardware.

The failure was particularly sensitive to machines with newer processors and higher core counts. Microsoft's tests had concentrated on less powerful equipment, allowing a problem that was more pronounced on production hardware to remain undetected.

The immediate conditions that exposed the defect developed shortly before the outage. Approximately 15 hours earlier, Microsoft Teams traffic had automatically moved from an alternative networking gateway to the primary gateway also used by Exchange Online, increasing the workload on that infrastructure.

At the same time, nearly 14% of the primary gateway's server capacity was already unavailable because of a separate issue. Microsoft said the reduced capacity would normally have been manageable, but the additional traffic placed sufficient pressure on the remaining servers to trigger the dormant defect.

02 / 03

Automatic traffic routing spread the failure across regions

As the affected servers stopped processing requests, Microsoft's automated traffic management systems redirected connections to machines that were still operating. Those servers then received additional traffic and encountered the same defect, producing a succession of failures across the gateway infrastructure.

The preliminary report states that more than 80% of the gateway servers were unresponsive at the most severe point of the incident. The failures spread across the fleet in less than two hours, while systems intended to isolate problems between infrastructure regions did not prevent the disruption from spreading globally.

Microsoft's network architecture normally allows several services to switch between independent gateway systems. Teams, OneDrive and SharePoint were able to move away from the affected infrastructure and recover earlier, but Exchange Online could not make the equivalent transition at the required capacity.

The consequences extended beyond email access. Microsoft's service health notices identified disruption to features involving Copilot, Defender and other enterprise applications, although functions that did not depend on the affected infrastructure remained available in some products.

Recovery was complicated by the accumulation of connection requests during the interruption. When engineers restarted servers, returning traffic could overwhelm them again before normal processing had resumed.

Microsoft also found that its usual software rollback mechanisms were poorly suited to reversing a change that had been deployed months before the incident. The company's automated recovery procedures had not been designed to restore service after such a large proportion of the gateway fleet stopped responding simultaneously.

03 / 03

Microsoft promises safeguards as further questions remain

Engineers worked to restore service by isolating affected infrastructure and returning servers to operation in controlled stages. Microsoft confirmed that active customer impact had ended by 16:10 UTC on November 13, following a recovery process that required careful management of returning connection traffic.

The company said it had since increased gateway capacity, strengthened monitoring and introduced additional protections intended to prevent traffic redirection from causing similar widespread failures. Charles Lamanna, Microsoft's executive vice-president of Copilot, Agents and Platform, apologised to customers for the disruption and acknowledged its effect on normal business operations.

The preliminary findings also exposed shortcomings in the company's testing arrangements. As Computerworld reported, the difference between laboratory and production hardware allowed a defect to pass testing even though it could cause serious failures under real operating conditions.

Some technical questions remain unanswered in the initial review. Microsoft has not fully explained the separate infrastructure problem responsible for the earlier capacity reduction or why protections intended to contain failures between regions proved insufficient.

The company expects to provide further findings in its final post-incident review. The preliminary report does not quantify the number of customers affected or establish the total financial consequences for organisations that depended on the disrupted services.

Related

Latest

Reputation Insider

The first media publication dedicated to reputation management, covering industry news and professional practice.