Microsoft scrambles as Outlook inbox goes dark

Detailed summary

You know it’s bad when the Outlook outage overshadows Jensen Huang’s additional cheating allegations. Yes, there are three women ...

You know it’s bad when the Outlook outage overshadows Jensen Huang’s additional cheating allegations. Yes, there are three women now.

“Is there a problem with Outlook?” was the question from my wife as we settled down for bed last night.

The reason, I found out, goes a little like this… At about 8pm our time, Microsoft acknowledged online that people in the Americas couldn’t reach their Exchange Online mailboxes.

The first theory was load. Microsoft said part of its infrastructure was straining, so it paused deployments, rerouted traffic and restarted servers. Then it corrected itself. Demand hadn’t surged. A number of servers had unexpectedly dropped out, and the survivors were buckling under the extra weight.

The second theory was the health check. Servers were failing automated tests and removing themselves from service, and Microsoft pointed to the component that processes server health. It found no recent change linked to the start of the trouble. A different health-check method brought a subset of servers back. A system built to tell healthy machines from sick ones may have been the thing that was sick.

Then came the false dawn. By evening in New York, which was about 1am here, Microsoft said restored servers were stable and users would see a gradual recovery as their clients reconnected. Reports had fallen to roughly 34,000.

A couple of hours later it admitted that the restored servers could go unresponsive again once traffic reached them. It began limiting some types of traffic in some locations and rebuilding a subset of servers on an earlier configuration. Its latest update says a change meant to improve serviceability hadn’t taken as expected, and engineers were looking for other ways to add capacity.

The blast radius is the real story: Copilot couldn’t reach work data. Defender’s Safe Links couldn’t verify URLs. Purview’s data-loss prevention, labelling and eDiscovery tools were hit, along with Intune enrolment, Graph API calls, Planner, To Do and Universal…

Unf…

Opening of the article, excerpted from Daily Maverick. To read the rest, open the article on the publisher's site.