← Back to front page
Oceania · General news · The Sydney Morning Herald

One router, one bug. How routine upgrades caused worldwide chaos

Published by The Sydney Morning Herald · Australia 12 h ago
Microsoft’s email and office software crashed after an upgrade meant to add capacity.AP Photo/Kirsty Wigglesworth
Microsoft’s email and office software crashed after an upgrade meant to add capacity.AP Photo/Kirsty Wigglesworth · The Sydney Morning Herald

Detailed summary

ASIO, councils and supermarket suppliers lost email for most of two days, all because of an upgrade meant to add capacity.

Microsoft wanted more capacity. America’s aviation regulator wanted to swap ageing copper cables for fibre. Last week, both got chaos.

Microsoft’s email and office software went dark for millions of people from shortly before 5am AEDT on Friday, after an upgrade meant to add capacity. A day later, US authorities grounded every departing flight in the country for five hours after a contractor installed a router at a remote site in Arkansas.

The two failures were unrelated and Microsoft says it has no indication its services played any part in the grounding.

The upgrades themselves were defensible: old systems commonly have to be replaced, and software has to be patched – that’s normal. What failed was containment: one fault in one place was allowed to course through an entire network.

Australians felt it within hours. ASIO and the federal police lost email and so did councils, courts and schools. Australia Post pulled its phone payment system, while suppliers couldn’t tell supermarket distribution centres what was on the way. In Germany, 15 hotel guests were reportedly hurt by chlorine gas after a pool safety system that sent its warnings by email went silent.

A tech company like Microsoft understands the risk as well as anyone. “There are no low-risk deployments to production,” its guidance to engineers warns, meaning no change to a live system is ever truly safe. Its advice is to change things in small steps, roll them out slowly, and stop at the first wobble.

On Friday the company said the outage “was introduced by an issue during an earlier network capacity upgrade that surfaced” after a later network change. Its preliminary notices say traffic moved around during that maintenance and exposed a hidden bug in the systems that connect people to their mailboxes. Requests banked up until the servers choked.

Then the dominoes fell. Work from the broken servers spilled onto healthy ones, which buckled in turn.

The flight grounding was similar: it started with a single router…

Opening of the article, excerpted from The Sydney Morning Herald. To read the rest, open the article on the publisher's site.

Read the full article on The Sydney Morning Herald Sign in to save

MarketNews shows the headline and summary published by the outlet. The full article is on the publisher's website. (www.smh.com.au)

Also worth reading