Microsoft’s Californian outage: unplug it and plug it back in again – Computerworld
Microsoft cloud and Azure services hosted on the West Coast of the US went down for hours on Thursday when network connectivity failed. Although services running entirely within Microsoft’s West US cloud region were unaffected, any traffic entering or leaving the facilities was affected.
Microsoft has now published a Preliminary Post Incident Review (PIR) of the incident, reporting that connectivity was lost for five hours between 14.44 UTC (7.44 a.m. Pacific Time) and 19.41 UTC on July 23. The problem was caused when a set of IP routes was removed in error while isolating a device for routine maintenance.
Before starting the maintenance work, Microsoft checked that at least one of the two redundant paths to the facility remained operational. When it came to starting the work, however, automated systems included some additional devices in the perimeter to be isolated, and removing some IP routes that had not been included in the initial assessment.
