A backup test took Workday down. Several employers were back Saturday
Workday took customer systems offline on Friday after a disaster-recovery test left production data in an inconsistent state. In a notice on its Community Status Page, reposted by Washington University's IT team at 8:05 p.m. Eastern, Workday said that while "validating the weekly Maintenance Window for Disaster Recovery tenants," services at the disaster-recovery site "unexpectedly connected to Production databases." That left data "in an inconsistent state," which could cause errors reaching tenants and make jobs, integrations and tasks fail. Workday placed all WD1 production tenants into maintenance while its teams investigated "the best recovery path." Workers at affected organizations could not clock in or out, view pay information, request time off or finish benefits enrollment. LiveNOW from FOX reported that Downdetector logged nearly 3,000 outage reports shortly after 5 p.m. Eastern. Workday's website says more than 11,500 organizations use the platform. A spokesperson told LiveNOW only that "some of our customers are experiencing a service disruption."
Emerson College said Workday took the system offline at about 5:15 p.m. Eastern and extended open enrollment to noon Eastern on Monday, November 16. Reports naming Walmart, Target, Chipotle and Skechers as affected rest on an anonymous source; the retailers have not confirmed it, and being a Workday customer is not the same as being hit. By Saturday morning, a Workday message posted by Florida Atlantic University's IT team said "reindexing and tenant release is now expected to complete for the majority of customers by 11 AM EST." Florida Atlantic said at 10:30 a.m. Eastern that "Workday Production is now available." Washington University said at 12:24 p.m. Eastern that "the Workday unplanned outage has been resolved," though "updates between systems and within Workday may be temporarily delayed." WebProNews reported that Saint Joseph's University, Emerson College and Iowa State University also said service was back by then. Workday has not said whether every affected customer is back, and it has not publicly added to its first account of the cause.
This is friction for security, and it touches goods, because payroll and timekeeping for thousands of employers run through one platform. Workday froze those systems after it found inconsistent data, rather than leave payroll and HR records in use while they were wrong. Posts spread over the same two days treating the Workday outage, a Microsoft 365 outage and a nationwide FAA ground stop as a single collapse or an attack. There is no public evidence they are linked. Each one had a different cause, and each started with a change the operator made itself.
Microsoft's outage began on Thursday, November 12, and ran for more than 24 hours across more than 70 countries, hitting Outlook, Teams and other Microsoft 365 services. Microsoft said it was "a technical issue not related to cyber or AI / agents," introduced "during an earlier network capacity upgrade that surfaced following a recent network change." That account is in manyminded.com/a/one-microsoft-fault-nearly-a-day-of-lost-o3nmo. On Friday, the FAA halted U.S. departures for about four and a half hours. On Saturday, Transportation Secretary Sean Duffy said contractor L3Harris was replacing old copper lines with fiber at a remote facility when a router flooded the network; officials were still determining whether that was user or installer error or a faulty router. FAA Administrator Bryan Bedford said the agency saw "zero evidence of anything nefarious, and zero evidence of anything AI-related." Duffy said, "we don't believe this was cyber at all." Our earlier account of the ground stop is at manyminded.com/a/a-network-crash-grounded-every-us-flight-w8kdn. The through line is ordinary, and it is fixable: test disaster-recovery systems in isolation, keep backups truly separate from live data, stage rollouts, and replace aging equipment with enough care that a single change cannot spread. For the employers who have reported back, Workday's systems recovered within about a day.
Source: WebProNews
Many Minded