MWeb Business outage cause: 'Someone pushed the wrong button'


Details are emerging of the provisioning error which caused MWeb Business to erase its customers' virtual machines (VMs). Customers who have interacted with technical staff at MWeb report that a provisioning tool was allegedly updated, with an error in a script causing it to end-date all running VMs, leading to their instant erasure.

One customer tells us his account manager confirmed that it was, ultimately, human error. "Someone pushed the wrong button," the account manager summed it up ruefully.

This raises a host of further questions: why are end-of-life VMs instantly erased, rather than triggering a notification and a grace period, to avoid backlash in the event of, say, a billing error, user configuration error, or indeed calamitous system-wide failure? Why was the script in question not sufficiently tested, in a controlled environment, prior to deployment throughout the entire VM portfolio?
Source: ITWeb Africa 21 July 2015