Lithium battery explosion paralyzes Alibaba Cloud data center
On September 10, a fire broke out in Alibaba Cloud’s Availability Zone C data center in Singapore, causing serious disruptions to services hosted by major technology companies such as Lazada and ByteDance. The incident caused widespread errors among internal teams and sellers using the cloud platform.
According to the official Alibaba Cloud “Fire in Singapore Availability Zone C Data Center” announcement (hereinafter referred to as the Fire Announcement), the cause of the fire and temperature rise in the Singapore data center was a lithium battery explosion.
The Singapore Fire Department issued an announcement stating that the fire affected two battery rooms, two power supply rooms and an equipment storage room in the attic on the third floor of the data center.
There are also reports that the Singapore Fire Department was still on site on the 11th to deal with the risk, and Alibaba Cloud operation and maintenance engineers were waiting to enter the data center. If the on-site assessment results cannot be restored on the spot, a server equipment migration recovery plan will be implemented.
According to the official statement released by Alibaba Cloud, due to this accident, its key cloud products were affected, including cloud databases Redis, MongoDB, RDSMySQL, object storage OSS, table storage OTS and cloud native big data computing service MaxCompute.
For industries that are highly dependent on cloud services, including cross-border e-commerce and online education, service interruptions mean a decline in user experience and even direct economic losses. The failure of Alibaba Cloud’s Singapore data center caused a series of problems, including slow website access, API call failures, and unreachable cloud storage services.
However, earlier yesterday, Alibaba Cloud updated the progress of the fire in the C computer room in Singapore Availability Zone, saying that most of the cloud products affected by the network had resumed normal service early this morning. The remaining power-off computer room business still needs to wait for the restoration of physical conditions.
At 20:23 last night, the fire department was still dealing with the risks on the building site, and the operation and maintenance engineers were waiting for permission to enter the computer room building. If the physical conditions for on-site recovery are not met after the on-site assessment, the emergency team will implement the server equipment migration and recovery plan.
Data centers are critical infrastructure for data storage and processing, and their security is of paramount importance. In recent years, data center fires around the world have caused huge economic losses. Looking back at recent data center fire accidents, the severity and urgency of this problem are shown.
In August 2022, an explosion occurred at a Google data center in Council Bluffs, Iowa, injuring three people. The data center is one of Google’s largest data centers and was first opened in 2009. After the accident, more than 40,000 people in the United States reported that they could not use Google Search, according to data from Downdetector.com, a downtime tracking website.
Accidents caused by battery fires are not uncommon. In October 2022, a fire broke out at the C&C Banqiao data center of SK Company in South Korea, and the fire was extinguished after about 8 hours. The investigation found that all five battery racks installed in the electrical equipment room on the third underground floor were burned, and the fire seemed to be caused by electrical factors near the batteries and racks.
The fire paralyzed about 32,000 servers and affected the services of tens of millions of users. After the fire in the data center, Kakao series services, including Kakao Talk, were interrupted for about a day before gradually recovering. After the fire, Lee Jong-ho, South Korea’s minister of science and technology, apologized for the fire in the data center that paralyzed the network platform. A few days later, Kakao’s co-CEO also resigned.
French cloud giant OVH also had a data center offline due to a UPS (uninterruptible power supply) fire. In March 2021, the French operator’s SBG2 data center caught fire that affected the entire building, causing one of the four data centers in the area to be completely burned and another partially damaged.
After the fire, about 3.6 million French government, corporate and public utility websites were paralyzed, and some game developers’ businesses in Europe were also affected. Some servers located in the data center were burned. Among them, the game “Rust” stated that 25 European servers were completely destroyed, there was no backup, and the data could not be repaired. Afterwards, more than 130 customers joined the class action lawsuit, accusing OVHcloud of failing to fulfill its obligations and not providing adequate compensation to the companies that suffered losses. The company has been slow to disclose the cause of the fire and insisted that it must wait for an official report.
A year after the fire, the Bas-Rhin Fire Department released an investigation report strongly criticizing the French operator’s facility. According to the accident investigation report, the data center had a considerable number of fire hazards, including: a wooden ceiling with a nominal fire resistance of only one hour, no automatic fire extinguishing device, and no general electric disconnect switch. However, what made the firefighters even more disappointed was that the facility also had a natural cooling design, which also created a “chimney effect” that could increase the fire.
