This is a pretty major outage for Facebook and it seems it may not be ending very soon. According to the site Down Detector, the problem started around 11:30 Eastern Time and is still ongoing as I write this.
Facebook users around the world reported Monday that they were unable to access its family of social-media apps, including the main social network, photo-sharing app Instagram and messaging service WhatsApp.
Some internal services used by Facebook employees, including the company’s Workplace tool for communicating among teams, is also down for some employees, according to a Facebook Inc. spokesperson.
More than 125,000 Facebook users were reporting problems accessing the app at 11:54 a.m. in New York, according to website DownDetector. At least 98,000 people reported trouble with Instagram, and about 35,000 said WhatsApp was having issues. Facebook’s network of apps has more than 2.7 billion daily users on average.
CNN reports that the consensus is a DNS failure and notes that an outage of this duration for a company of this size is pretty unusual.
The reason for the outage was not immediately clear. However, multiple security experts quickly pointed to a Domain Name System (DNS) problem as a possible culprit. Around 1 pm ET, Cisco’s internet analysis division ThousandEyes said on Twitter that its tests indicate the outage is due to an ongoing DNS failure. The DNS translates website names into IP addresses that can be read by a computer. It’s often called the “phonebook of the internet.”…
“I don’t know If I’ve seen an outage like this before from a major internet firm,” said Doug Madory, director of internet analysis at network monitoring firm Kentik.
Ars Technica has more on the nature of the failure. Apparently, this is the result of a config update created by Facebook which inadvertently cut the engineers off from access to the servers, meaning they would need to be on site to fix it.
According to u/ramenporn—who claims to be a Facebook employee and part of the recovery efforts—this is most likely a case of Facebook network engineers pushing a config change that inadvertently locked them out, meaning that the fix must come from data center technicians with local, physical access to the routers in question. The withdrawn routes do not appear to be the result of nor related to any malicious attack on Facebook’s infrastructure.
Meanwhile, some employees can’t even get into the buildings because the security systems are down.
Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors.
— Sheera Frenkel (@sheeraf) October 4, 2021
Overall, it sounds like a real disaster and the market is reacting.
The company’s shares dropped 5.6% to $323.99 at 3:40 p.m. They had already been on the decline Monday after a Facebook whistle-blower appeared on news program “60 Minutes” to discuss the company’s internal research about its effect on society.
The fact that 60 Minutes ran a negative report on the company last night makes this unusual outage seem like quite a coincidence. An outside hack isn’t looking likely but that doesn’t rule out the possibility this was internal sabotage rather than a mistake. I guess we’ll have to wait a while to find out.