Updated A major transatlantic cable linking the EU and the US appears to have been cut, causing major problems for cloud operators.
Global frontend security service Cloudflare and huge hoster DigitalOcean both reported significant connectivity problems on Monday as a consequence of a fault in the Telia transatlantic cable.
Most large internet companies route traffic through multiple transit providers, so a single cable going out isn't a fatal problem, but it does tend to cause some short-term disruption, as was the case here.
"Upstream networking issues are causing connectivity issues in Eastern Europe which is affecting multiple US and EU PoPs," CloudFlare wrote on Monday afternoon.
"One of our upstreams (Telia) appears to have a trans-Atlantic cable cut. We're routing around in many EU locations now," the company explained in a subsequent tweet to a miffed user.
Startup web hoster DigitalOcean suffered a network outage in its AMS1 and AMS2 European datacenters.
"Preliminary investigation indicates that Telia Transatlantic cable are down. We are working to resolve the issue and apologize for any interruption this causes for you," the company wrote on its status page.
A web monitoring service from content delivery provider Akamai showed global network traffic dipped down two percent while residents of the UK, France, and other European countries settled in for their post-dinner Netflix splurge (pictured).
It's highly likely that the damaged cable is the "TAT-14" cable system, which links the UK, France, the Netherlands, Germany, and Denmark with the US via multiple 10Gbs direct access interfaces.
At the time of writing, Telia's parent company, TeliaSonera, had not responded to a request by El Reg for further information.
In a tweet, Telia acknowledged the problem.
Vår tidigare driftstörning på vår internet trafik mot USA och Asien är avhjälpt sen 23-tiden, det var en planerad uppdatering som gick fel— Telia Kundservice (@Telia_service) May 19, 2014
According to Google's (flawed) translation engine, this reads roughly as: "Our earlier disruption on our internet traffic to the U.S. and Asia has been remedied since 23 o'clock, it was a planned update that went wrong."
Cloudflare's chief executive, Matthew Prince, was kind enough to return our call. "It now appears more likely that [Telia] misrouted traffic through to North America," he explained. "The most likely reason would be some kind of routing misalignment."
Cloudflare became aware of a problem when European traffic stopped going to the US via the transatlantic cable and instead "started taking a very circuitous route through Hong Kong," he explained. The problem was fixed by Cloudflare relatively quickly, and he believes Telia dealt with it within an hour or so.
"Whenever one of the big tier one ISPs has an issue it sort of shows the fragility of the internet. I think this was most likely human error," he said. ®