The Channel logo


By | Alexander J Martin 12th June 2015 12:59

ISP Level 3 goes TITSUP after giganto traffic routing blunder

Explanations spread way faster than Level 3 users' packets

ISP Level 3's customers have been left without internet access since this morning, after the provider seems to have leaked routes to a Tier 1 transit provider in Malaysia.

An incident report from CloudFlare said that while "the Tier 1 transit provider of the ISP leaking routes appears to have stopped accepting these announcements," some routing changes may still be re-occurring.

Taking a stab at explaining route leaking, amalcon at Hacker News offered:

Basically what's happened here is that Telecom Malaysia told one of Level3's networks (AS3549 - Global Crossing) that it was capable of delivering traffic to... anywhere on the Internet. Global Crossing apparently didn't have the usual sanity checks in place.

Because of how BGP works, once [Level 3] decided the route seemed reasonable, it immediately proceeded to dump huge amounts of traffic on the tiny Telecom Malaysia.

This isn't incorrect behavior on [Level 3]'s part (accepting the routes was incorrect, but sending the traffic was correct once they were accepted). This is way more traffic than Telecom Malaysia was prepared to handle.

Every tier 1 network gets rid of traffic as soon as possible (because it reduces their costs), completely ignoring performance or whether a route seems sensible. Telecom Malaysia therefore claimed to [be] the most attractive route from anywhere near Malaysia to most of the world.

Traffic should now be back to normal, CloudFlare says, but many on Twitter find that this is not the case.

The complaints began earlier this morning and seemed to affect users across the globe. Plenty of debate continues as to the exact cause and nature of the outage.

Tweets from Level 3 suggest it is dealing with the problem but that the issues are ongoing.

A Level 3 spokeswoman told The Register: "Level 3's network, alongside some other ISPs', experienced service disruptions affecting customers in EMEA and globally on June 12th. Maintaining a reliable, high-performing network for our customers is our top priority. Our network technicians worked to resolve the issue, and services have been widely restored."

"The root cause has been isolated and linked back to an Asian provider issue; we are working closely together to address this and prevent it from recurring." ®

comment icon Read 25 comments on this article or post a comment alert Send corrections


Frank Jennings

What do you do? Use manual typwriters or live in a Scottish croft? Our man advises
A rusty petrol pump at an abandoned gas station. Pic by Silvia B. Jakiello via shutterstock

Trevor Pott

Among other things, Active Directory needs an overhaul
Baby looks taken aback/shocked/affronted. Photo by Shutterstock

Kat Hall

Plans for 2 million FTTP connections in next four years 'not enough'
Microsoft CEO Satya Nadella


League of gentlemen poster - Tubbs and Edward at the local shop. Copyright BBC
One reselling man tells his tale of woe