The Channel logo

News

By | Alexander J Martin 12th June 2015 12:59

ISP Level 3 goes TITSUP after giganto traffic routing blunder

Explanations spread way faster than Level 3 users' packets

ISP Level 3's customers have been left without internet access since this morning, after the provider seems to have leaked routes to a Tier 1 transit provider in Malaysia.

An incident report from CloudFlare said that while "the Tier 1 transit provider of the ISP leaking routes appears to have stopped accepting these announcements," some routing changes may still be re-occurring.

Taking a stab at explaining route leaking, amalcon at Hacker News offered:

Basically what's happened here is that Telecom Malaysia told one of Level3's networks (AS3549 - Global Crossing) that it was capable of delivering traffic to... anywhere on the Internet. Global Crossing apparently didn't have the usual sanity checks in place.

Because of how BGP works, once [Level 3] decided the route seemed reasonable, it immediately proceeded to dump huge amounts of traffic on the tiny Telecom Malaysia.

This isn't incorrect behavior on [Level 3]'s part (accepting the routes was incorrect, but sending the traffic was correct once they were accepted). This is way more traffic than Telecom Malaysia was prepared to handle.

Every tier 1 network gets rid of traffic as soon as possible (because it reduces their costs), completely ignoring performance or whether a route seems sensible. Telecom Malaysia therefore claimed to [be] the most attractive route from anywhere near Malaysia to most of the world.

Traffic should now be back to normal, CloudFlare says, but many on Twitter find that this is not the case.

The complaints began earlier this morning and seemed to affect users across the globe. Plenty of debate continues as to the exact cause and nature of the outage.

Tweets from Level 3 suggest it is dealing with the problem but that the issues are ongoing.

A Level 3 spokeswoman told The Register: "Level 3's network, alongside some other ISPs', experienced service disruptions affecting customers in EMEA and globally on June 12th. Maintaining a reliable, high-performing network for our customers is our top priority. Our network technicians worked to resolve the issue, and services have been widely restored."

"The root cause has been isolated and linked back to an Asian provider issue; we are working closely together to address this and prevent it from recurring." ®

comment icon Read 25 comments on this article or post a comment alert Send corrections

Opinion

Baby looks taken aback/shocked/affronted. Photo by Shutterstock

Kat Hall

Plans for 2 million FTTP connections in next four years 'not enough'
Microsoft CEO Satya Nadella
Stranded_ships

Chris Mellor

Thousands of layoffs announced as spinning rust enters its death spiral

Features

STRASBOURG, JUNE 29, 2016: The seat of the European Parliament. by Marco Aprile for shutterstock. EDITORIAL USE ONLY
Plan b, image via Shutterstock
EU workers, new markets: post-Brexit pressure on May & Co
Tough question, pic via Shutterstock
Honest mistake with your licensing? Audit police look at it on a 'case by case basis'