The Channel logo

News

By | Gavin Clarke 18th November 2015 13:04

Telecity's engineers to spend SECOND night fixing web hub power outage

Irate customers left in the dark

Updated Telecity engineers will tonight make a second attempt to fix broken power systems at its critical Sovereign House internet facility.

They will be working 24 hours after their first efforts failed and resulted in a second power outage.

The internet service giant had planned to repair and fully fix the power supply in the early hours of Wednesday morning. But based on commnunications The Register has seen, things did not get fixed – in fact, they worsened, producing another power outage.

A outage struck Sovereign House Tuesday afternoon, when both a primary supply and its supposedly reliable backup went down. The outage coincided with a noticeable dip in internet traffic across the London Internet Exchange (LINX).

Sources told The Reg Wednesday afternoon Telecity was gearing up for a second crack at fixing the problem.

Telecity, whose Sovereign House data centre appears to handle about 10 per cent of the UK’s internet traffic, promised to update customers at 1430 GMT – some 24 hours after its service started to suffer outages.

Our source reckoned Sovereign House had suffered "significant" power surges that had caused IT equipment to fail.

Servers had fried and had to be replaced, we've learned.

Expecting a fix on the first night, Telecity customer Dotmailer told users of its service at 0414 GMT yesterday:

Telecity have been forced to re-evaluate the situation and we await their decision on how they will proceed. Unfortunately dotmailer services must remain offline until we receive a stable power feed.

Dotmailer sys admin lead Jason Sanchania Tweeted:

One Reg reader got in touch with us mid-morning today to confirm that Telecity’s planned repair work was not successful, and had resulted in another power failure.

The reader told us Telecity had told him its services were reliant on mains power and are not protected by UPS. Telecity handles critical business services for big names including Amazon and Microsoft, with each firm's AWS and Azure products.

Another source reckoned customers had paid Telecity for the use of dual, fully independent power suppliers to avoid outages but the fact their service was down indicated they hadn't actually got what they'd paid for.

Customers’ Office 365 and Lync/Skype connections had been affected with AWS Direct Connect also dropping VPC tunnels. The problem seems to have struck Amazon's AWS EU-WEST-1 region.

Also, day-to-day services, including VoIP and cloud hosting for names such as charity donation payment and donations service JustGiving, have been hit.

AWS said the issue of packet loss has been resolved for its service, which it said was now operating normally.

El Reg could not reach Telecity for comment despite repeated phonecalls and emails.

Are you or your firm affected by the Telecity outage? Contact us: news@theregister.co.uk or phone the London newsdesk on +44 (0)20 3770 3147.

comment icon Read 20 comments on this article or post a comment alert Send corrections

Opinion

Baby looks taken aback/shocked/affronted. Photo by Shutterstock

Kat Hall

Plans for 2 million FTTP connections in next four years 'not enough'
Microsoft CEO Satya Nadella
Stranded_ships

Chris Mellor

Thousands of layoffs announced as spinning rust enters its death spiral

Features

Locker room jocks photo via Shutterstock
Best locker-room strategy: Avoid emulating AWS directly
STRASBOURG, JUNE 29, 2016: The seat of the European Parliament. by Marco Aprile for shutterstock. EDITORIAL USE ONLY
Plan b, image via Shutterstock
EU workers, new markets: post-Brexit pressure on May & Co
Tough question, pic via Shutterstock