This article is more than 1 year old

The Navitaire cock-up, filleted

Datacentre details behind Virgin Blue outages

The Register has found out more about the Navitaire data centre responsible for the 21-hour Virgin Blue outage on Sunday and the second outage on Tuesday. It makes the prolonged first outage even more puzzling.

Accenture's Eric Ulmer says (Google Docs) that the Navitaire system has a high transaction volume, and the total value of web-based travel transactions exceeds Amazon.com transactions. The daily credit authorisations reach $135m and some 750,000 batch jobs are run each month.

There are high service level agreement penalties for outages, and application response time waits. There are no scheduled maintenance windows and Navitaire in effect pays to take airline customers down. The first 21-hour outage caused more than 100 Virgin Blue flights to be cancelled with more than 100,000 passengers affected.

The net effect of the two outages could be that Navitaire pays millions of Australian dollars in compensation to Virgin Blue and, perhaps, loses the airline reservation system contract.

We are told by an anonymous source that Navitaire's Sydney-based data centre has more than 32 units of Texas Memory Systems' RamSan 500 front ended by NetApp V-Series V6040 6080 controllers. Ulmer says the RamSans deliver 8TB of usable storage.

We understand Navitaire takes snapshots of the core OLTP database and SnapVaults it to nearline storage. This then gets dumped to local DataDomain systems, which Ulmer says are used as virtual tape libraries. Navitaire also SnapMirrors the primary array OLTP data to another data centre.

We already know the RamSans have a lot of internal data protection features and that Navitaire staff attempted to repair the failed "solid state disk infrastructure" as Virgin Blue described it in a statement.

The fact of these SnapVault and SnapMirror arrangements being in place raises the question why did it take 21 hours to restore the system? Couldn't Navitaire just recover the snapshot data to repaired hardware?

Alternatively, why didn't it run the system off the remote data centre holding the SnapMirror data? The implication is that, because it didn't do this, the Snapmirror process of sending OLTP data to the second data centre was useless.

Another point: It's understood the Sydney data centre services several airlines so why was only Virgin Blue affected?

It is becoming more and more obvious that Navitaire's business continuance and disaster recovery provisions failed completely in this outage. There should have been standby systems ready to take on the load of any failed system or system component, but there weren't any. That is a blunder of the first magnitude by whoever designed, implemented and ran the system. ®

More about

TIP US OFF

Send us news


Other stories you might like