Queue lots of people who know nothing about banking IT to make comments about them not having any DR, this is what you get for systems running VB code, that this sort of problem should be avoidable (which sort of problem?) etc. etc. etc.

Try not to dissapoint.

Shut up you tart - that's why there are comments

.. if HSBC were using Macs!!

oh sorry, wrong thread.

Typical. They don't have any DR. Not the teeniest little amount. Also, this is what you get for having a system that runs VB code, meaning that this situation was entirely avoidable.

Now - Did anyone apart from "First Direct" Fraser understand that?

IT in banking is extremely complex and has been made more so by the different platforms which so may different banks use. Any payment system is going to be inherrently unstable because it is going to have to interact with a number of other disparate systems and that's where VB hacks come in to glue the whole thing together.

Add to that the fact that if a complex and interdependant system goes down it is often very difficult to restart it again quickly as some parts will have to be taken down in order that other ones may start and the dependancies can be the death of you.

All in all they have not done badly at all.

...a small but critical part of this was running on a box under someone's desk, it got 'accidently' turned off, the delay was in working out exactly *which* desk it was under and finding someone who could remember the admin password to fire it all back up?

I guess we have nuked any automated emails.

Does anyone have the email address that these come from so I can add it to my whitelist

thanks

Kev K

I wouldn't like to be in their IT department right now - even the guy who puts toner in the printers is probably getting stiffed for this.

@Fraser

It's actually "Cue", as in a signal, such as a word or action, used to prompt another event in a performance, such as an actor's speech or entrance, a change in lighting, or a sound effect.. not a line of waiting people or vehicles, more commonly found in any British "fast food" outlet where the concept of FAST is above them....

Is there always someone around to say that a total ballsup and utter failure is not "doing too bad, really"?

OK, so in an imperfect world balls-ups and failures will happen, though they shouldn't, but no, that doesn't make them all right.

Can we please have a T5 icon? Or even a bag with a BA tag on it would do.

I wasn't suggesting that this shouldn't have happened, or that it isn't a massive screw up, rather that banking systems are far more complex than generally given credit for.

Consider: A production server fails, you fail over to the DR server with it's copy of the disk in a remote site. Easy, probably even automated.

However: A database corruption occurs, this corruption would have been instantly transfered to the DR disk, so that DR server is totally useless. You (probaby) have snapshots from Start of day, or end of previous day (pre-batch). Did the batch corrupt the database? Do you want to recover from pre-batch, or post batch? If the batch corrupted the database, how? Do you need to re-run the batch, can it be re-run the next night? How long does it take to run? Did another server cause the corruption, at a guess a dialin system like merchant handsets would be using a bigass unix box, or Tandem, almost certainly talking to a back end mainframe, probably via some sort of broker... etc. etc. etc.

This is just one of many scenarios that could have happened, rather simplified one as well, but illustrates how DR can be rendered pretty useless. It doesn't even consider the reuqirement to recover from tape.

Will it happen again this month? Rather not take the risk again and I do understand that banks have problems but so did my customers grrrrrr that kicked off with me! switched to Nochex.com