Analysis The fallout from the outage of UK payment processing firm Protx is continuing.
Thousands of online merchants were unable to take payments last Wednesday as a result of a mishandled upgrade that left payment systems at Protx unavailable. Several Reg readers also reported they were unable to reach the firm by either phone or email when trying to resolve the problem.
The delayed upgrade - originally due to be applied in June - involved changes in the processing of Maestro cards, a new system for handling delayed settlement of transactions, and other changes. The most significant (and troublesome, as it turned out) of these changes involved the introduction of 3D Secure, which is similar to an online version of "Chip and PIN". 3D Secure is being introduced by the banks to help reduce fraud for ecommerce transactions. There's some debate over whether Protx applied this system before it strictly had to.
Protx, a Sage subsidiary that's one of the largest payment services in the UK, processes credit and debit card payments for more than 10,000 online and mail order businesses. Problems with the system affected a great number of (mostly small) ecommerce operations.
Root cause analysis
In a letter to merchants sent out on Thursday, Protx chief exec Michael Alculumbre apologised for the inconvenience they experienced as a result of outage of the Protx payment gateway. Soon after the upgrade - the biggest Protx has implemented since November 2005 - was applied at 0600 on Wednesday a bottleneck developed in the system. This lead to a queuing of transactions that eventually overloaded the payment gateway. Protx said the issue had not cropped up during testing.
The root cause of the problem was eventually identified at a low level within Protx's database. "It was established that one table in the database had not been indexed properly and that this was the primary cause for the failure," the letter explains.
"Once we had identified and rectified the problem we were able to restore full service at around 2.30pm... It was my decision as CEO to resolve the system outage on the new platform and not to roll back to the old Protx payment platform."
Although Protx said its service was back online by 1430 BST on Wednesday, a number of users are continuing to experience problems, according to emails from Reg readers and comments on Protx's forums. "I have been getting spurious transaction errors at all times of the day during and after this crisis was supposedly resolved from all kinds of cards," said Reg reader Paul.
Redirects from old URLs also caused problems on Wednesday but it was the timeout issues that caused the greatest headaches. Several readers reported problems with timeout errors - a problem that could lead to customers being billed twice for transactions, as Register reader Gareth explains.
"The main issue, from my perspective was that transactions were timing out. I feel that Protx do not see this as a failure. They are still processing transactions flat out, but their queues just backup and most vendors receive no response whatsoever. I think this is why they say their system was backup at 2:30pm. The problem for many is that after a minute or two they retry the TX [transaction] and the customer gets charged twice," he said.
Merchants tried to shield their customers from the effect of problems with the transaction processing system. Several reported sending stock out without taking payment, in the hope and expectation that payments would go through successfully later.
Protx said merchants who experienced system problems beyond Wednesday afternoon had only themselves to blame for not applying changes it had informed them about over the last six months. "As a result these merchants may have been unable to process domestic Maestro transactions and/or transactions through the Protx virtual terminal."
Protx users who've contacted us with problems involving lack of customer testing facilities and sliding timescales at Protx in the run up to last week's outage, as well as its aftermath, reacted angrily to this criticism. Paul, a contractor, told us: "How dare they risk my business and then claim that the issues I am now experiencing are my fault? Why was this done at 6am on a weekday - why not 3am on a Sunday morning as most normal financial institutions would have done?"
"We implemented changes as we were told. We also got an exemption from Streamline for the Maestro 3D requirement. The system was certainly defunct well after 2:30pm," Gareth told us.
Jumping the gun on 3D Secure
The upgrade would have been simpler if Protx had not made the leap to 3D Secure. Merchants were getting conflicting advice from Protx and their banks on when this system needed to be rolled out. The experiences of Reg reader Gary seem typical in this respect. "Barclay Merchant Services said it's not due to be implemented until July 2008 which is why they hadn't felt the need to inform us. Perhaps ProtX are jumping the gun a bit on this, possibly even being good guys when others are less prepared. Anyway, the whole situation is a shambles," he told us.
Protx chief exec Michael Alculumbre acknowledged that differing messages from it and banks on the issue caused confusion.
"I am embarrassed by the different and sometimes conflicting advice that many of you will have received from your acquiring bank and from Protx regarding 3D Secure registration and separate merchant accounts for MOTO and e-commerce transactions. I also realise that this must have been extremely frustrating and I would again apologise for any inconvenience that this has caused you," he writes.
"My staff did their utmost to ensure that Protx met bank requirements but we found this to be a significant challenge in trying to legislate and manage the banks' changing requirements where one rule would be applied for one merchant and another rule would be applied for a different merchant." ®