It's looking increasingly likely that the RBS "computer glitch" was actually a common-or-garden human error, as per usual, and specifically a failure of change control. Banks don't tend to use beta software, so it's unlikely the direct cause of this was a problem in a CA-7 patch. The length of time taken to resolve the issue (they've been knife-and-forking it for a week now) also supports the suspicion that the rollback plan failed last Tuesday. I think it safe to assume that they had a rollback plan, but that it was probably compromised by human error. What they clearly did not have was an effective contingency plan for such a failure. This is more surprising, given that their computing system is clearly well-established. All the actions and announcements to date look like they're working it out as they go along, so their crisis management is perhaps not up to snuff either.
A suggestion being made (though with only limited circumstantial evidence as yet), is that the human error occurred in a outsourced team in India. There is no secret that outsourcing increases risk where management mistakes a process requiring tacit knowledge for a commodity that can be contracted out for the lowest price. Given the criticality of the overnight batch jobs that apply the day's transactions to customer accounts, I'd be surprised if RBS did not control this through an inhouse operations team. I'd also be surprised if their change control team wasn't inhouse. It is more likely that the software maintenance team are outsourced, but this in turn implies loose control if a failed software release caused the job schedule to be trashed.
Unless you have a particular fascination with job-scheduling software (I'm really not that sad) then you probably don't think there is much to learn from this fiasco, though there is much incidental schadenfreude to enjoy in the spectacle of RBS chief Stephen Hester admitting he doesn't really know what is going on: "It is like, I don't know, the landing path at Heathrow or something, once you get out of sequences it takes some time to get back into sequence even if the original fault is put right". Hester turned down his £1m bonus for 2011 earlier this year following outrage that a taxpayer-owned entity should be continuing with such bonuses when the bank was losing money (£2bn in the year) and laying off thousands of staff. I don't fancy his chances of getting a bonus for 2012 now.
As I mentioned at the time, there is no reason to believe that Hester is any more aware of the real risks within the banking group he manages than his predecessor Fred Goodwin. Apparently, risk and control is one of the 5 key performance dimensions used to determine his bonus, which is what you would expect of a bank, though I'm sure it was there in Goodwin's day too. If the "drains up" review he has proposed shows that the contingency plan was inadequate, then he really has nowhere to hide. The really toxic revelation though would be evidence that the "financial results" performance dimension had come into conflict with the risk and control dimension. If the human error was a consequence of outsourcing, and thus UK redundancies, then Hester will be lucky to keep his job.
Where did I read that a large part of management is working out what you really do, and NOT outsourcing that?
ReplyDelete