“Although RBC Financial Group finally outlined details of the technology failure that for days kept money out of millions of Canadian bank accounts, a marketing expert warned the bank has only days to do a better job of smoothing things over with customers,” writes Fawzia Sheikh in Friday’s issue of IT Business.ca.

“The bank said it will determine both the cost impact to its business and possibility of employee dismissals once it reviews the cause of what’s now being called a human error and the steps taken to clean up the mess.”

“IBM has been asked to conduct an independent review to find these answers, but for reasons of contractual confidentiality, a company spokeswoman said she could not provide specific details about how it will help RBC avoid a similar situation or case study examples of how its business continuity and recovery program has assisted other clients. RBC spokespeople refused comment, directing media enquiries to information posted on its Web site late Wednesday.”

“For the most part, data recovery experts like Ron Austin, vice-president of business development at ActionFront Data Recovery Labs in Markham, Ont., believed the “situation was handled quite well internally.”

” ‘Typically, what really compounds any IT problem is panicking by system administrators and the database admistrators and the people that are on the front lines to keep things running and address problems,’ he said. ‘It may have taken them a couple of days to do it simply because they weren’t panicking. They were taking a step-by-step approach and being quite conservative.’ “

“On its Web site, RBC explained a mistake was made in a software programming change that was corrected within two hours. But correcting customers’ bank account balances took longer because the bank had to stop production until it made sure the error couldn’t harm other systems. Once production re-started, RBC was held up again as it was forced to process two days of banking transactions on the same date.”

“Adding to the backlog was the requirement that all transaction dates be manually checked. RBC said it decided to give priority to payroll transactions.”

“The bank said it didn’t resort to its backup system once the problem was discovered because this secondary facility functions only if the primary system isn’t working. Since RBC policy dictates that program changes be made at the same time to both systems, its back-up facility would not have resolved the problem. In the future, Canada’s largest bank said it may make its backup facilities ‘more robust.’ “

“RBC’s practice of making simultaneous programming upgrades on its main and secondary systems is ‘a little unusual or it may be unusual in my experience, simply because if you start an upgrade on the main system, often you want to have a backup that’s untouched so that you can revert to it,’ said Austin. ‘When they say “more robust,” I assume maybe they’re going to add another layer of back-up . . . that’s always left alone.’ “

“Bill Margeson, general manager of CBL Data Recovery in Markham, said it would have been difficult to foresee problems occurring on both systems. Describing putting in a second backup system as ‘a little bit of overkill,’ Margeson said RBC will probably be forced to go down this path.”