Sign in to follow this  
Jamie123

Fujitsu, Horizon and the Post Office

Recommended Posts

(I just posted this as a round-robin e-mail at work, but some of you guys may be interested too)

This article is actually over a month old, but I only read it yesterday after web-searching about this week’s news of the Post Office/Fujitsu/Horizon appeals.

https://www.computerweekly.com/news/252496560/Fujitsu-bosses-knew-about-Post-Office-Horizon-IT-flaws-says-insider

(Just in case you’re unfamiliar with the story, about 20 years ago Fujitsu was commissioned to develop an IT system for the British Post Office. This was called “Horizon”. Shortly after it was installed, money started disappearing from branch accounts. Fujitsu insisted there was nothing wrong with their system, so the Post Office concluded that its own employees were stealing. Almost 1,000 were sacked, made bankrupt by repaying the money, or even sent to prison. Now it turns out the system was full of bugs, convictions are being overturned, and the Post Office is asking for a government bail-out to pay all the compensation that’s going to be claimed. Furthermore, Fujitsu executives who testified of the infallibility of their system are now likely to be prosecuted for perjury.) 

The “insider information” in this article could be the disgruntled ramblings of an angry ex-employee, so hopefully the government inquiry will get to the real truth of the matter. Nevertheless, I think there are some important lessons here that we should pass on to students:

  • The importance of formal methods in the high-level planning of a project, particularly a large project involving a large number of coders. Each developer should know exactly what his/her component of the system should do, and what it should not be allowed to do.
  • The importance of robust testing. Make sure each component of a system works correctly in isolation before connecting it to other components written by other developers. (This is what I was taught as a 1st year undergraduate learning Pascal. We always had to show evidence that each sub-program we wrote had been tested individually.)
  • The importance of continued vigilance. Even with the best formal methods and the best testing, there WILL still be bugs. It is NOT more likely that 900 previously honest postal employees suddenly turned criminal, than that there may be a bug in a system someone has told you is infallible. (And this applies even if that "someone" represents a big-brand name like Fujitsu.)

Happy Easter.

Edited by Jamie123
(I can't spell Fujitsu)

Share this post


Link to post
Share on other sites

He told Computer Weekly: “To my knowledge, no one on the team had a computer science degree or any degree-level qualifications in the right field. They might have had lower-level qualifications or certifications, but none of them had any experience in big development projects, or knew how to do any of this stuff properly. They didn’t know how to do it.”

This is not even remotely believable, yet is apparently true. I have no words.

Share this post


Link to post
Share on other sites
3 hours ago, Vort said:

He told Computer Weekly: “To my knowledge, no one on the team had a computer science degree or any degree-level qualifications in the right field. They might have had lower-level qualifications or certifications, but none of them had any experience in big development projects, or knew how to do any of this stuff properly. They didn’t know how to do it.”

This is not even remotely believable, yet is apparently true. I have no words.

Like I say, we don't know that it's true, but I can well imagine a narrative in which it might be true.

The Post Office puts out its invitation to tender, and several companies, including Fujitsu UK, decide to respond

Now there's a very high-up and ultra-ambitious "suit" in Fujitsu UK who I'll call Mr. Big Suit. (Maybe its a Ms. Big Suit, but I've a gut feeling it was a Mr.) Mr. Big Suit got promoted to where he is not by being particularly clever or competent (which he isn't) but because of a ruthless determination to get things done. He's a "mover and a shaker", and as such has often come in useful to his superiors.

Mr. Big Suit is determined to get the contact, and he will get the contact whatever it takes! 

He knows that with a brand name like Fujitsu behind him, he doesn't have to prove he can deliver the goods. But he does need to undercut the competition. So he instructs his underlings - his department heads - (with a variety of threats and promises) to quote the lowest price possible for each aspect of the project. He tells them that the quote needs to be low, and if he loses this contract because it's too high, he will not be pleased. At all. 

So Mr. Big Suit's underlings give him what he wants and Fujitsu wins the contract. They now need to deliver. They soon find that with the budget they have for this project, they can't afford to employ their best developers on it. So they put together a rag-tag team of assorted people whose salaries they can afford to pay. This rag-tag team does its best, and after a year or so has created a sort-of system which sort-of does the job, and sort-of doesn't. Mr. Big Suit now fears he's heading for a train wreck, so to cut his losses he finally does send in some of his top developers.

The top developers take one look at what the rag-tag team have created and throw up their hands in dismay. "This needs to be rewritten from scratch" they say.

"We haven't the time nor the money for that," says Mr. Big Suit. "What do you think I pay you for? Make it work. Or else!"

So the top developers do their best, but the deadline is looming and new bugs are still popping up like weeds. Their complaints to Mr. Big Suit fall on deaf ears. He has a deadline to meet, and if he doesn't meet it then some Mr. Even Bigger Suit at Head Office will have his hide.

So the top developers battle on. The discovery of bugs slows down, but as the clock strikes twelve no one really believes that they have all been found, but...

"Hey, we don't know that there are any more bugs. Maybe the one we found and corrected earlier this morning was the last!"

The system is delivered on time. Mr. Big Suit gets his bonus. Phew.

Then the problems start.

Postmasters across the country are reporting accounting errors. Fujitsu is consulted.

The news comes to Mr. Big Suit as he sits on the shaded balcony of his office overlooking an ornamental garden. For some months he's experienced a deeply-suppressed dread of this moment, and now it's finally come. But what can he do? Admit that he's made a complete dog's dinner of the entire project? What will that do for the Fujitsu brand name? Share prices will plummet! Thousands of jobs will be put at risk! It will be ALL his fault, and Mr. Even Bigger Suit will skin him alive!

He puts down his Martini (which no longer tastes so good) and thinks hard.

"Well, I did employ my best people," he says. "Only at the end mind you, but there's no need to stress that too much. And what were they doing for those last six months? They were testing! They were looking for bugs!" (Mr. Big Suit isn't exactly sure what a "bug" is, though he has vague a mental image of a beetle crawling around inside a computer chip, eating bits of wire here and there.) "That's what I'll tell them!"

Presently Mr. Big Suit (Fujitsu) sends his reply to Mr. Big Suit (Post Office). "Horizon has been subjected to six months of intensive testing by Fujitsu's top engineers, and we are confident in the product we have supplied." The first statement is 100% true, and the second is only a white lie. After all, who knows if the problems the Post Office is having are any fault of Fujitsu? Perhaps people aren't using it correctly. Perhaps...oh, perhaps anything! We really don't know. Let's play another game of golf and try not to think about it too much.

Meanwhile Mr. Big Suit (Post Office) is at a total loss. The figures are right in front of him in black and white. Money has gone missing. Fujitsu has assured him that they are not to blame. And they are Fujitsu after all. If its anything to do with computers they should know!

So what else could be the explanation? The famous words of Sherlock Holmes drift across his mind...

"Once you have eliminated the impossible, whatever remains, however unlikely, must be the truth!"

The rest is history.

Edited by Jamie123

Share this post


Link to post
Share on other sites

It dawns on me that I used to work for Fujitsu Consulting (a division which went belly up with the telecom bubble burst of 2002).  It was my second professional job out of college, and I spent a lot of my time trying to figure out what I was supposed to be doing.  I was absolutely not the best nor the brightest, and I was in constant amazement that someone as unskilled and inexperienced as I could be making the money I was.  But like I said, this was BEFORE the telecom bubble burst.  The Enron and Worldcom scandals hadn't hit yet.  Everyone was hiring anyone with a pulse.  We made jokes about the "consultant cannon", where you show up for a job interview, they load you into a cannon, and shoot you through some random high-rise window into a desk with a computer, where you became instantly billable.  

I remember working for a broadband internet company with the stated objective of bringing high-speed internet to rural areas.  I was a tester for their billing system.  We got raw data from a report, and had to check line-by-line against actual customer records, to see if the report was accurate.  The raw data included unencrypted credit card information.  We were just expected to not be crooks.  One day, some IT folks came up behind me and happily announced they had 'caught' a 'violator'.  My boss argued it out with them, and eventually won the notion that I was just doing the job I had been trained for.  From what I could tell from looking at the billing records, almost nobody was paying their bills.  The team in charge of turning off service due to nonpayment had all been reassigned to sexier jobs.  This was back in the day when telecom startups were assuming future revenue projections were as good as gold.  In the midst of all this chaos, we moved to a newly and expensively constructed office building, with on-site chair massages as one of the perks, even for consultants like me.  (This was the job where I had this incredibly powerful spiritual experience.) 

Then I went to work for Ma Bell, where I saw a project where a ton was invested in "dark fiber" (fiber cabling dug into the ground that wasn't expected to actually be used for anything any time soon).  Then Enron hit, then Worldcom hit, then 66% of the Telecom workers in Denver were on the street, including me.

For whatever reason, all my memories of those years include background music of a circus calliope.  

Edited by NeuroTypical

Share this post


Link to post
Share on other sites

Update: 39 of the appealing postal workers have just been cleared by the Court of Appeal Postmasters accused of theft by Post Office have had convictions overturned | ITV News

(There were a few cases where the judges concluded the convictions were safe, but these were cases which did not depend on IT evidence. I suppose that's only to be expected: even the Post Office can't get it wrong all the time!)

The important thing is that the judges allowed the appeals not only because the evidence was unreliable but because the convictions were an "affront to public conscience" (something the Post Office bosses were still contesting). That's going to sting. Now there'll be renewed pressure for a proper public enquiry, with evidence given under oath, where the Post Office and Fujitsu executives can't dodge the bullet by "declining to comment".

Edited by Jamie123

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this