Boeing 737 Max in the News Again.

 I've waited a few days to get the salient information on the latest Boeing 737 issue, the blow-out in flight of a "plug door" on an Alaska Airlines Max 737-9.

The plug door in question is not actually a door. It's a plug in a door-shaped hole. The Max-9 has different variants with different seating arrangements, but to save assembly complexity and cost the same fuselage. The Fuselage has the same number of openings, but not all of them are used.

The Max-9 can seat a maximum of 230 passengers. On higher density versions with over 189 seats the door just behind the wings is actually a fully working emergency exit. On variants with seating for under 189 passengers, the emergency door is not required and instead the door shaped hole is plugged by... a plug.

The plug panel has a window in it and it is firmly held in place. On the inside of the aircraft the plug is covered by a panel with just a window in it. The only thing you'd notice inside the aircraft would may be a difference in the spacing of the window compared to others. 

The plug is not supposed to be removed, although it can be removed with difficulty for maintenance. To open the plug, you need to remove four locking bolts that firmly stop any action of the door to open. Once the bolts are removed  The bolts are torqued to spec and have castellated nuts. The nut then have cotter or split pins inserted so the nut cannot loosen.

Anyone that knows castellated nuts and split pins, know it a pretty secure method for holding a nut on a bolt. The main failure mode of split pins is metal fatigue by reusing old pins. But everyone knows you ALWAYS fit a brand new pin....right?

What seems to have happened is all 4 bolts have removed themselves from the door so it was able to release itself from the fuselage in flight.

The FAA have instructed Mazx-9 operators to check the plug door installations. It's already been reported that loose bolts have been found. Not just the locking bolts that prevent the plug from opening, but also bolts holding the hardware to the door and/or fuselage.

Now, the issue has been identified, the issue needs to be tracked back to where the bolts failed to be tightened and inspected correctly. Was it a Boeing after the fuselage assemblies had been delivered, or was it at Spirit where the fuselages were manufactured?

With two variants with two different door assemblies, were the assembly instructions up to scratch covering both options? Was there an issue with the instructions that allowed a plug door to be fitted incorrectly? For instance was a check sheet not followed for the locking bolts? For instance the emergency exit option doesn't have them, so was there enough clarity in the assembly instructions that on the plug option that the locking bolts should be fitted, torqued and split pins fitted and the installation checked before the cabin trim was installed over the plug?

Was there a oversight in the instructions that allowed a plug door to not be assembled?

I can imagine a scenario where due to assembly pressures the interior trim was installed before the plug door bolts were properly installed and checked, but the assembly instructions said once the trim was installed there was no need to check the door, because it was assumed that at that point the door had been checked and inspected. I just wonder if there was a gaping hole in the plug door inspection sign-off process that allowed the trim to be installed before the door inspection, unless someone falsified the inspection records..... That would be a pretty severe lapse in safety though.

All speculation at the moment, but having experience of manufacturing processes, I can imagine a few scenarios where assembly instructions didn't clearly indicate the hierarchy of processes i.e. that the interior trim should not be installed until the door was checked and signed off. 

It there even was a process to check and sign off the door installation. Maybe it wasn't classed as a safety issue because it wasn't a safety exit? Maybe it didn't receive the appropriate scrutiny?

Another issue is were aviation experts involved in the writing of the assembly instructions? Were they aware of the severity of an in-flight door loss? Did they have enough experience, instruction or training to understand the repercussions of an insecure door plug?

The final aspect could be that the door didn't fit the frame correctly. That's a slim possibility because it's already been established the locking bolts are missing on the Alsaka Airplanes aircraft, but not impossible. There may not have been instructions concerning the fit of the door in the frame. It may have just been assumed it would be an accurate fit and there were no instructions regarding clearances between the plug and the frame.

Who knows. Only time will tell, but I get the impression so far that Boeing and Spirit will be let off once again with a slap on the wrist and no meaningful sanctions will be applied for this lapse in safety. 

UPDATE 25/03/2024:

Looks like I was pretty close to the mark on this one. The plug in question was opened for remedial work and then closed again. Somehow the plug was closed without the important locking bolts being fitted at all. Even worse, an employee took a picture of the plug without the bolts in place, not realising the importance of the locking bolts to the safety of the aircraft!

So was the procedure to do the repair faulty? Was the step to replace the locking bolts not included? The employee looks to be happy to take a picture without the important bolts in place, so were they a member of the team that fitted the interior trim and weren't aware of the importance of the bolts? Did they even know of the existence of the bolts?

Boeing have allegedly failed to produce sufficient information regarding the closure of the door plug to the FAA. What appears to signal is either a lock of instruction for a procedure that wasn't planned for or a malicious disregard of instructions and a "just wing it" attitude at the factory.

I was wrong on the consequences for Boeing employees though. The CEO has just gone and board positions are being reshuffled. We'll see how that impacts aircraft safety. I suspect the effect will be minimal. 

Stephanie Pope becomes the new CEO with immediate effect. With her coming from the financial and services sector, sadly it appears to confirm the emphasis on the board at Boeing is not engineering. 

The Biggest Software Scandal in the UK: The Horizon Postal System.

 At work there's a big hubbub about the series "Mr Bates Vs The Post Office", the series that relates the sad tale of the Royal Mail Postmasters/mistresses that were prosecuted for missing funds. Funds that were not missing, but inaccurately calculated by a faulty software system called Horizon.

I was aware of it from the beginning and was stunned at the viciousness of the prosecutions based on what exactly? A computer program that was clearly faulty?

I talked in my previous post about poorly written code that uses a subroutine maybe once in a blue moon under very special circumstances that never gets tested and could continue to harbour faulty code unless the test regime was very strict and tested every single scenario.

That's what I suspect happened in the case of the Horizon system: that under normal day-to-day operations there wasn't an issue, but the unfortunate postal workers that created a rare scenario that used the faulty code came across the issue and through no fault of their own suffered from the effects of bad code.

That the people in charge of the system didn't start to doubt the system when more people started to report issues was a black mark against them. Certainly there was poor management of the situation at the middle management level and above.

It's one thing to ignore issues and possibly lie to cover up bad code, but when it gets to the level of court cases, prosecutions and jail time, the software must not be trusted. It needs to be forensically checked to confirm if the software is doing what it's supposed to do and it shouldn't be assumed that the code is perfect. It very rarely is. 

There's always the chance of an undetected bug in the code and it's the responsibility of the software department to investigate thoroughly. It should also be the remit of the court to bring in independent investigators to check the accuracy of the code and run through the scenarios the people accused of fraud went through, to eliminate software errors. 

It's pretty damning that the Post Office covered up the issues when they knew they were there. When people in the Post Office couldn't themselves make the system work like it should.

The people involved in managing the situation should hang their heads in shame. Those at the top should be prosecuted for misuse of public office. The fact that the Post Office has extraordinary powers to criminally prosecute people in court without the Crown being involved should also mean that those in charge have a duty to not misuse those powers and should themselves face prosecution if they do.

Not Being Aware of How Good You Are....

 We've just had a new computer system go in at work. Lets say that it works, sort of, but it leaves a lot to be desired and there is a lot of tweaking that isn't tweaking. Stuff that should have been caught during testing.

It reminds me of back when I wrote applications for Blue-Chip companies. I wrote the first HSBC telephone banking application, the first B.T. 150/151 call steering application, and managed installations and upgrades for the likes of Goldman-Sachs and Commerzbank. 

Had 9/11 not happened I might be still doing the same sort of job. Actually I might have been retired by now. But 9/11 sort of changed my aspect on family and rather than fly all over the world and rake in the money, instead I concentrated on the family and my disabled son. The IR35 tax rules sort of provided a perfect storm of circumstances that persuaded me it wasn't worth the hassle.

It's interesting and a little frustrating to watch other people's programming standards. I get that this application is a little website and associated back end system, no -one is going to die or lose millions if it doesn't work correctly, but it just annoys me a bit that other people aren't as fastidious about their adherence to good practice. 

For instance there's supposed to be a functional spec, but none of us users have seen it, so we've not had any input on the way the application works for example. That's been done and signed off by someone that has never visited us and seen how we work. Instead we've been presented with an application that accepts orders from a website and that's about it. All the "Fun" stuff like cancelling orders for customers before shipping, or amending the order before shipping? Nah, not there. Oh, you want to create orders for foreign customers and send them details in advance so they can pay by PayPal outside the system? Oh, we didn't know you did that. 

Using an off-the-shelf open-source system to create the website and all the back end functionality isn't the best option and I get it, there are limitations. But if you can't program round the limitations of the platform, then really it isn't up to the job. But then that's the job of the Project manager to assess the suitability of the chosen platform and if necessary change it. It's a bit strange selecting a platform and then bludgeoning it to suit your requirements. That reminds me of the HSBC job, where the original contractor writing the program coded himself into a corner and swanned off to the other side of the World at short notice. I stepped in and sorted that one out with a bit of lateral thinking. That earned me 70K.

I was just struck this morning that as an independent IT contractor back ooh, 23 years ago, that I must have been working at a heck of a level and never realised it. For instance working for a bank: when you have £4m public liability insurance, that would cover maybe a slow couple of hours of downtime if something you did caused the bank's systems to go down. That sort of focusses you on being definitely on the money and delivering a system that works.

I wrote applications for British Telecom, I managed the first successful upgrade of the Goldman-Sachs call-recording system that Thales had installed. That was a £14m project. 

I just didn't realise at the time how big a deal it was. 

My colleagues are currently highly focussed on IT issues by this install and are also now watching the new TV series about the False Accusations of Fraud created by the Horizon System used by the Post Office. I used to work for ICL that eventually became Fujitsu, so I was aware of the Post Office debacle as it happened in real time and I could never understand how the victims could never get a forensic analysis of the Horizon code done in order to identify any issues in the code.

It's arrogant to assume a system is foolproof and error free. There are many examples of unintended consequences where you write a bit of code and it inadvertently affects the code somewhere else in the system. I always tried to boil my applications down to a known set of subroutines that do specific tasks. The processes are built up by calling each subroutine in turn so that if there is an error in a subroutine, it happens everywhere that subroutine is called and it's easily and quickly identified and dealt with.

But I have seen some sloppy code where every process is a new routine, so it's entirely possible that in some rare instance a routine is called that is incorrect, but it's called so rarely, under special circumstances that it's virtually impossible to identify the issue and rectify it. That is, unless you have a VERY accurate test regime that tests for EVERY option and has the correct result already in the test results, so the error can be spotted.

But those were the days, co-ordinating huge call centres to stress-test my applications, Fully documented test regimes and pretty flawless applications. My last one at Carlsson Marketing ran so well we kept getting calls and visits from senior staff asking if it was actually live, because they'd had no complaints, no errors, no issues. The app kept on doing what it was supposed to be doing even if customers tried to break it. 

And that's what I liked the most: coding for exceptions and issues and making sure they were handled in a correct manner, making the computer handle the issue politely, if you like. After all a phone line can go down, the link to a server can fail, anything could happen. It's up to the programmer to code for the unexpected and handle it in a manner that doesn't crash the system and informs the human using the system exactly what has happened and why.