CBOSS Dumpster Fire Update

The CBOSS development lifecycle,
as anticipated in 2009 on this blog.
Today we are at "point of no return."

The deadly crash of an Amtrak train near Tacoma, Washington, which would likely have been prevented if a PTC (Positive Train Control) system had been in place, has renewed the discussion of the status of PTC systems in the Bay Area. Caltrain officials say everything will be OK with CBOSS, Caltrain's very own flavor of PTC. Despite those assurances, a potent brew of ingredients is mixing together.

Bonfire of Lawsuits: After a well-chronicled program failure involving delays, cost overruns, and failure to meet milestones, Caltrain terminated the CBOSS prime contractor, Parsons Transportation Group, in February 2017. PTG and Caltrain promptly sued each other, with PTG claiming wrongful termination and Caltrain seeking up to $98 million in damages. A rich trove of documents can be accessed online under San Mateo Superior Court case number 17CIV00786, and chronicles in detail everything that went wrong with the CBOSS program. With Caltrain likely to recover some damages, PTG has now sued Alstom (formerly PTG's subcontractor and the supplier of CBOSS hardware and software) for failure to deliver a working solution. One is left to wonder how this motivates Alstom to finish the CBOSS project, since delivering a working solution to Caltrain would undermine the claim that Alstom was given an impossible task.

Dying Product: The hardware and software underlying CBOSS is known as I-ITCS, a product originally developed by GE Transportation Systems Global Signalling. While a precursor known as ITCS briefly operated on Amtrak corridors in Illinois and Indiana, it is now being displaced by the de-facto standard freight PTC system known as I-ETMS, with ITCS relegated to controlling only the grade crossing functionality in these corridors. Alstom, which acquired GE Transportation Systems in 2015, is not likely to see a future in the I-ITCS product, leaving Caltrain with a globally unique hardware and software solution. This does not bode well for product support over the lifetime of CBOSS.

Looming Deadline: the deadline imposed by Congress and the Federal Railroad Administration to successfully complete a PTC revenue service demonstration is just a year away, at the end of 2018. One year is not enough to finish, and Caltrain will almost certainly blow this deadline. Will FRA grant another extension and allow Caltrain to continue operating without PTC?

Sole Source Savior: in July 2017, avionics firm Rockwell Collins' subsidiary ARINC was awarded a sole source contract to figure out what it will take to pick up the pieces and complete the CBOSS project. ARINC completed this assessment in September, and will soon (by sheer programmatic necessity, since failure is not an option) be awarded a name-their-price sole source contract to finish a minimally working version of CBOSS that passes FRA muster. With the leverage that ARINC enjoys under these circumstances, the "re-procurement" of CBOSS will likely be (1) expensive and (2) structured such that Caltrain bears all of the risk of continued failure, i.e. cost-plus-fixed-fee rather than fixed price. With the clock ticking, the re-procurement effort has already fallen behind the planned fall 2017 schedule.

Budget Crunch: To date, Caltrain has spent over $200 million (yes, one fifth of a billion dollars!) on CBOSS with nothing to show for it. All the money allocated for CBOSS is spoken for, and a lot more (several tens of millions) will be needed to finish the project. Some of that will come from damages, but it is quite likely that 2018 will bring emergency financial maneuvers to throw more good money after bad.

Descoping of Functionality: while the first 'I' in Caltrain's I-ITCS solution stands for "Interoperable," which was one of the original selling points of CBOSS, this feature is now being thrown over the transom. Interoperability requirements contributed to the scope creep that triggered a re-design of the supposedly off-the-shelf ITCS software. It didn't help that Union Pacific was (as per usual) actively non-cooperative in helping to develop an interoperable solution, leading to Caltrain throwing in the towel and spending an additional $21.7 million (from an FRA "interoperability grant," no less!) to dual-equip seven diesel consists with the I-ETMS freight PTC system for operating on the Gilroy branch owned by UPRR. How I-ETMS freight trains will be accommodated on the peninsula corridor in I-ITCS territory is a burning question, for which the range of answers includes ditching I-ITCS and replacing it with the more viable I-ETMS, following the Amtrak example.

System Integration and Testing is Hard: while Caltrain never fails to remind us that all of the components of CBOSS are physically installed on the trains and the tracks, that is the easy part. The hard part is getting everything to operate together reliably every day, and Caltrain and their shifting band of contractors are barely getting started on this most difficult phase of the development of a new and complex safety-critical system. Integration and Testing is where the best design intentions meet cold harsh reality, and all the mistakes and omissions made during the design phase become painfully apparent. While PTG claimed in court filings that they were 90% done with CBOSS when their contract was abruptly terminated, that last 10% of troubleshooting commonly takes far more than 10% of the budget or schedule.

PTC is Hard: the legal declarations from PTG managers who ran the CBOSS program (see 17CIV00786) reveal a long list of underlying factors that caused much acrimony and remain unchanged today: (1) the specifications and standards for PTC continue to evolve, triggering continued changes and penalty testing; (2) Caltrain and its in-house consultants (the so-called "owner's team") are woefully ill-equipped and uncoordinated in their approach to complex safety-critical avionics technology development; (3) the formal contractual interactions between the "owner's team" and the vendor are complicated and delay-prone; (4) working with UPRR is a huge pain in everyone's caboose; (5) the underlying systems over which CBOSS is supposed to "overlay" are kludged-together stove pipes that, incidentally, will require nearly total re-design for the electrification program; (6) testing PTC on an operating railroad requires extensive coordination that has been demonstrated to be lacking; and so on. Strike PTG and substitute ARINC.

These ingredients will produce a situation where CBOSS does less than was promised, later than planned, and for a lot more money. No crystal ball is needed to predict that CBOSS will continue to "fail forward" to a finish line somewhere beyond 2018.