25 December 2017

CBOSS Dumpster Fire Update

The CBOSS development lifecycle,
as anticipated in 2009 on this blog.
Today we are at "point of no return."
The deadly crash of an Amtrak train near Tacoma, Washington, which would likely have been prevented if a PTC (Positive Train Control) system had been in place, has renewed the discussion of the status of PTC systems in the Bay Area. Caltrain officials say everything will be OK with CBOSS, Caltrain's very own flavor of PTC. Despite those assurances, a potent brew of ingredients is mixing together.

Bonfire of Lawsuits: After a well-chronicled program failure involving delays, cost overruns, and failure to meet milestones, Caltrain terminated the CBOSS prime contractor, Parsons Transportation Group, in February 2017. PTG and Caltrain promptly sued each other, with PTG claiming wrongful termination and Caltrain seeking up to $98 million in damages. A rich trove of documents can be accessed online under San Mateo Superior Court case number 17CIV00786, and chronicles in detail everything that went wrong with the CBOSS program. With Caltrain likely to recover some damages, PTG has now sued Alstom (formerly PTG's subcontractor and the supplier of CBOSS hardware and software) for failure to deliver a working solution. One is left to wonder how this motivates Alstom to finish the CBOSS project, since delivering a working solution to Caltrain would undermine the claim that Alstom was given an impossible task.

Dying Product: The hardware and software underlying CBOSS is known as I-ITCS, a product originally developed by GE Transportation Systems Global Signalling. While a precursor known as ITCS briefly operated on Amtrak corridors in Illinois and Indiana, it is now being displaced by the de-facto standard freight PTC system known as I-ETMS, with ITCS relegated to controlling only the grade crossing functionality in these corridors. Alstom, which acquired GE Transportation Systems in 2015, is not likely to see a future in the I-ITCS product, leaving Caltrain with a globally unique hardware and software solution. This does not bode well for product support over the lifetime of CBOSS.

Looming Deadline: the deadline imposed by Congress and the Federal Railroad Administration to successfully complete a PTC revenue service demonstration is just a year away, at the end of 2018. One year is not enough to finish, and Caltrain will almost certainly blow this deadline. Will FRA grant another extension and allow Caltrain to continue operating without PTC?

Sole Source Savior: in July 2017, avionics firm Rockwell Collins' subsidiary ARINC was awarded a sole source contract to figure out what it will take to pick up the pieces and complete the CBOSS project. ARINC completed this assessment in September, and will soon (by sheer programmatic necessity, since failure is not an option) be awarded a name-their-price sole source contract to finish a minimally working version of CBOSS that passes FRA muster. With the leverage that ARINC enjoys under these circumstances, the "re-procurement" of CBOSS will likely be (1) expensive and (2) structured such that Caltrain bears all of the risk of continued failure, i.e. cost-plus-fixed-fee rather than fixed price. With the clock ticking, the re-procurement effort has already fallen behind the planned fall 2017 schedule.

Budget Crunch: To date, Caltrain has spent over $200 million (yes, one fifth of a billion dollars!) on CBOSS with nothing to show for it. All the money allocated for CBOSS is spoken for, and a lot more (several tens of millions) will be needed to finish the project. Some of that will come from damages, but it is quite likely that 2018 will bring emergency financial maneuvers to throw more good money after bad.

Descoping of Functionality: while the first 'I' in Caltrain's I-ITCS solution stands for "Interoperable," which was one of the original selling points of CBOSS, this feature is now being thrown over the transom. Interoperability requirements contributed to the scope creep that triggered a re-design of the supposedly off-the-shelf ITCS software. It didn't help that Union Pacific was (as per usual) actively non-cooperative in helping to develop an interoperable solution, leading to Caltrain throwing in the towel and spending an additional $21.7 million (from an FRA "interoperability grant," no less!) to dual-equip seven diesel consists with the I-ETMS freight PTC system for operating on the Gilroy branch owned by UPRR. How I-ETMS freight trains will be accommodated on the peninsula corridor in I-ITCS territory is a burning question, for which the range of answers includes ditching I-ITCS and replacing it with the more viable I-ETMS, following the Amtrak example.

System Integration and Testing is Hard: while Caltrain never fails to remind us that all of the components of CBOSS are physically installed on the trains and the tracks, that is the easy part. The hard part is getting everything to operate together reliably every day, and Caltrain and their shifting band of contractors are barely getting started on this most difficult phase of the development of a new and complex safety-critical system. Integration and Testing is where the best design intentions meet cold harsh reality, and all the mistakes and omissions made during the design phase become painfully apparent. While PTG claimed in court filings that they were 90% done with CBOSS when their contract was abruptly terminated, that last 10% of troubleshooting commonly takes far more than 10% of the budget or schedule.

PTC is Hard: the legal declarations from PTG managers who ran the CBOSS program (see 17CIV00786) reveal a long list of underlying factors that caused much acrimony and remain unchanged today: (1) the specifications and standards for PTC continue to evolve, triggering continued changes and penalty testing; (2) Caltrain and its in-house consultants (the so-called "owner's team") are woefully ill-equipped and uncoordinated in their approach to complex safety-critical avionics technology development; (3) the formal contractual interactions between the "owner's team" and the vendor are complicated and delay-prone; (4) working with UPRR is a huge pain in everyone's caboose; (5) the underlying systems over which CBOSS is supposed to "overlay" are kludged-together stove pipes that, incidentally, will require nearly total re-design for the electrification program; (6) testing PTC on an operating railroad requires extensive coordination that has been demonstrated to be lacking; and so on. Strike PTG and substitute ARINC.

These ingredients will produce a situation where CBOSS does less than was promised, later than planned, and for a lot more money. No crystal ball is needed to predict that CBOSS will continue to "fail forward" to a finish line somewhere beyond 2018.

18 comments:

  1. They really should've just thrown the whole thing out and started over with ACSES (or I-ETMS), since they have to completely overhaul their track circuits anyway for electrification. I just hope they don't end up with a mess like Denver with grade crossings not operating reliably.

    ReplyDelete
    Replies
    1. Any word on what the technical problems are that have caused their issues? All I’ve ever heard is that the gates don’t come at the exact right time, and stay down too long. They’ve been noticeably mute on the specifics, probably because it is a really simple and stupid glitch, but complicated by small-use proprietary software.

      Delete
    2. Denver's PTC system (I-ETMS) apparently out-smarts itself in the presence of real-world factors like station dwells, train handling, dispatcher intervention, etc., resulting in excessive down time, which sometimes triggers unsafe motorist behavior. Shout-out to Reality Check for providing links below:

      Here is an overview of the Denver grade crossing situation.

      Here is a more detailed briefing on the grade crossing issue to the RTD board (starts page 4).

      RTD's contractor says they're done, that's just how the technology works, kthxbye.

      CBOSS, of course, was selected because of the high number of grade crossings on the peninsula. One of its marquee selling points was supposed to be the ability to keep a grade crossing on the far side of a station platform (such as in Menlo Park or Burlingame) inactive if a train was stopping. The odds of that feature ever working are of course pretty remote given that ARINC is being tasked with the bare minimum.

      The grade crossing issue is currently the top risk (see Appendix F, page 141) for the Peninsula Corridor Electrification Program.

      Delete
  2. Let's be clear, it isn't long gate downtime that "triggers" unsafe motorist behavior. The problem is dangerous idiots who should never have been permitted to drive.

    Step 1: install cameras
    Step 2: mandatory $1,000 fine and license suspension for violators

    In Germany, the penalty is 700 euros, 2 points, and 3-month license suspension (similar severity as a DUI).

    ReplyDelete
  3. @Drunk: it makes sense that the longer that drivers see gates are lowered without a train passing through, the likelihood of them driving around the gates increases. This is what the Sept 29, 2017 University of Colorado A Line and B Line Grade Crossing Update (PDF/packet page 19) states:

    Effects of Warning Time on Driver Behavior and Safety at Grade Crossings Study
    • Warning Times in Excess of 30-40 seconds caused many more drivers to engage in risky crossing behavior.
    • Most drivers expect a train to arrive within 20 seconds from the moment the activation begins.
    • Drivers begin to lose confidence in the traffic control system if the warning time exceeds approximately 60 seconds.
    • Suggested warning time guidelines (20 sec minimum, 25-35 sec desirable, 60 sec maximum)
    • Four-quadrant gates should be used if 10% of warning times exceed 60 sec.
    • Reference: Assessment of Warning Time Needs at Railroad Highway Grade Crossings with Active Traffic Control, Stephen H. Richards and K.W. Heathington, Transportation Research Record 1254

    ReplyDelete
  4. The "excess" gate downtime is all of 20 seconds. If that is enough to "trigger" motorists, then God help us.

    Note also that traffic engineers are perfectly fine with extending the red-time at traffic lights by much more than that, without any regard to how much it will increase jaywalking. It is quite the double standard.

    ReplyDelete
    Replies
    1. @Drunk: have a look at the Crossing Warning Time vs. Frequency of Value graph (pg. 16) for the 4-track (one pair for UP and another pair RTD) Dahlia crossing. It shows that for 1893 RTD crossing activations (presumably UP activations were ignored) in a 2-week period, that while the majority were in the 32-42 second range, well over one hundred were in the 50-70 second range. While 33 seconds was the most common (300 out of 1893) warning time, you've still got at least a dozen of 1 minute or more. Weird. And hardly constant time warning.

      As for what's enough to "trigger" a motorist ... consider that nearly 100% of humans who try, eventually manage to get a driver's license (or learner's permit) ... including (and especially) high school kids.

      Delete
    2. You can cut and dice the data all you want. According to the PUC, gates are down a mere 20 seconds longer than specified in Federal regs. Not even the FRA gives a damn (they granted a 5-year waiver).

      Drivers have no problem waiting several minutes (minutes!!) at a normal red light, but somehow turn into crazed lunatics if forced to wait a few extra seconds at a crossing gate.

      Delete
    3. I'm not taking sides, and I agree there's a big double standard in terms of how occasionally waiting a minute or less for a train to pass at lowered crossing gates is somehow perceived as way more unacceptable than the much longer and more frequent wait for almost any red light at a (multi-way) signalized road intersection. (Nevermind that the train is often carrying far more people than all the cars during a complete traffic light cycle at a signalized road intersection.)

      However, while motorists are fairly tolerant of waiting for passing traffic, they even get antsy at traffic lights when they're facing a long red with an empty intersection devoid of cross-traffic (e.g. late at night, etc.). This is a normal characteristic of RR crossings, there's a wait with nothing happening (ie. seemingly waiting for "nothing") until there's something: a train zips by and then its over.

      Delete
    4. We can argue every which way about how motorists should behave, but these systems have to be designed around how they do behave.

      Delete
  5. For the amount of money that's been wasted on CBOSS thus far, we could be well on our way to grade-separating the handful of station-adjacent grade crossings where that particular feature would actually matter.

    ReplyDelete
    Replies
    1. @Joey, depends on how you define "well on our way."

      Here's a bit of a reality check on that claim:

      Broadway in Burlingame is one such crossing, and the city approved design for that single grade separation is being estimated at around $250m:


      [Burlingame Public Works Director] Syed Murtuza said he expects the early engineering approved Monday in the deal with the San Mateo County Transit Authority and Peninsula Joint Powers Board to be completed in 18 months. Funding for the work is fully secured and includes $3.5 million grant from the Transit Authority, plus a previously approved $500,000 contribution from Burlingame.

      Looking ahead though the path is much murkier, said Murtuza, who suggested ground may not break on constructing the separation project until 2025 — and that’s assuming all $250 million needed to finish the currently unfunded project is landed.

      “I could see this project going to construction by 2025, if we have all the money in the world. And that’s a big if,” said Murtuza.

      Delete
    2. It doesn't cost anything to close the useless Broadway station.

      Delete
    3. @Drunk, indeed. But the point is that the excess spend for CBOSS may not even pay for one grade sep at those rates.

      Separately, and as you may be aware, Caltrain and the respective cities keep talking about how the peppier EMUs that will accompany electrification will allow Broadway and Atherton to be reopened for weekday service.

      Delete
    4. San Bruno was $155m, 25th Ave is targeted at $180m, so yeah, I guess it's only about the cost of 1 grade separation or so.

      Still though, I wonder if it's worth spending money on such a feature given the inevitability of grade separations. On the other hand, some of the remaining stations with this issue will be rather complex to grade separate (e.g. San Mateo) so it could be a while before they're all done.

      Delete
  6. OT question: while ACSES is listed as an Alstom product, the top result for "ACSES II" is a Siemens website: https://w3.usa.siemens.com/mobility/us/en/Events/railway-interchange/Documents/SIE_BRO_ACSES%20II.pdf

    Did some transaction happened for ACSES-compatible PTC systems? Considering regulatory approval hasn't been granted for Alstom and Siemens merging, it's not likely Siemens started listing Alstom products under its name, or not?

    ReplyDelete
  7. Didn't someone predict all this 9 years ago?
    http://caltrain-hsr.blogspot.com/2009/10/peninsula-train-control-ptc-cboss-and.html

    ReplyDelete