25 December 2017

CBOSS Dumpster Fire Update

The CBOSS development lifecycle,
as anticipated in 2009 on this blog.
Today we are at "point of no return."
The deadly crash of an Amtrak train near Tacoma, Washington, which would likely have been prevented if a PTC (Positive Train Control) system had been in place, has renewed the discussion of the status of PTC systems in the Bay Area. Caltrain officials say everything will be OK with CBOSS, Caltrain's very own flavor of PTC. Despite those assurances, a potent brew of ingredients is mixing together.

Bonfire of Lawsuits: After a well-chronicled program failure involving delays, cost overruns, and failure to meet milestones, Caltrain terminated the CBOSS prime contractor, Parsons Transportation Group, in February 2017. PTG and Caltrain promptly sued each other, with PTG claiming wrongful termination and Caltrain seeking up to $98 million in damages. A rich trove of documents can be accessed online under San Mateo Superior Court case number 17CIV00786, and chronicles in detail everything that went wrong with the CBOSS program. With Caltrain likely to recover some damages, PTG has now sued Alstom (formerly PTG's subcontractor and the supplier of CBOSS hardware and software) for failure to deliver a working solution. One is left to wonder how this motivates Alstom to finish the CBOSS project, since delivering a working solution to Caltrain would undermine the claim that Alstom was given an impossible task.

Dying Product: The hardware and software underlying CBOSS is known as I-ITCS, a product originally developed by GE Transportation Systems Global Signalling. While a precursor known as ITCS briefly operated on Amtrak corridors in Illinois and Indiana, it is now being displaced by the de-facto standard freight PTC system known as I-ETMS, with ITCS relegated to controlling only the grade crossing functionality in these corridors. Alstom, which acquired GE Transportation Systems in 2015, is not likely to see a future in the I-ITCS product, leaving Caltrain with a globally unique hardware and software solution. This does not bode well for product support over the lifetime of CBOSS.

Looming Deadline: the deadline imposed by Congress and the Federal Railroad Administration to successfully complete a PTC revenue service demonstration is just a year away, at the end of 2018. One year is not enough to finish, and Caltrain will almost certainly blow this deadline. Will FRA grant another extension and allow Caltrain to continue operating without PTC?

Sole Source Savior: in July 2017, avionics firm Rockwell Collins' subsidiary ARINC was awarded a sole source contract to figure out what it will take to pick up the pieces and complete the CBOSS project. ARINC completed this assessment in September, and will soon (by sheer programmatic necessity, since failure is not an option) be awarded a name-their-price sole source contract to finish a minimally working version of CBOSS that passes FRA muster. With the leverage that ARINC enjoys under these circumstances, the "re-procurement" of CBOSS will likely be (1) expensive and (2) structured such that Caltrain bears all of the risk of continued failure, i.e. cost-plus-fixed-fee rather than fixed price. With the clock ticking, the re-procurement effort has already fallen behind the planned fall 2017 schedule.

Budget Crunch: To date, Caltrain has spent over $200 million (yes, one fifth of a billion dollars!) on CBOSS with nothing to show for it. All the money allocated for CBOSS is spoken for, and a lot more (several tens of millions) will be needed to finish the project. Some of that will come from damages, but it is quite likely that 2018 will bring emergency financial maneuvers to throw more good money after bad.

Descoping of Functionality: while the first 'I' in Caltrain's I-ITCS solution stands for "Interoperable," which was one of the original selling points of CBOSS, this feature is now being thrown over the transom. Interoperability requirements contributed to the scope creep that triggered a re-design of the supposedly off-the-shelf ITCS software. It didn't help that Union Pacific was (as per usual) actively non-cooperative in helping to develop an interoperable solution, leading to Caltrain throwing in the towel and spending an additional $21.7 million (from an FRA "interoperability grant," no less!) to dual-equip seven diesel consists with the I-ETMS freight PTC system for operating on the Gilroy branch owned by UPRR. How I-ETMS freight trains will be accommodated on the peninsula corridor in I-ITCS territory is a burning question, for which the range of answers includes ditching I-ITCS and replacing it with the more viable I-ETMS, following the Amtrak example.

System Integration and Testing is Hard: while Caltrain never fails to remind us that all of the components of CBOSS are physically installed on the trains and the tracks, that is the easy part. The hard part is getting everything to operate together reliably every day, and Caltrain and their shifting band of contractors are barely getting started on this most difficult phase of the development of a new and complex safety-critical system. Integration and Testing is where the best design intentions meet cold harsh reality, and all the mistakes and omissions made during the design phase become painfully apparent. While PTG claimed in court filings that they were 90% done with CBOSS when their contract was abruptly terminated, that last 10% of troubleshooting commonly takes far more than 10% of the budget or schedule.

PTC is Hard: the legal declarations from PTG managers who ran the CBOSS program (see 17CIV00786) reveal a long list of underlying factors that caused much acrimony and remain unchanged today: (1) the specifications and standards for PTC continue to evolve, triggering continued changes and penalty testing; (2) Caltrain and its in-house consultants (the so-called "owner's team") are woefully ill-equipped and uncoordinated in their approach to complex safety-critical avionics technology development; (3) the formal contractual interactions between the "owner's team" and the vendor are complicated and delay-prone; (4) working with UPRR is a huge pain in everyone's caboose; (5) the underlying systems over which CBOSS is supposed to "overlay" are kludged-together stove pipes that, incidentally, will require nearly total re-design for the electrification program; (6) testing PTC on an operating railroad requires extensive coordination that has been demonstrated to be lacking; and so on. Strike PTG and substitute ARINC.

These ingredients will produce a situation where CBOSS does less than was promised, later than planned, and for a lot more money. No crystal ball is needed to predict that CBOSS will continue to "fail forward" to a finish line somewhere beyond 2018.

29 comments:

  1. They really should've just thrown the whole thing out and started over with ACSES (or I-ETMS), since they have to completely overhaul their track circuits anyway for electrification. I just hope they don't end up with a mess like Denver with grade crossings not operating reliably.

    ReplyDelete
    Replies
    1. Any word on what the technical problems are that have caused their issues? All I’ve ever heard is that the gates don’t come at the exact right time, and stay down too long. They’ve been noticeably mute on the specifics, probably because it is a really simple and stupid glitch, but complicated by small-use proprietary software.

      Delete
    2. Denver's PTC system (I-ETMS) apparently out-smarts itself in the presence of real-world factors like station dwells, train handling, dispatcher intervention, etc., resulting in excessive down time, which sometimes triggers unsafe motorist behavior. Shout-out to Reality Check for providing links below:

      Here is an overview of the Denver grade crossing situation.

      Here is a more detailed briefing on the grade crossing issue to the RTD board (starts page 4).

      RTD's contractor says they're done, that's just how the technology works, kthxbye.

      CBOSS, of course, was selected because of the high number of grade crossings on the peninsula. One of its marquee selling points was supposed to be the ability to keep a grade crossing on the far side of a station platform (such as in Menlo Park or Burlingame) inactive if a train was stopping. The odds of that feature ever working are of course pretty remote given that ARINC is being tasked with the bare minimum.

      The grade crossing issue is currently the top risk (see Appendix F, page 141) for the Peninsula Corridor Electrification Program.

      Delete
  2. I see you mention Illinois and Indiana, does the Michigan line still use ITCS? I’ve thought for a while that it was in the railroads best interest to go with ITCS due to their willingness to deploy GE’s Trip Optimizer, I’m sure the interoperability between the two programs would be much smoother than between TO and I-ETMS, which is only now beginning to happen. It sounded from their press releases, that they laid fiber optic along the entire corridor, if that’s so, I’m sure it wouldn’t be a hinderance in the deployment of I-ETMS. This post makes a pretty good argument for throwing in the towel on ITCS once and for all. And given that HSR will use ERTMS (and potentially I-ETMS on the Antelope Valley line), only having 2 (and potentially 1) train control systems, versus the proposed 3, would certainly be a better solution in the long run. Good post, keep em coming!

    ReplyDelete
  3. Let's be clear, it isn't long gate downtime that "triggers" unsafe motorist behavior. The problem is dangerous idiots who should never have been permitted to drive.

    Step 1: install cameras
    Step 2: mandatory $1,000 fine and license suspension for violators

    In Germany, the penalty is 700 euros, 2 points, and 3-month license suspension (similar severity as a DUI).

    ReplyDelete
  4. @Drunk: it makes sense that the longer that drivers see gates are lowered without a train passing through, the likelihood of them driving around the gates increases. This is what the Sept 29, 2017 University of Colorado A Line and B Line Grade Crossing Update (PDF/packet page 19) states:

    Effects of Warning Time on Driver Behavior and Safety at Grade Crossings Study
    • Warning Times in Excess of 30-40 seconds caused many more drivers to engage in risky crossing behavior.
    • Most drivers expect a train to arrive within 20 seconds from the moment the activation begins.
    • Drivers begin to lose confidence in the traffic control system if the warning time exceeds approximately 60 seconds.
    • Suggested warning time guidelines (20 sec minimum, 25-35 sec desirable, 60 sec maximum)
    • Four-quadrant gates should be used if 10% of warning times exceed 60 sec.
    • Reference: Assessment of Warning Time Needs at Railroad Highway Grade Crossings with Active Traffic Control, Stephen H. Richards and K.W. Heathington, Transportation Research Record 1254

    ReplyDelete
  5. The "excess" gate downtime is all of 20 seconds. If that is enough to "trigger" motorists, then God help us.

    Note also that traffic engineers are perfectly fine with extending the red-time at traffic lights by much more than that, without any regard to how much it will increase jaywalking. It is quite the double standard.

    ReplyDelete
    Replies
    1. @Drunk: have a look at the Crossing Warning Time vs. Frequency of Value graph (pg. 16) for the 4-track (one pair for UP and another pair RTD) Dahlia crossing. It shows that for 1893 RTD crossing activations (presumably UP activations were ignored) in a 2-week period, that while the majority were in the 32-42 second range, well over one hundred were in the 50-70 second range. While 33 seconds was the most common (300 out of 1893) warning time, you've still got at least a dozen of 1 minute or more. Weird. And hardly constant time warning.

      As for what's enough to "trigger" a motorist ... consider that nearly 100% of humans who try, eventually manage to get a driver's license (or learner's permit) ... including (and especially) high school kids.

      Delete
    2. You can cut and dice the data all you want. According to the PUC, gates are down a mere 20 seconds longer than specified in Federal regs. Not even the FRA gives a damn (they granted a 5-year waiver).

      Drivers have no problem waiting several minutes (minutes!!) at a normal red light, but somehow turn into crazed lunatics if forced to wait a few extra seconds at a crossing gate.

      Delete
    3. I'm not taking sides, and I agree there's a big double standard in terms of how occasionally waiting a minute or less for a train to pass at lowered crossing gates is somehow perceived as way more unacceptable than the much longer and more frequent wait for almost any red light at a (multi-way) signalized road intersection. (Nevermind that the train is often carrying far more people than all the cars during a complete traffic light cycle at a signalized road intersection.)

      However, while motorists are fairly tolerant of waiting for passing traffic, they even get antsy at traffic lights when they're facing a long red with an empty intersection devoid of cross-traffic (e.g. late at night, etc.). This is a normal characteristic of RR crossings, there's a wait with nothing happening (ie. seemingly waiting for "nothing") until there's something: a train zips by and then its over.

      Delete
    4. We can argue every which way about how motorists should behave, but these systems have to be designed around how they do behave.

      Delete
    5. @drunk engineer :
      There are several things you are not considering in your assessment of driver behavior:

      1: Caltrain's row is an existing railroad with an existing...and quite good...constant warning time circuitry. There is decades of experience and expectation that the gates go up <10 seconds after the train passes. Gate downtimes of 20+ additional seconds would not be treated as 'normal' operations.

      2: your road-intersection example isn't analogous: when cars are at a red light, there is usually cross traffic while waiting, and if there isn't, at many intersections the lights change when there is no cross traffic.

      3: with more trains on the row in the near-ish future, more gate downtime is an inevitable consequence. Combining that with a degraded cwt system will exacerbate gridlock at the crossings. We need BETTER cwt, not worse.

      Delete
  6. For the amount of money that's been wasted on CBOSS thus far, we could be well on our way to grade-separating the handful of station-adjacent grade crossings where that particular feature would actually matter.

    ReplyDelete
    Replies
    1. @Joey, depends on how you define "well on our way."

      Here's a bit of a reality check on that claim:

      Broadway in Burlingame is one such crossing, and the city approved design for that single grade separation is being estimated at around $250m:


      [Burlingame Public Works Director] Syed Murtuza said he expects the early engineering approved Monday in the deal with the San Mateo County Transit Authority and Peninsula Joint Powers Board to be completed in 18 months. Funding for the work is fully secured and includes $3.5 million grant from the Transit Authority, plus a previously approved $500,000 contribution from Burlingame.

      Looking ahead though the path is much murkier, said Murtuza, who suggested ground may not break on constructing the separation project until 2025 — and that’s assuming all $250 million needed to finish the currently unfunded project is landed.

      “I could see this project going to construction by 2025, if we have all the money in the world. And that’s a big if,” said Murtuza.

      Delete
    2. It doesn't cost anything to close the useless Broadway station.

      Delete
    3. @Drunk, indeed. But the point is that the excess spend for CBOSS may not even pay for one grade sep at those rates.

      Separately, and as you may be aware, Caltrain and the respective cities keep talking about how the peppier EMUs that will accompany electrification will allow Broadway and Atherton to be reopened for weekday service.

      Delete
    4. San Bruno was $155m, 25th Ave is targeted at $180m, so yeah, I guess it's only about the cost of 1 grade separation or so.

      Still though, I wonder if it's worth spending money on such a feature given the inevitability of grade separations. On the other hand, some of the remaining stations with this issue will be rather complex to grade separate (e.g. San Mateo) so it could be a while before they're all done.

      Delete
  7. OT question: while ACSES is listed as an Alstom product, the top result for "ACSES II" is a Siemens website: https://w3.usa.siemens.com/mobility/us/en/Events/railway-interchange/Documents/SIE_BRO_ACSES%20II.pdf

    Did some transaction happened for ACSES-compatible PTC systems? Considering regulatory approval hasn't been granted for Alstom and Siemens merging, it's not likely Siemens started listing Alstom products under its name, or not?

    ReplyDelete
  8. Didn't someone predict all this 9 years ago?
    http://caltrain-hsr.blogspot.com/2009/10/peninsula-train-control-ptc-cboss-and.html

    ReplyDelete
  9. FYI: On Feb 6, 2018, the NTSB is scheduled to announce it's final "probable cause" determination for two train crashes (Hoboken, NJ in 2016 and Brooklyn, NY in 2017). In both cases, trains crashed at the end of the line, into the stop-bumpers (NJ: one dead, 100 injured) and train operators were apparently asleep (sleep apnea?). The NJ operator has no recollection of the accident, and the NJ black box shows that the train accelerated into the station until almost the last moment. In both cases, it is suspected that PTC would have prevented the accidents.

    ReplyDelete
  10. Any thoughts on the new Tri-Valley-San Joaquin Valley Regional Rail Authority?

    https://www.acetobart.org/

    https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=201720180AB758

    DMU ACE into a shared ACE/BART Livermore station? Lessons from Caltrain will apply.

    ReplyDelete
  11. CBOSS update: https://drive.google.com/file/d/1OlK8DvOwp9smGRZ0ILpG8mNbekPY7kP-/view

    ReplyDelete
  12. There is a brown wagon marked "FRA Office of Safety" parked immediately north of the junction between the UP and Caltrain tracks in Santa Clara, otherwise known as ground zero for ITCS/ETMS Interoperability (or lack thereof).

    This is truly fascinating...

    ReplyDelete
  13. Breaking News!!!!
    http://www.caltrain.com/Assets/__Agendas+and+Minutes/JPB/2018/2018-03-01+JPB+packet+final+to+include+revised+date.pdf (click on #8)

    ReplyDelete
  14. Caltrain is set to award a $45m no-bid contract to Wabtec to pick up the pieces from the failed and mired in lawsuits CBOSS PTC effort with Parsons Transportation Group. Aiming to meet this year's mandated federal PTC deadline, Wabtec will install their same proprietary I-ETMS PTC solution that ACE & UP are set to use ... which is better for interoperability.

    Here's the Caltrain press release: New Contractor Expected for Caltrain’s Positive Train Control Project

    The detailed staff report to the board is on PDF page 165 of the March 1, 2018 Caltrain JPB meeting agenda package.

    ReplyDelete
    Replies
    1. http://www.caltrain.com/Assets/__Agendas+and+Minutes/JPB/Board+of+Directors/Presentations/2011/10-6-11+Award+of+Contract+for+CBOSS+PTC+System.pdf (slide 7)

      Delete
    2. http://vtaorgcontent.s3-us-west-1.amazonaws.com/Site_Content/01_06.pdf (page 46)

      Delete
    3. Let's not forget the "Owner's Team" costs ballooning from $49,726,798 to $98,928,447 (http://www.caltrain.com/Assets/_Finance/Quarterly+Capital+Program+Status+Report/JPB/FY18+Q2+JPB+Quarterly+Report.pdf) Page 9.

      Delete