08 December 2018

Grade Crossing Trouble Ahead

Grade crossing in Denver (photo: RTD)
Denver's RTD has been operating a new 25 kV electrified commuter railroad since 2016. There's a big problem with it: the grade crossings gates are down for too long, which the FRA and Colorado PUC consider hazardous because impatient motorists frustrated by a longer-than-expected wait may drive around the gates just as the train finally shows up. The problem has festered, with  millions spent on human flaggers to supervise traffic at each grade crossing, contractual acrimony leading to lawsuits, and in recent days a threat by the FRA to shut down the entire railroad until the issue is resolved.

What does any of this have to do with Caltrain? The peninsula corridor electrification project uses the same electrification technology installed by the same contractor (Balfour Beatty), uses the same positive train control technology installed by the same contractor (Wabtec), must contend with more than three times as many grade crossings, and therefore, faces the same looming grade crossing problem. For months, the issue has topped the list of risks that threaten the project, and the search for a viable solution is causing the electrification contractor to fall significantly behind schedule.

How grade crossings are supposed to work

The simplest way to activate a grade crossing is for the train to shunt a track circuit at some set distance before the crossing. This is known as a conventional track circuit warning system, and doesn't work well if different trains arrive at different speeds. The point where the crossing activates must be set far enough ahead to give the required warning time before the fastest train arrives at the crossing; this makes the gates stay down too long for slower trains.

The usual solution to this problem is a Constant Warning Time (CWT) system, which uses electrical signals sent through the track to sense the distance and speed of the approaching train. The grade crossing controller can then predict when to activate the crossing such that the warning time is approximately constant regardless of train speed. This is the type of warning system installed today on the many grade crossings of the peninsula rail corridor.

The FRA provides a nice overview discussion of how various types of grade crossings work. The applicable federal regulations are under 49 CFR Part 234.

What happened in Denver

Because the Denver system is electrified, there are large 60 Hz AC traction return currents (at safe low voltage!) commonly present in the rails when a train is nearby. These currents interfere with and prevent the use of a traditional Constant Warning Time system.

The contractor came up with a "smart" solution: the crossings have a traditional track circuit warning system overlaid with a wireless crossing activation system (WCAS) that interfaces with the positive train control system. Software sends wireless messages back and forth between the train computer and the crossing controller. The train and crossing enter into a contract: the train predicts when it will arrive at the crossing and promises not to get there any sooner, and the crossing commits to activate at some fixed time interval before the appointed arrival, staying closed until the train passes. Depending on the circumstance, the train may arrive at the crossing later than anticipated when the contract was entered into, resulting in extended gate down time. When WCAS is inoperative, the old-school track circuit takes over, also resulting in extended gate down time when a train is operating at less than maximum speed.

In early 2016, before the Denver train opened for revenue service, FRA and PUC inspectors found that the crossings activation times were inconsistent, with frequent occurrence of long gate down times and erosion of what is known as "credibility" of the warning system. Things went gradually downhill from there:
  • So as not to delay the much anticipated start of revenue service, the regulatory agencies granted a temporary waiver to allow RTD to begin operating without WCAS, on the condition that human flaggers supervise traffic at each affected crossing, at the expense of the contractor.
  • The contractor tried to tweak the WCAS software to make warning times more consistent. A fudge factor known as the "Approach Condition Adjustment Factor" (ACAF, so known because every fudge factor needs an acronym to sound legitimate) was applied based on the observed statistical distribution of warning times at each crossing.
  • In September 2017, the FRA gave RTD relief in its interpretation of the consistency required for gate downtime, relaxing its unofficial consistency criterion from +/-5 seconds or +/-10% of programmed warning time to +15/-5 seconds for RTD's system.
  • Performance of WCAS failed to satisfy the increasingly picky regulatory agencies. RTD began to penalize the contractor for failing to deliver a working grade crossing solution. FRA inspectors kept writing up excessive downtime violations.
  • The FRA forbade the start of revenue service on a newer rail line that has since been completed. The original plan to create quiet zones, where train horns are not used at grade crossings, was delayed indefinitely to the continuing aggravation of neighboring residents.
  • In September 2018, the contractor decided that the regulatory agencies had invented and enforced new consistency requirements that were not in the official regulations, and sued RTD claiming "force majeure" of a regulatory change. The complaint makes a fascinating read.
  • In October 2018, the FRA provided the latest inspection report (of many) showing continuing non-compliance with the -5/+15 second consistency tolerance.
  • On November 15th, 2018, the FRA fired off a letter indicating that it was fed up with the continuing grade crossing non-compliance, among other things, and threatened to shut down the entire commuter rail system by revoking the 2016 waiver.
  • RTD is lawyering up against the FRA, and submitted a strongly worded legal memorandum with numerous exhibits effectively claiming that the grade crossing problem exists solely in the imagination of the regulators. RTD provided evidence that other railroads (including Caltrain!) commonly experienced long gate down times in violation of the criteria imposed on RTD.
Whatever happens next is sure to be dramatic. The entire saga can be reviewed under docket FRA-2016-0028, which organizes all the documents exchanged between RTD and the FRA relating to the temporary operating waiver.

Some Observations
Measured distribution of 38255 grade
crossing activation times in Denver.
  1. Denver solved the wrong problem. They tried to invent a better mousetrap, something more sophisticated than a constant warning time grade crossing predictor. All they needed to do was to provide the same simple function with a substitute detection method that didn't rely on traditional audio-frequency AC circuits, which are incompatible with electrification. Instead, they decided to invent a better mousetrap involving lots of software, GPS, and wireless messaging, which naturally attracted regulatory scrutiny.
     
  2. Complexity is bad. Multiplying the number of interfaces and creating dependencies between elements of the system leads to expensive aerospace avionics-like hardware and software that is cumbersome to deploy, test and maintain. System complexity leads to a proliferation of strange and unanticipated corner cases and failure modes.
     
  3. Software can anticipate when to activate a crossing and prevent a train from showing up too soon, but there is no software in the world that can make a train show up on time.
     
  4.  Grade crossing activation times naturally follow a statistical distribution that arises from random environmental factors beyond the control of the warning system. The low end of the distribution must never be shorter than the mandated 20 seconds, but the long end of the distribution will inevitably have some outliers. The diagram above shows the measured distribution of 38255 crossing activation times on RTD. Notice the long tail.
     
  5. Even traditional "constant" warning time systems have this statistical tail. If the FRA inspectors applied the same regulatory zeal to Caltrain as they did to RTD, Caltrain would certainly be found in non-compliance. This isn't idle speculation: RTD gathered the data to prove it.
     
  6. The criteria for non-compliance, namely a "significant difference" from the prescribed warning time, are subjective. Guidance from the FRA acknowledges as much: "Thus, prudent judgment must be exercised when reviewing the results of warning time testing to determine whether the actual warning time provided during testing was compliant with the standard."
     
  7. The regulators painted themselves into a corner. They imposed a strict -5/+15 second criterion, which is easy to verify for an inspector with a stop watch and a clip board, but makes the long tail of the activation time distribution an automatic violation that is almost impossible to avoid. In recognition of the environmental factors beyond the control of the warning system, the regulators should have used controlled test conditions or applied a different criterion, such as X% of activations within Y% of programmed warning time. This is harder to verify for an inspector with a clipboard, but the grade crossing controller ought to be able to maintain these statistical records across a very large number of crossing activations.
     
  8. While electrification is relatively rare in the US, there are numerous railroads abroad that have solved the constant warning time problem in electrified territory. This probably isn't rocket science. The mousetrap already exists.
Lessons for Caltrain
With the grade crossing warning system already at the top of the Caltrain electrification project's risk list and the contractor falling behind, this problem is already getting a lot of attention. The people involved hopefully already realize:

Keep it simple - the job is to come up with a grade crossing predictor that works in the presence of traction return currents. It will be tempting to come up with a more sophisticated custom solution that uses lots of software, but we learned from the CBOSS project, and Denver's travails, that complexity usually leads straight to disaster. The dumber the better.
Document existing conditions - a large database of activation time statistics should be assembled for each crossing as it exists today, to head off a conflict over the subjective nature of the FRA warning time consistency criteria. In the event of a Denver-like disagreement with FRA or CPUC, Caltrain would be in a position to quantify precisely how much more (and hopefully not less) consistent the new warning solution will be, regardless of the selected criterion. Caltrain enjoys the advantage that it isn't building new crossings like Denver, so there is an existing system performance baseline that is already accepted by regulators. That baseline will only be useful if it is thoroughly documented.
Plant the goal posts firmly - Work with FRA towards mutually agreed verification criteria that don't repeat the mistakes made in Denver of specifying a rigid range and then testing in the uncontrolled conditions of revenue service. The activation time distribution will always have a statistical tail. If the consistency criterion can't be met by today's existing grade crossing system, then it's probably a bad criterion.
Make sure we aren't paying for Denver - the contractor needs to be held accountable for the extent to which Caltrain electrification funds (and schedule delays!) are accruing to the Denver project's benefit, if the same grade crossing solution is ultimately pursued in both projects.

32 comments:

  1. Over here in old Blighty the gates usually come down very early and stay down for a very long time. It's normal, and expected. Different regulations, much more cautious I suppose.

    ReplyDelete
    Replies
    1. We cannot practically or politically allow routine crossing gate activations to last 2 or more minutes, as I have timed in rural areas of "old Blighty". At most any busy Caltrain crossing, such long gate closures would result in horrible traffic backups. And with increasing train frequencies, it will be more important than ever to keep the gate activations in as tight a time envelope around the train's passage as possible.

      As you can see here (at Caltrain's first and only train horn "quiet zone" ... with quad gates and obstacle-detection in sleepy Atherton), road traffic is only blocked for about 40 seconds during the entire gate activation cycle. (And which is substantially shorter than the red-light cycle time of almost any non-trivial road intersection in Caltrain's service area ... and yet people — even train advocates — jammer about how $250m-each grade separations simply must occur before we can run too many more trains. What a perfectly insane double-standard! Waiting for hundreds aboard a train to pass in under a minute = unacceptable! ... Waiting longer and far, far more frequently for dozens to pass in single-occupant cars = normal! )

      Delete
  2. It seems that all that would be needed is a simple and reliable way to measure approaching train speed and position (distance from crossing). With those two bits of information, it should be simple to calculate gate activation time to ensure a guaranteed minimum warning time. Maximum warning time be made to equal the minimum time for constant speed trains. Of course, you'd need to have an extra cushion to cover the maximum possible acceleration that could occur after gate activation. The maximum warning time would occur when a train goes into maximum braking after gate activation ... possibly resulting in a time-out release (e.g. if the train stops after activation). Can this problem really be that hard for an electrical/electronic and/or signaling engineers to solve simply and elegantly!?

    ReplyDelete
    Replies
    1. Euro-style axle counters or wheel sensors come to mind as a nice way of handling closely-spaced crossings with independent approach zones that overlap with each other. At 110 mph design speed, you need to know when and how fast a train is coming about a mile from a crossing.

      I would be curious to find out if we are going to get the same system as Denver or a different system-- that fact alone would already tell us a lot.

      Delete
    2. According to https://www.railengineer.uk/2018/03/22/reducing-the-risk-from-automatic-level-crossings/, implementing constant-down-time crossings, with high variance in train speeds, in an ETCS (level 2) environment, is still an open problem.

      I don't see how axle-counters help, unless you're proposing to measure time between axles, and do a ground-side computation of train speed. (As opposed to train-speed). Is that the idea?

      Delete
    3. Multiple wheel sensors placed in the approach to a crossing would tell you how fast the train is coming (no counting required) with a ground-side calculation that does not rely on a PTC software or hardware interface. It wouldn't care about return currents and associated EMI/EMC issues either. But what do I know, maybe the Denver system works just fine.

      Delete
    4. In many places in Europe, grade crossings are signal-protected, which means that the gates must be down earlier than operational braking time. And it would need a continuous supervision (as implemented in Euroloop or LZW) to lift the braking curve when the gates are down, and the train is already braking.

      OTOH, if the braking time were more or less similar for all train types, things could be simplified.

      Delete
    5. @Clem: so you're saying two (or more) axle-counter installations, a known distance apart, and ground-side velocity calculation based on the time difference between the two signals (counts)? I didn't infer that from your plural
      .
      And no, I never said the Denver installation is working fine. No need to put words in people's mouths.

      @Max: You mean there are still crossings that are _not_ interlocked !?

      I actually wrote a counterfactual, "what if", if Europloops / balises could handle two-way communication. The train could send its speed to the balise. Install a set of balises in the approach to the level-crossing. The balises are speed-bucked on when they trigger the gate-down. Further away balises would trigger gate-down at higher speeds than the closer balises.

      But, as far as I know, extant balises do not support train-to-ground communication. Wikipedia says "Downlink" was removed from the spec. Personally, I don't see where a passively-powered (from train RF) would send information, _to_, in any case.

      Does anyone know the history of ETCS train-to-balise/Euroloop downlink?

      Delete
    6. To be clear, I said that it's possible that the Denver system works just fine from a technical perspective. To me, the disagreement appears to stem from the FRA regional office painting itself into a corner in its manner of measuring and enforcing gate timing. It's going to be tough for them to admit that and back down.

      Delete
    7. Secondary crossings are not interlocked. Secondary means either secondary line or secondary road.

      I am not aware of any downlink capabilities, neither with Euroloop nor with LZB. With ETCS, the downlink works via GSM-R. Actually, when I wrote my comment, I was not thinking of delaying lowering the gates, but releasing the braking when the gates are down.

      Downlink to balises would be pretty tricky, considering the short time available for the communication. The datagrams from balises are short. In ETCS-L2, the permissions are not sent via balises, but via GSM-R; the balises are mainly used to reset the odometer in the on-board unit, and/or to provide fixed info.

      The use of a loop (or LZB) is not part of ETCS-L2, but it is used in "legacy" systems (and L1-LS), in order to provide continuous information (such as for enforcing the brake curve, or to relieve the restriction).

      Delete
  3. The absolutely shocking thing about the article is that the Denver A-line is doing 18k boardings with 8 stations and under 25 miles of track, two years after opening. And that's with only two big destinations - downtown and the airport, every other station is still in a giant parking lot.There's a lot they did right in Denver. They are now running trains every 15 peak and every 30 off peak, from 330am to 1am.

    ReplyDelete
  4. New developments in Denver: RTD submits plan to ease federal concerns over A-Line safety gate issues

    https://www.denverpost.com/2018/12/14/rtd-submits-plan-to-federall-regulators-a-line-safety/

    ReplyDelete
  5. "Invensys Rail has integrated its Coded Audio Track Circuit (PSO 4000) and its GCP 4000 Crossing Controller technologies to provide an Occupancy Detector System that can be configured to provide constant warning time in electrified territory."
    https://www.railwayage.com/cs/raising-the-bar-on-crossing-protection/

    ReplyDelete
  6. Uh, that's from 2012. Doss Siemens still sell that?

    Next sentence in the cited page: "This means that a coded, audio, overlay track circuit is available for electrified territory in an integrated crossing package—the “Crossing-In-A-Box” for electrified territory.

    Still an overlay, still audio frequency. Not going to help with handing a range for speeds from 110mph EMUs, to gravel trains with flat wheels hauled by old diesels.

    ReplyDelete
    Replies
    1. Siemens acquired Invensys in 2013: https://www.railwaygazette.com/news/business/single-view/view/siemens-completes-invensys-rail-acquisition.html. The VTA purchased a bunch of PSO 4000s last year but they don't need CWT for the light rail.

      Delete
  7. Another interesting twist to Denver's problems: Denver’s tall buildings partly to blame for A-Line, G-Line problems, RTD tells feds

    https://www.denverpost.com/2018/12/17/rtd-denver-commuter-rail-g-line-plan/

    ReplyDelete
  8. Here is the RTD Crossing Warning Time Action Plan submitted to FRA on 14 December 2018.

    ReplyDelete
  9. Flaggers to once again be relieved of duty on RTD’s A-Line, B-Line crossings

    The FRA decision to allow the gates to operate without flaggers was confirmed by Jeremy Story, a Denver Transit Partners spokesman. Flaggers monitoring the 16 at-grade crossings on the G-Line, whose opening is over 2 years delayed, will remain in place for testing.

    It’s unclear whether the FRA’s decision is related to RTD’s submission last weekend of a plan to address the longstanding crossing gate timing issues. They stay down longer than mandated and regulators are concerned that could prompt impatient motorists to drive around them and into the path of an oncoming train.

    Robert Lauby, chief safety officer with the FRA, wrote to RTD on Friday saying that the plan it submitted “provides a viable path forward to correct deficiencies on the RTD grade crossing activation system within one year.” [Holy crap, another year!?]

    Other than a short time this summer when crossings operated without flaggers, they've stood guard with portable stop signs along the A-Line since the 23-mile service opened in April 2016, costing Denver Transit Partners tens of millions of dollars in staffing expenses.

    ReplyDelete
  10. Replies
    1. Oops, it’s its, of course ... dammit!

      Delete
    2. I sure appreciate good grammar.

      Delete
    3. Watch as Atherton City Council decides not to close its Caltrain station just yet ... primarily so as not to cede any "leverage" it may have over Caltrain:

      Delete
    4. What does Atherton want leverage for?

      As spelled out in the staff report, they want Caltrain to agree to stuff like:

      • take all necessary steps to make the corridor as safe as possible including maintenance of the quad gates at Fair Oaks Lane and addition of quad gates at Watkins Avenue
      • continue to enforce Fair Oaks Lane Quiet Zone compliance
      • ensure any long term corridor rail plan is limited to two tracks through Atherton

      Delete
  11. This will NEVER happen to Caltrain: https://www.progressiverailroading.com/passenger_rail/news/LIRR-trains-collide-after-car-drives-around-lowered-gates--56875

    ReplyDelete
    Replies
    1. I dunno. Is a loaded Caltrain heavier and has better "plow" design that would shred the care more before derailing? I don't have an any stats, but I recall hearing of more derailments after vehicle crashes on Metro North and LIRR than on Caltrain/ACE/Metrolink/Amtrak in California. But that just could be due to less train service.

      Delete
    2. It's really easy to enter your name -- that would be "Roland Lebrun" -- to the "anonymous" (come on!) comments featuring random context-free URLs, "breaking news", "NEVER happen to Caltrain", dodgy dropbox and bit.ly and so.on links to WTF, gubbish about British whatnot and especially CTRL, or instructions to watch an hour minute of some random youtube link for unspecified purposes.

      It's not that hard!

      Even I can do it, with no "google account" or anything.

      Delete
    3. "About 200 feet of third rail ... through the area was damaged"

      Yes, we can be reasonably confident that Caltrain will never suffer such 3rd rail damages!

      Delete
    4. When I first saw the pictures I thought wow that train plowed right through the platform slab without deformation. Later I realized that it was the opposite, the platform slab plowed right through the train. Truly amazing that nobody on board was hurt.

      Delete
    5. @Clem, there was only one rider in the cab car. The engineer evacuated the cab and moved all the way to the rear of the cab car, moving the sole rider out of harm's way in the process. These aerial photos show that the front right corner of the cab car stripped the yellow ADA edge panels off a significant length of concrete platform slabs like corn on the cob!

      Delete
  12. Not to be completely off topic, but why are Caltrain trains derailing practically daily in Diridon? Saw this comment during last nights fiasco.

    https://twitter.com/weauxheaux/status/1100575459601920000

    Nothing official from Caltrain, except something that sounds like track repair from the derailment continues into the morning:
    https://twitter.com/Caltrain/status/1100779528324956160

    ReplyDelete
    Replies
    1. Picked up by news stories, but not much detail:
      https://www.mercurynews.com/2019/03/02/caltrain-three-trains-derailed-in-four-days-along-same-stretch-of-tracks/

      Delete
    2. Hit the media:
      https://www.mercurynews.com/2019/03/02/caltrain-three-trains-derailed-in-four-days-along-same-stretch-of-tracks

      Delete