Grade Crossing Trouble Ahead

Grade crossing in Denver (photo: RTD)

Denver's RTD has been operating a new 25 kV electrified commuter railroad since 2016. There's a big problem with it: the grade crossings gates are down for too long, which the FRA and Colorado PUC consider hazardous because impatient motorists frustrated by a longer-than-expected wait may drive around the gates just as the train finally shows up. The problem has festered, with  millions spent on human flaggers to supervise traffic at each grade crossing, contractual acrimony leading to lawsuits, and in recent days a threat by the FRA to shut down the entire railroad until the issue is resolved.

What does any of this have to do with Caltrain? The peninsula corridor electrification project uses the same electrification technology installed by the same contractor (Balfour Beatty), uses the same positive train control technology installed by the same contractor (Wabtec), must contend with more than three times as many grade crossings, and therefore, faces the same looming grade crossing problem. For months, the issue has topped the list of risks that threaten the project, and the search for a viable solution is causing the electrification contractor to fall significantly behind schedule.

How grade crossings are supposed to work

The simplest way to activate a grade crossing is for the train to shunt a track circuit at some set distance before the crossing. This is known as a conventional track circuit warning system, and doesn't work well if different trains arrive at different speeds. The point where the crossing activates must be set far enough ahead to give the required warning time before the fastest train arrives at the crossing; this makes the gates stay down too long for slower trains.

The usual solution to this problem is a Constant Warning Time (CWT) system, which uses electrical signals sent through the track to sense the distance and speed of the approaching train. The grade crossing controller can then predict when to activate the crossing such that the warning time is approximately constant regardless of train speed. This is the type of warning system installed today on the many grade crossings of the peninsula rail corridor.

The FRA provides a nice overview discussion of how various types of grade crossings work. The applicable federal regulations are under 49 CFR Part 234.

What happened in Denver

Because the Denver system is electrified, there are large 60 Hz AC traction return currents (at safe low voltage!) commonly present in the rails when a train is nearby. These currents interfere with and prevent the use of a traditional Constant Warning Time system.

The contractor came up with a "smart" solution: the crossings have a traditional track circuit warning system overlaid with a wireless crossing activation system (WCAS) that interfaces with the positive train control system. Software sends wireless messages back and forth between the train computer and the crossing controller. The train and crossing enter into a contract: the train predicts when it will arrive at the crossing and promises not to get there any sooner, and the crossing commits to activate at some fixed time interval before the appointed arrival, staying closed until the train passes. Depending on the circumstance, the train may arrive at the crossing later than anticipated when the contract was entered into, resulting in extended gate down time. When WCAS is inoperative, the old-school track circuit takes over, also resulting in extended gate down time when a train is operating at less than maximum speed.

In early 2016, before the Denver train opened for revenue service, FRA and PUC inspectors found that the crossings activation times were inconsistent, with frequent occurrence of long gate down times and erosion of what is known as "credibility" of the warning system. Things went gradually downhill from there:

• So as not to delay the much anticipated start of revenue service, the regulatory agencies granted a temporary waiver to allow RTD to begin operating without WCAS, on the condition that human flaggers supervise traffic at each affected crossing, at the expense of the contractor.
• The contractor tried to tweak the WCAS software to make warning times more consistent. A fudge factor known as the "Approach Condition Adjustment Factor" (ACAF, so known because every fudge factor needs an acronym to sound legitimate) was applied based on the observed statistical distribution of warning times at each crossing.
• In September 2017, the FRA gave RTD relief in its interpretation of the consistency required for gate downtime, relaxing its unofficial consistency criterion from +/-5 seconds or +/-10% of programmed warning time to +15/-5 seconds for RTD's system.
• Performance of WCAS failed to satisfy the increasingly picky regulatory agencies. RTD began to penalize the contractor for failing to deliver a working grade crossing solution. FRA inspectors kept writing up excessive downtime violations.
• The FRA forbade the start of revenue service on a newer rail line that has since been completed. The original plan to create quiet zones, where train horns are not used at grade crossings, was delayed indefinitely to the continuing aggravation of neighboring residents.
• In September 2018, the contractor decided that the regulatory agencies had invented and enforced new consistency requirements that were not in the official regulations, and sued RTD claiming "force majeure" of a regulatory change. The complaint makes a fascinating read.
• In October 2018, the FRA provided the latest inspection report (of many) showing continuing non-compliance with the -5/+15 second consistency tolerance.
• On November 15th, 2018, the FRA fired off a letter indicating that it was fed up with the continuing grade crossing non-compliance, among other things, and threatened to shut down the entire commuter rail system by revoking the 2016 waiver.
• RTD is lawyering up against the FRA, and submitted a strongly worded legal memorandum with numerous exhibits effectively claiming that the grade crossing problem exists solely in the imagination of the regulators. RTD provided evidence that other railroads (including Caltrain!) commonly experienced long gate down times in violation of the criteria imposed on RTD.

Whatever happens next is sure to be dramatic. The entire saga can be reviewed under docket FRA-2016-0028, which organizes all the documents exchanged between RTD and the FRA relating to the temporary operating waiver.

Some Observations

Measured distribution of 38255 grade
crossing activation times in Denver.

1. Denver solved the wrong problem. They tried to invent a better mousetrap, something more sophisticated than a constant warning time grade crossing predictor. All they needed to do was to provide the same simple function with a substitute detection method that didn't rely on traditional audio-frequency AC circuits, which are incompatible with electrification. Instead, they decided to invent a better mousetrap involving lots of software, GPS, and wireless messaging, which naturally attracted regulatory scrutiny.
    
2. Complexity is bad. Multiplying the number of interfaces and creating dependencies between elements of the system leads to expensive aerospace avionics-like hardware and software that is cumbersome to deploy, test and maintain. System complexity leads to a proliferation of strange and unanticipated corner cases and failure modes.
    
3. Software can anticipate when to activate a crossing and prevent a train from showing up too soon, but there is no software in the world that can make a train show up on time.
    
4.  Grade crossing activation times naturally follow a statistical distribution that arises from random environmental factors beyond the control of the warning system. The low end of the distribution must never be shorter than the mandated 20 seconds, but the long end of the distribution will inevitably have some outliers. The diagram above shows the measured distribution of 38255 crossing activation times on RTD. Notice the long tail.
    
5. Even traditional "constant" warning time systems have this statistical tail. If the FRA inspectors applied the same regulatory zeal to Caltrain as they did to RTD, Caltrain would certainly be found in non-compliance. This isn't idle speculation: RTD gathered the data to prove it.
    
6. The criteria for non-compliance, namely a "significant difference" from the prescribed warning time, are subjective. Guidance from the FRA acknowledges as much: "Thus, prudent judgment must be exercised when reviewing the results of warning time testing to determine whether the actual warning time provided during testing was compliant with the standard."
    
7. The regulators painted themselves into a corner. They imposed a strict -5/+15 second criterion, which is easy to verify for an inspector with a stop watch and a clip board, but makes the long tail of the activation time distribution an automatic violation that is almost impossible to avoid. In recognition of the environmental factors beyond the control of the warning system, the regulators should have used controlled test conditions or applied a different criterion, such as X% of activations within Y% of programmed warning time. This is harder to verify for an inspector with a clipboard, but the grade crossing controller ought to be able to maintain these statistical records across a very large number of crossing activations.
    
8. While electrification is relatively rare in the US, there are numerous railroads abroad that have solved the constant warning time problem in electrified territory. This probably isn't rocket science. The mousetrap already exists.

Lessons for Caltrain

With the grade crossing warning system already at the top of the Caltrain electrification project's risk list and the contractor falling behind, this problem is already getting a lot of attention. The people involved hopefully already realize:

Keep it simple - the job is to come up with a grade crossing predictor that works in the presence of traction return currents. It will be tempting to come up with a more sophisticated custom solution that uses lots of software, but we learned from the CBOSS project, and Denver's travails, that complexity usually leads straight to disaster. The dumber the better.

Document existing conditions - a large database of activation time statistics should be assembled for each crossing as it exists today, to head off a conflict over the subjective nature of the FRA warning time consistency criteria. In the event of a Denver-like disagreement with FRA or CPUC, Caltrain would be in a position to quantify precisely how much more (and hopefully not less) consistent the new warning solution will be, regardless of the selected criterion. Caltrain enjoys the advantage that it isn't building new crossings like Denver, so there is an existing system performance baseline that is already accepted by regulators. That baseline will only be useful if it is thoroughly documented.

Plant the goal posts firmly - Work with FRA towards mutually agreed verification criteria that don't repeat the mistakes made in Denver of specifying a rigid range and then testing in the uncontrolled conditions of revenue service. The activation time distribution will always have a statistical tail. If the consistency criterion can't be met by today's existing grade crossing system, then it's probably a bad criterion.

Make sure we aren't paying for Denver - the contractor needs to be held accountable for the extent to which Caltrain electrification funds (and schedule delays!) are accruing to the Denver project's benefit, if the same grade crossing solution is ultimately pursued in both projects.