Billions of Seconds Wasted

The latest tweaks to the design of the San Francisco Downtown Extension (DTX) rail alignment can be seen in a March 2018 track plan and profile drawing. Because it largely follows the street grid, it's no secret that the alignment is full of sharp curves, which can only be traversed at slow speed. However, compared to a 2012 drawing, speed limits have dropped in several places from 40 mph to just 30 mph, because train speed evidently isn't a design priority when civil engineers get a blank check.

Back in 2012, the speed profile sort of made sense: starting from the basement of the Transbay Transit Center (left end of the diagram) the train would screech at about 20 mph through the sharp curve towards 2nd Street, speeding up to 35 mph along 2nd and through the curve towards Townsend. On that mostly straight bit along Townsend, speeds could pick up to 40 mph before dropping back briefly to 35 mph through the curve to 7th Street, then exiting along 7th Street at 40 mph (right end of diagram). If only one criticism were allowed, it wasn't clear why that final curve should be limited to 35 mph; there was plenty of space at Townsend and 7th to flatten it out to 40 mph, resulting in a simple and efficient stepped speed profile for the approach to Transbay.

Fast forward to 2018, and things are much worse. There is a new kink in the alignment where it connects to the existing tracks. The new underground 4th and Townsend station, at the city's request, has been shoved into the Townsend Street right of way in the hope of freeing up the existing rail terminal parcels for high rise redevelopment (where the 2012 alignment might have clashed with new building foundations). The rigid requirement for a straight island platform has resulted in a series of 30 mph kinks in the track. Elsewhere, the speed limit along Townsend has dropped by 5 mph.

The designers might argue this is only a few seconds lost, so no big deal, right?

How many seconds are wasted?

A train traversing the DTX will have to observe the speed limits not just for the length of each speed restriction, but for the added length of the train itself, as the limit applies from the moment the head end of the train enters a speed restriction until the tail end leaves the speed restriction. High-speed trains will be up to 400 m long, so this can really add up. We can simulate the time needed for a train to travel from a standing start at the end of a Transbay platform to a 40 mph entry into the existing Tunnel 1, a distance of about 2.2 miles. The results depend on the train type, and whether a stop is made at 4th and Townsend:

• 2012 alignment, single-length HSR: 4:04
• 2012 alignment, double-length HSR 4:17
• 2012 alignment, 8-car Caltrain EMU, no stop at Townsend 4:06
• 2012 alignment, 8-car Caltrain EMU, 30-second stop at Townsend: 5:04
   
• 2018 alignment, single-length HSR: 4:25 (+21 sec)
• 2018 alignment, double-length HSR 4:42 (+25 sec)
• 2018 alignment, 8-car Caltrain EMU, no stop at Townsend 4:27 (+21 sec)
• 2018 alignment, 8-car Caltrain EMU, 30-second stop at Townsend 5:19 (+15 sec)

To summarize and simplify, we can assume that every Caltrain will stop at Townsend, so the performance loss is 15 seconds per Caltrain movement, and roughly 20 seconds per HSR movement. That doesn't sound like much, but consider that trains are carrying hundreds of passengers, each of whom are individually delayed. The collective waste of time can be measured by multiplying the train delay by the expected ridership.

Today Caltrain has about 15,000 weekday boardings in SF, a number that Caltrain says could eventually quadruple. Let's say it only triples, and that 35,000 of those weekday boardings occur at Transbay and 10,000 at 4th and Townsend (which we won't count) making for 70,000 trips through the DTX approach. That's 70,000 trips x 15 seconds/trip = a million seconds wasted every weekday, or about 3 person-years of productive labor time per month of DTX operation. Over a year, about a quarter billion seconds would be wasted!

HSR eventually expects 18 million annual trips originating in the Bay Area, of which maybe half might involve Transbay. Combine that with a similar number of HSR trips terminating at SF, and you get 18 million annual HSR trips through the DTX approach. That would be a waste of another third of a billion seconds.

Every year then, about half a billion seconds would be wasted due to careless DTX alignment design.

How do we fix it?

Fixing it involves realizing that

1. every second matters, a lot
2. the marginal cost of the next second saved is more expensive than the last
3. saving seconds is most efficiently and cheaply done in the slow parts of a system

Making up 20 seconds through minor fixes to the DTX track alignment design, before any concrete is poured, is far cheaper and easier and more productive than trying to make up 20 seconds somewhere faster, for example in the Central Valley by running trains at 220 mph instead of 215 mph.

What ought to still be possible is an alignment that starts at 20 mph through the screecher to 2nd Street, rises to 35 mph along 2nd Street, then rises to 40 mph along Townsend continuing without slowing around the curve to 7th Street. With this improved speed profile, train run times from Transbay to Tunnel 1 (relative to the 2018 alignment plans) would be:

• Single-length HSR: 4:02 (23 seconds faster)
• Double-length HSR 4:14 (28 seconds faster)
• 2018 alignment, 8-car Caltrain EMU, no stop at Townsend 4:04 (23 seconds faster)
• 2018 alignment, 8-car Caltrain EMU, 30-second stop at Townsend 5:02 (17 seconds faster)

The combined annual time savings would exceed half a billion seconds per year. As we watch the cost of the DTX project reach ever more dizzying heights, we should at the very least expect to get more transportation value out of the project. Careless and inexcusable engineering of a rail alignment that wastes so much of everyone's time only adds insult to the injury.

Grade Crossing Trouble Ahead

Grade crossing in Denver (photo: RTD)

Denver's RTD has been operating a new 25 kV electrified commuter railroad since 2016. There's a big problem with it: the grade crossings gates are down for too long, which the FRA and Colorado PUC consider hazardous because impatient motorists frustrated by a longer-than-expected wait may drive around the gates just as the train finally shows up. The problem has festered, with  millions spent on human flaggers to supervise traffic at each grade crossing, contractual acrimony leading to lawsuits, and in recent days a threat by the FRA to shut down the entire railroad until the issue is resolved.

What does any of this have to do with Caltrain? The peninsula corridor electrification project uses the same electrification technology installed by the same contractor (Balfour Beatty), uses the same positive train control technology installed by the same contractor (Wabtec), must contend with more than three times as many grade crossings, and therefore, faces the same looming grade crossing problem. For months, the issue has topped the list of risks that threaten the project, and the search for a viable solution is causing the electrification contractor to fall significantly behind schedule.

How grade crossings are supposed to work

The simplest way to activate a grade crossing is for the train to shunt a track circuit at some set distance before the crossing. This is known as a conventional track circuit warning system, and doesn't work well if different trains arrive at different speeds. The point where the crossing activates must be set far enough ahead to give the required warning time before the fastest train arrives at the crossing; this makes the gates stay down too long for slower trains.

The usual solution to this problem is a Constant Warning Time (CWT) system, which uses electrical signals sent through the track to sense the distance and speed of the approaching train. The grade crossing controller can then predict when to activate the crossing such that the warning time is approximately constant regardless of train speed. This is the type of warning system installed today on the many grade crossings of the peninsula rail corridor.

The FRA provides a nice overview discussion of how various types of grade crossings work. The applicable federal regulations are under 49 CFR Part 234.

What happened in Denver

Because the Denver system is electrified, there are large 60 Hz AC traction return currents (at safe low voltage!) commonly present in the rails when a train is nearby. These currents interfere with and prevent the use of a traditional Constant Warning Time system.

The contractor came up with a "smart" solution: the crossings have a traditional track circuit warning system overlaid with a wireless crossing activation system (WCAS) that interfaces with the positive train control system. Software sends wireless messages back and forth between the train computer and the crossing controller. The train and crossing enter into a contract: the train predicts when it will arrive at the crossing and promises not to get there any sooner, and the crossing commits to activate at some fixed time interval before the appointed arrival, staying closed until the train passes. Depending on the circumstance, the train may arrive at the crossing later than anticipated when the contract was entered into, resulting in extended gate down time. When WCAS is inoperative, the old-school track circuit takes over, also resulting in extended gate down time when a train is operating at less than maximum speed.

In early 2016, before the Denver train opened for revenue service, FRA and PUC inspectors found that the crossings activation times were inconsistent, with frequent occurrence of long gate down times and erosion of what is known as "credibility" of the warning system. Things went gradually downhill from there:

• So as not to delay the much anticipated start of revenue service, the regulatory agencies granted a temporary waiver to allow RTD to begin operating without WCAS, on the condition that human flaggers supervise traffic at each affected crossing, at the expense of the contractor.
• The contractor tried to tweak the WCAS software to make warning times more consistent. A fudge factor known as the "Approach Condition Adjustment Factor" (ACAF, so known because every fudge factor needs an acronym to sound legitimate) was applied based on the observed statistical distribution of warning times at each crossing.
• In September 2017, the FRA gave RTD relief in its interpretation of the consistency required for gate downtime, relaxing its unofficial consistency criterion from +/-5 seconds or +/-10% of programmed warning time to +15/-5 seconds for RTD's system.
• Performance of WCAS failed to satisfy the increasingly picky regulatory agencies. RTD began to penalize the contractor for failing to deliver a working grade crossing solution. FRA inspectors kept writing up excessive downtime violations.
• The FRA forbade the start of revenue service on a newer rail line that has since been completed. The original plan to create quiet zones, where train horns are not used at grade crossings, was delayed indefinitely to the continuing aggravation of neighboring residents.
• In September 2018, the contractor decided that the regulatory agencies had invented and enforced new consistency requirements that were not in the official regulations, and sued RTD claiming "force majeure" of a regulatory change. The complaint makes a fascinating read.
• In October 2018, the FRA provided the latest inspection report (of many) showing continuing non-compliance with the -5/+15 second consistency tolerance.
• On November 15th, 2018, the FRA fired off a letter indicating that it was fed up with the continuing grade crossing non-compliance, among other things, and threatened to shut down the entire commuter rail system by revoking the 2016 waiver.
• RTD is lawyering up against the FRA, and submitted a strongly worded legal memorandum with numerous exhibits effectively claiming that the grade crossing problem exists solely in the imagination of the regulators. RTD provided evidence that other railroads (including Caltrain!) commonly experienced long gate down times in violation of the criteria imposed on RTD.

Whatever happens next is sure to be dramatic. The entire saga can be reviewed under docket FRA-2016-0028, which organizes all the documents exchanged between RTD and the FRA relating to the temporary operating waiver.

Some Observations

Measured distribution of 38255 grade
crossing activation times in Denver.

1. Denver solved the wrong problem. They tried to invent a better mousetrap, something more sophisticated than a constant warning time grade crossing predictor. All they needed to do was to provide the same simple function with a substitute detection method that didn't rely on traditional audio-frequency AC circuits, which are incompatible with electrification. Instead, they decided to invent a better mousetrap involving lots of software, GPS, and wireless messaging, which naturally attracted regulatory scrutiny.
    
2. Complexity is bad. Multiplying the number of interfaces and creating dependencies between elements of the system leads to expensive aerospace avionics-like hardware and software that is cumbersome to deploy, test and maintain. System complexity leads to a proliferation of strange and unanticipated corner cases and failure modes.
    
3. Software can anticipate when to activate a crossing and prevent a train from showing up too soon, but there is no software in the world that can make a train show up on time.
    
4.  Grade crossing activation times naturally follow a statistical distribution that arises from random environmental factors beyond the control of the warning system. The low end of the distribution must never be shorter than the mandated 20 seconds, but the long end of the distribution will inevitably have some outliers. The diagram above shows the measured distribution of 38255 crossing activation times on RTD. Notice the long tail.
    
5. Even traditional "constant" warning time systems have this statistical tail. If the FRA inspectors applied the same regulatory zeal to Caltrain as they did to RTD, Caltrain would certainly be found in non-compliance. This isn't idle speculation: RTD gathered the data to prove it.
    
6. The criteria for non-compliance, namely a "significant difference" from the prescribed warning time, are subjective. Guidance from the FRA acknowledges as much: "Thus, prudent judgment must be exercised when reviewing the results of warning time testing to determine whether the actual warning time provided during testing was compliant with the standard."
    
7. The regulators painted themselves into a corner. They imposed a strict -5/+15 second criterion, which is easy to verify for an inspector with a stop watch and a clip board, but makes the long tail of the activation time distribution an automatic violation that is almost impossible to avoid. In recognition of the environmental factors beyond the control of the warning system, the regulators should have used controlled test conditions or applied a different criterion, such as X% of activations within Y% of programmed warning time. This is harder to verify for an inspector with a clipboard, but the grade crossing controller ought to be able to maintain these statistical records across a very large number of crossing activations.
    
8. While electrification is relatively rare in the US, there are numerous railroads abroad that have solved the constant warning time problem in electrified territory. This probably isn't rocket science. The mousetrap already exists.

Lessons for Caltrain

With the grade crossing warning system already at the top of the Caltrain electrification project's risk list and the contractor falling behind, this problem is already getting a lot of attention. The people involved hopefully already realize:

Keep it simple - the job is to come up with a grade crossing predictor that works in the presence of traction return currents. It will be tempting to come up with a more sophisticated custom solution that uses lots of software, but we learned from the CBOSS project, and Denver's travails, that complexity usually leads straight to disaster. The dumber the better.

Document existing conditions - a large database of activation time statistics should be assembled for each crossing as it exists today, to head off a conflict over the subjective nature of the FRA warning time consistency criteria. In the event of a Denver-like disagreement with FRA or CPUC, Caltrain would be in a position to quantify precisely how much more (and hopefully not less) consistent the new warning solution will be, regardless of the selected criterion. Caltrain enjoys the advantage that it isn't building new crossings like Denver, so there is an existing system performance baseline that is already accepted by regulators. That baseline will only be useful if it is thoroughly documented.

Plant the goal posts firmly - Work with FRA towards mutually agreed verification criteria that don't repeat the mistakes made in Denver of specifying a rigid range and then testing in the uncontrolled conditions of revenue service. The activation time distribution will always have a statistical tail. If the consistency criterion can't be met by today's existing grade crossing system, then it's probably a bad criterion.

Make sure we aren't paying for Denver - the contractor needs to be held accountable for the extent to which Caltrain electrification funds (and schedule delays!) are accruing to the Denver project's benefit, if the same grade crossing solution is ultimately pursued in both projects.