Download the final report from the peer review here (500 kB PDF).
It's fair to say our worst fears have come true:
- the project manager does not have the requisite technical experience
- there is no project schedule, and October 2016 is just another month on the calendar
- inter-operability requirements and test methods are not defined or agreed upon
- configuration management is not just out of control, but completely lacking as a process
- software and network security is an afterthought
- animosities between project management and the contractor are impeding the resolution of technical issues
- operator training has not started, and the materials for such do not yet exist
- weekly top-level status meetings between Caltrain management, the program management consultant, and the contractor had not been occurring
The list of documents reviewed by the panel in Appendix C would make a juicy FOIA request.
A little bird overheard some discussions that do not appear in the APTA final report, because the report is intended to provide only constructive criticism to help Caltrain out of this mess. It's even uglier than you could have imagined:
A little bird overheard some discussions that do not appear in the APTA final report, because the report is intended to provide only constructive criticism to help Caltrain out of this mess. It's even uglier than you could have imagined:
- Parsons Transportation Group (PTG), Caltrain's prime contractor, does not have the right skills mix to manage complex system integration on 13 different subcontracts
- PTG is fearful that the commercial terms of the CBOSS contract expose them to legal action by Caltrain, contributing to the lack of transparency
- Subcontractor General Electric (now Alstom) discovered that simply re-using the existing ITCS product wasn't going to work. The inter-operable version of the product is incurring massive increases of scope that were not accounted for in the original contract
- Because of the extent of the changes made to ITCS, the FRA is requiring the same certification and type approval process as for a new PTC system, undermining Caltrain's claim to reusing an off-the-shelf technology
- The FRA has taken the position that Caltrain is really installing two PTC systems, requiring full testing of both I-ITCS and IETMS (the system that will be used by Union Pacific freight trains on the peninsula corridor)
- Inter-operability means not only allowing IETMS equipment to operate in CBOSS territory, but also allowing CBOSS equipment to operate in IETMS territory, something that Union Pacific has been concerned about testing thoroughly
- Poor coordination for accessing an operating railroad for system installation and testing has been and continues to be a bottleneck
- Additional funding is going to be needed, but nobody knows how much more
- A change of contract operator (currently Transit America Services, Inc, soon coming up for re-bid) would introduce significant program execution risk
- Getting all the CBOSS-equipped trains into revenue service could take up to 5 months
The already egregious sum of $231 million to cover a measly 51 route-miles with PTC is about to increase significantly, something you would never guess from the latest CBOSS update provided to Caltrain's laissez-faire board of directors.
Fast forwarding to whatever year it eventually takes place, the RSD (Revenue Service Demonstration) will consist of flipping the "on" switch and transforming rush hour into an epic cascade of software glitches reminiscent of the 1998 MUNI Meltdown. On that day, we will all know that this CBOSS turkey has finally come home to roost, as was foretold way back in 2009.
Posts
So when HSR comes through with ERTMS, the Peninsula will have to deal with 3 operating PTC systems? Wow.
ReplyDeleteNot necessarily; if the High Speed rolling stock is equipped with CBOSS, there is no need for ERTMS on the Peninsula.
DeleteHowever, it might actually become cheaper to trash CBOSS and get ERTMS instead; ERTMS is operational in several places, with higher traffic density, higher speeds, and more complex environment than ever possible on the Peninsula.
Yes, of course, but there still has to be a section for the ERTMS-CBOSS switchover. And HSR trains would also need to be equipped for Metrolink PTC -- in addition to CBOSS and ERTMS.
DeleteSwitching signalling system while running is pretty much routine; SNCF has been doing so for 30 years. Multiple signalling systems are also routine, for example on the Eurostar trains, or the German ICE 3 which are certified for operation in the Netherlands, or the TGVs operating into Germany and Switzerland.
DeleteIn other words, absolutely feasible, and proven methods.
Well, with Freight moving to PTC and HSR to ERTMS, the options are:
Delete1) Install ERTMS and find a way to deal with UP (historically not a HSR friendly) traffic. It's politically challenging.
2) Install PTC which makes UP interoperability easy, but adds burden to HSR.
3) Work with businesses to remove Freight altogether.
#3 is just a non-starter given the volume of traffic. 4 trains a day with ~50 cars each is a LOT to offload.
Given dealing with UP vs HSR, dealing with HSR seems easier since both share common goal, so slapping PTC on HSR trains seems easier (which might be necessary in LA area anyway)
I think the things went bad when Caltrain got greedy and asked for too many ERTMS features bolted onto PTC.
Max,
DeleteI'm not sure it's quite the same. Caltrain, apparently, will have two signal systems (I-ITCS and IETMS) operating simultaneously on the same track. That is not a routine matter for Sncf (or anyone else).
It's a single set of wayside and back office systems that will have to undergo twice the testing. It's not two separate systems.
Delete"Well, with Freight moving to PTC and HSR to ERTMS, the options are:"
DeleteHardly.
The best, cheapest, most reliable, most efficient, most ecological solution for the Caltrain corridor is what it always has been and always will be:
* Install ETCS/ERTMS (off the multi-vendor shelves, zero customization including no mph), get rid of all line side signals, and get rid of freight until regulation changes.
* Terminate and prosecute anybody in any way associated with the multi-hundred-million dollar outright unambiguous fraud that is CBOSS.
That's it. Done.
The second best solution remains what it always has been:
* Install ETCS/ERTMS (off the multi-vendor shelves, zero customization including no mph), get rid of all line side signals.
* Terminate and prosecute anybody in any way associated with the multi-hundred-million dollar outright unambiguous fraud that is CBOSS.
* 100% segregate Caltrain from FRA+freight in San Jose and Santa Clara. Freight, ACE, Amtrak get one track and a pair of platforms tracks at SJ Cahill, one track and one platform at Santa Clara. A second FRA+freight track SJ-Santa Clara is a medium-term (but likely unnecessary and unjustifiable) build-out that might be investigated.
* If freight is (stupidly, uneconomically, unproductively,) forced to be accommodated along a route where there is neither economic nor ecological justification, do the following:
** Create three huge blocks that extend
1. Santa Clara ("CP Coast" junction) to Redwood Junction;
2. Redwood Junction to South San Francisco; and
3. (God help us, as the self-perpetuating insanity of Port of San Francisco simply needs to be killed off) South San Francisco to Quint Street.
** These huge blocks are switchable from ETCS/ERTMS or whatever-the-hell-freight with no "overlay", no mixed use. The section of track is either Caltrain or Not-Caltrain. No train need be equipped for dual-mode (Caltrain and Not-Caltrain operation, a massive, massive implementation and maintenance and regulatory savings.
** Switch-over occurs only when no train of any type can be proven to occupy that block.
** Overrun of the ends of the blocks and overrun into the blocks into or from other-system territory are enforced by turnouts and derails; it is physically impossible for a wrong-system train to enter or exit the block. (This is, or at least definitely was, completely acceptable per FTA PTC mandates.)
** The huge blocks include only the "northbound" (easternmost) through tracks (MT-2, MT-4) and associated sidings and freight leads. "Southbound" (westernmost) through tracks are 100% ETCS/ERTMS at all times. All MT-1/MT-2 crossovers are locked in parallel mode at all times when the containing block is in Not-Caltrain mode.
** It really doesn't matter what non-Caltrain signaling system is used in the huge blocks. All that matters is that when they are in Not-Caltrain mode that Caltrain-mode trains are not present and cannot enter, and that Not-Caltrain trains cannot exit. Ideally it would be "dark territory", with only a single train authorized to move, occupying the entire block, doing whatever the hell it likes, with no intermediate signals at all; just the absolute signals that mark the ends of the huge blocks. But it could be some sort of whacky AAR PTC freight business: the point is that it simply doesn't matter and it is somebody else's problem; all the Caltrain side knows is that big chunks of northbound track have been "removed from the map" and are unavailable, as if those tracks simply don't exist.
** There almost certainly isn't a need for three giant blocks. Just one. A competently operated Caltrain with very small amounts of strategic infrastructure (island platforms and FSSF where Takt-headway train crossings occur) can easily be operated off-peak on a single-track line. Or, simpler yet, confine freight to a 1am to 5am window and don't even think about Caltrain meets.
@Drunk Engineer: Actually, it is routine operation for SNCF. The Paris - Strasbourg HSL has ETCS-L2 AND TVM-430 in concurrent use. ETCL-L2 has precedence, which means, if the vehicle is ETCS-L2 enabled, it will operate under this system; if not, it will run under TVM-430.
DeleteA similar scenario can be found in the subway system in Nürnberg, which has concurrent driverless and driver-operated operation.
Or to stay in Germany, some HSL have concurrent LZB and wayside signalling in operation.
Therefore not that common, but operational.
@Martin: What about option 4: turn the Caltrain line into a shortline with no through-running from UP. Locomotives for the Caltrain shortline would be CBOSS-equipped.
DeleteOr, 4a, following Richard's suggestion, dump CBOSS, and have the locomotives for the Caltrain shortline ERTMS-equipped.
Interfaces suck. For a good system design, remove interfaces if you can.
DeleteMax: You and I have very different definitions of 'routine'. The ETCS/TVM combination was a major effort.
DeleteClem: According to the bible, the amount of validation needed is far more than 2X. This scheme is unnatural.
@Drunk Engineer - I think Max meant switching over between systems while the trains are in motion is routine, not installing overlay services.
Delete@Drunk Engineer: With "routine" I mean that something happens regularly, in regular (day-by-day) operation. This does not mean that the way to make it routine was easy. It is extremely difficult, particularly as the virtual and physical blocks don't match.
DeleteI don't see why CHSR is wanting to drive itself off a cliff with a system that is reliant on a full time wireless link. Use the rails like TVM430 or an antenna loop like LZB. Haven't the railroads noticed that the promised "low cost" of wireless PTC has been anything but? The fallacy of low wireless cost (promoted by signal vendors) is the whole reason we're in this PTC mess to begin with.
Delete@Martin @Max Wyss:
DeleteRead #6 on this:
http://caltrain-hsr.blogspot.com/2010/05/top-10-reasons-for-peninsula-bart.html
Isn't Shinkansen still using track-circuit based PTC systems? Ironically, if CHSRA went with the original vision of segregated tracks for HSR trains, it would have a more simplified environment to deploy less complicated PTC systems for both Caltrain and CHSR.
Delete@Mike B: ETCS Level 2 is communication-based. It uses, however, Balises to improve the precision of the determination of the actual position. (this eliminates the incertainity of GPS positioning).
DeleteThe problem with LZB is that it is very expensive; way more than ETCS L2. With LZB, it is possible to run moving blocks, and the position of the train is always known with sufficient precision.
TVM430 is extremely inflexible, with fixed blocks, and less capacity than ETCS L2. It also has a very limited number of line speeds, and upgrades are complicated. In fact, the bad accident last year with the test train on the Paris - Strasbourg line required to deactivate TVM430, and thus lose all protection (with fatal results, as shown).
Comparable overspeed tests in the Gotthard Base Tunnel were run under full protection of ETCS, by adjusting some parameters, such as maximum speed for line and vehicle. Sure, the modifications are not for daily changes, but there was no (fast) test run in the Gotthard Base Tunnel which was not fully protected by ETCS.
There is a "low cost" variant of ETCS, namely some kind of Level 3 (which includes the integrity of the train), used with (D)MUs on secondary lines. Such a low-capacity system can be set up pretty cheaply.
Communication-based signalling does cost less than conventional signalling, because of the much reduced amount of fixed installations, which not only are expensive in procurement, but also in operation.
That's why Caltrain is doing an overlay PTC system. It has all the costs of a bleeding-edge communications-based solution, plus the O&M expense of a full set of conventional line side signals! Truly the gift that keeps on giving. Overlay away!
Delete"It has all the costs of a ... plus ... Truly the gift that keeps on giving. Overlay away!"
DeleteIt's even better than that!
Bundled into the "electrification" budget (and one should also use scare-quotes around "budget", because they've already blown it out many times, even starting at an unsupportable crazy high level at which it already had negative return!) is total replacement of all the lineside signals and track circuits and relay-based "control points".
Because what the world needs is more track circuits with old fashioned insulated joints and impedance bonds. And relays. (Relays!) And lineside signals. And new lineside signals, because the electrification masts and portals (so so many of them!) get in the way of then old ones. Because without all of this, without the relays and the track circuits and the many many lineside signals, what would CBOSS have to Overlay with?
You can't make this shit up. But they did. They keep making up more shit, and getting away with it. Many hundreds of millions of dollars of shit, in a huge steaming shit pile that keeps growing and growing and growing to the heavens.
Who is to blame ... should heads roll, and if so, which ones?
ReplyDeleteHow did we get here?
And what's the best path out of this money furnace? Cut loses now?
They Promised Us Jet Packs. They Promised the Bosses Profit.
"One of the best ways to save money [...] is to encourage employees to kill projects before they become expensive [with bonuses]. X staff meetings begin with a “premortem” process in which people predict how various technologies might fall short. Employees’ laptops are decorated with stickers of crumpled-up paper that represent the end of past efforts."
Serious question: "SO WHAT?"
ReplyDeleteThis is a project in which every involved party, without exception, profits from more and larger failure.
Nobody profits from rethinking.
Contractors : further problems, further delays = more added scope, more Engineering Change Orders, more ...
Oh, and the contractors are the sole source for proprietary unique technology.
Caltrain "owner's representatives" (another set of contractors, permanently ensconced within the public agency but never working for the public interest): massive payoffs, rocketing to the heavens as the depth and height of the steaming pile increases. Fingers in every pie. In a perfect position. Create the difficulties themselves, report on intractability of difficulties, blame others for difficulties, design fixes which create more difficulties, enlarge oversight role in reviewing implementation of (non-)fixes, etc, etc, etc. Oh, and they write their own budgets, which Caltrain's board will approve without question.
Caltrain in-house public agency technical staff: negligible in number, negligible in influence. Anybody who might rock the boat has been pushed overboard, or was never allowed on.
Caltrain in-house public agency executive staff: CEO utterly totally unqualified to hold this or any other position. His interests, like his culpable predecessor, lie solely in drawing three public salaries (SMCTA, SMCTD PCJPB) and amassing three public pensions. No interest in service, no domain expertise, no managerial skill, no analytical ability, no motivation to rock the boat.
Caltrain political oversight board: Laugh Out Loud.
Where will any pressure come from to fix what readers of this blog perceive to be problems, but which are not problems at all for any of the actors involved?
Caltrain political oversight board?
Caltrain executive staff?
State funding agencies? (Answer: no, as their executive/staff/consultant level of functionality differs little from Caltrain's)
Federal funding agencies? Unknown, but seems unlikely given that they've gone along with this farce for a decade.)
FTA? This is just one more failing Not Invented Here PTC system which won't meet the mandates. Deadlines will be allowed to slip. No pressure?
Legal action? By whom? With what standing? Impossible burden of proof for criminal prosecution.
Why not deploy etcs level 2 on the entire caltrain-hsr-Metrolink corridor and pay for UP to install the equipment on a few gensets, and either provide funding for or require Metrolink to install the etcs equipment on their f125's?
ReplyDeleteThat would have been a perfectly sensible thing to do from a purely technical standpoint. But arguments about interoperability centered around the all-important accommodations for tenant railroads, which make the system integration job infinitely more exciting (as Caltrain is now finding out). One of the smartest ways out of this mess is to segregate the blended system as described by Richard Mlynarik in the comment a few slots above.
DeleteAccommodations weren't seen as an issue on the NEC where CSX and NS have to maintain a small pool of ACSES/ATC/LSL equipped locomotives. Freight traffic along the Coast and Peninsular subs run at about the same level of intensity as the NEC.
DeleteMetrolink does seem even worse than Caltrain in pretty much every fashion. (Yes, such a thing is possible, it turns out. Yes, you have to try really hard in this USA USA USA race to the bottom.)
DeleteAs far as I can judge, with very limited local knowledge, the best thing to do is simply ignore the existing Metrolink "network" (LAMTA seems to be doing exactly that with its various transit expansion plans, for better or worse) and build separate, segregated, non-FRA, non-freight, HSR-compatible, ETCS-only rail adjacent to (or replacing, in some locations) the freight/Metrolink tracks in the Tejon—Magic Mountain—Santa Clarita—Sylmar—Burbank—LAUS—Fullerton—Anaheim—Orange corridor.
Again, no dual-system trains, no compatibility with freight, keep things clean, keep things simple, keep the dinosaurs as far away as possible.
Don't borrow trouble!
This does seem sort of a stupid move according to the "organization-electronics-concrete" mantra, but when utterly intractable organizational dysfunctional makes concrete cheaper than electronics, well, one has a public service obligation to pursue the most cost-effective option.
Since the Caltrain trunk is almost a passenger only isolated system they should go with cab signals and ACSES and just dual-equip the part from CP-COAST through LICK with IETMS for UP, ACE and Amtrak run-through. Then just have ACSES equipped UP units for the local traffic on the Caltrain trunk.
ReplyDeleteDon't dual equip. Segregate. Eliminate all interfaces that aren't absolutely necessary, and build additional tracks if needed between CP Coast and San Jose. This is way cheaper than complex system integration.
DeleteStanding up two parallel systems systems IS segregation. This isn't CBTC. Yes it costs more, but you don't have to have stuff talk to eachother. They just work as intended on their own. It's like how one can have cab signals and wayside signals on the same track.
DeleteClem writes:
ReplyDelete"A little bird overheard some discussions that do not appear in the APTA final report, because the report is intended to provide only constructive criticism to help Caltrain out of this mess. It's even uglier than you could have imagined:"
Clem,
I cannot imagine why *your* imagination -- or your estimate of your readers' imagination -- is so low. It borders on insulting. What your "little-bird" reports, is pretty much what I expected.
Sad to say.
CBOSS is a stinking pile of crap.
ReplyDeleteSo what can be done about it? I have mentioned that it should be brought out in the open through one of the local TV stations investigative units. I suggested and communicated with NBC 11, since they have already investigated Samtrans for questionable accounting practices, but I can't provide the technical expertise on this stuff. Someone has not yet taken the ball and ran with it. Are Clem, Richard or Roland for real? Or is it that your arguments don't hold any water?
I watched the video of the JPB meeting: https://youtu.be/PPdDRL8e0aw
Roland mentions something about cover-ups, the federal grand jury, etc. The CBOSS fiasco has made the front page of the Palo Alto Daily Post. So when is the serious investigation of Caltrain management going to begin? What about former judge Quentin Kopp? He supports HSR but questions the so-called blended plan. Something needs to be done to stop the lunatic fringe at Caltrain before they sink all our tax money and transportation system into the ground.
Mike, no idea. No idea at all.
DeleteRichard, what do you mean by that?
DeleteSorry used the wrong link. Use this for JPB meeting: https://youtu.be/UDyOwIdSoNU?t=9
ReplyDelete@Mike B2 ... serial anti-HSRA suing attorney and CC-HSR board member Mike Brady, is that you?
DeleteNo that's not me, although I do support the efforts of CC-HSR and High Speed Boondoggle.
DeleteFun fact: I offered the former Caltrain CEO (under whose oversight and leadership CBOSS was conceived and approved) a wager of a year's salary (his or mine) that CBOSS would be operational within two years of the advertised date, nor would it come in within 50% of budget.
ReplyDeleteOddly enough, he wouldn't take my bet.
... CBOSS wouldn't be operational ...
DeleteWhen was that? I have not seen Richard at any Caltrain meetings in over 5 years, maybe 10 years?
Delete@Jeff: Richard didn't say anything about a meeting ... believe it or not, people can converse outside of public meetings.
DeleteExactly!!! That's why I asked the question. I don't see it being a cordial conversation with all the vitriol Richard lashes out against them over the years.
Delete@Jeff, first you asked when Richard made the offer, adding that you hadn't seen Richard at any meetings in years. Now you're guessing it wasn't a cordial. Richard's statement was simple and clear, but I don't follow what you're getting at, or its relevance. If you have a point, try stating it succinctly and clearly.
DeleteWhatever... I just posed a simple question and made a casual remark. Can't be anymore concise than that. It's not worth any more discussion.
DeleteOK. I mistakenly thought you might have been trying to make a point or add some useful information or perspective on Caltrain's CBOSS misadventure.
DeleteWhat I would like to see is a simple spreadsheet of PTC systems each agency plans to use or is using. The details could include cost, track mileage, rolling stock, routes, shared with other agencies (freight or passenger), cost/mile, timeline, and any other relevant data. How does CBOSS compare to PTC for Metrolink, Chicago Metra, LIRR, Metro North, etc? I have asked Caltrain for these kinds of details a number of times, but have yet to see anything of relevance. Sure we can dig through pages upon pages of agency websites, but who has time for that? They love doing PowerPoint presentations, why can’t they provide a slide that shows such information?
DeleteWhy was Caltrain allowed to get into this mess?
DeletePTC was mandated by Congress after the deadly Metrolink Chatsworth collision in September 2008. PTC was supposed to be implemented by Dec. 31, 2015, but the railroads lobbied to extend the deadline, which is now Dec 31, 2018, which would be over 10 years since the Chatsworth accident. The primary objection was the cost, possibly in the billions to equip some 70,000 miles of track.
There are proven PTC systems in use around the world, some of which have been for decades, as has been pointed out numerous times. Why was each individual RR/Agency allowed to develop or implement their own? Why didn’t Congress, FRA, NTSB, FTA, APTA, etc. work to make use of a proven PTC system?
It shouldn’t be this difficult, well maybe not. After all this is the automobile-centric USA, crazy regulations, buy American, etc.
One dollop Not Invented Here, and a misplaced notion that America is so big that only radio-based solutions can work thanks to their minimal wayside footprint.
DeleteEven ACSES would have been cheaper than this unproven CBOSS system that is "94% done" and will remain so for some years to come, if it ever actually enters its operational phase of global uniqueness.
Remember how I said in the post that the list of documents would make a juicy public records request? CARRD made short work of it, and Caltrain delivered:
ReplyDeletehttps://www.dropbox.com/sh/1iysf74hawhzoiu/AADao5P_WSGZ9fYMATHIV_o5a?dl=0
clickable link to peer review supporting documents.
DeleteIs this the beginning of CBOSS-Chicago?
ReplyDeletehttps://metrarail.com/riding-metra/service-updates/positive-train-control-ptc
Oh… it is so especially complicated because the Chicago area rail network is so complex…
Is CBOSS embedded in the Electrification project? That is, are there parts of CBOSS that need to be finished, or redone as a function of Electrification?
ReplyDeleteThere is a lot of signalling work included in the electrification contracts. What is and isn't part of getting CBOSS to work properly is and will be murky, and it would be easy (given the large program reserves and contingencies) for some blurring to occur, in ways that are contractually unassailable.
Delete