BART service resumes following earlier computer problem

03.09.19

BART service resumes following earlier computer problem

 

Read the update from Thursday March 14th into the cause of the network problem here.


Monday March 11th Update

At 2:45 am on Saturday we experienced a computer network failure. The failure was software related at one switch that is part of a complex computer network.  As a result, trains were not dispatched between 6am and 9am Saturday morning.

There was no maintenance being done on the system at the time of the failure. Therefore, we know the cause wasn’t contributed to work being done overnight.

We were able to safely restart the system and provide train service after staff performed field resets at 46 field locations and 68 traction power substations. Since the field resets took place, all train systems have performed as designed without failure.

BART staff is waiting for failure analysis results from Cisco to understand the exact cause of the failure.  Once we understand the exact cause we can determine any next steps needed.

All systems are designed to fail in a safe mode to ensure the safety of our riders and employees.   


10:50am Update: We have now resumed normal service systemwide.  We still have the planned maintenance work happening at Richmond station. Info is here: https://www.bart.gov/news/articles/2019/news20190308


10:25am Update: BART is open and we are recovering from earlier system wide computer and power problems. We are in the process of resuming normal service. However, there is currently no train service beyond Colma towards SFO and Millbrae. A bus bridge has been established between Daly City and Millbrae. 

New information related to the cause of the problem is that at 2:45 am we experienced a computer network failure. While we originally believed this was caused by planned maintenance work to our uninterruptible power supply system, we have now determined this work did not cause the network failure. 

BART will perform a complete forensic evaluation to determine exactly what happened. This analysis is expected to take several days due to the complexities of our system.


9am Update: BART is now open and is offering limited service while we resolve an earlier computer problem.

There is currenlty no train service between Daly City and Millbrae and SFO.  SamTrans is also providing a bus bridge between Millbrae and Daly City. There is no service to SFO at this time.

We will post updates here. 

HEADS Up: there are official service advisories that we tried to put out earlier but couldn't because of the network computer problem. Now that the system is back up they are being put out.  These are old and not accurate. We are in fact open and have limited service.


ARCHIVED INFO

BART crews working overnight on BART’s uninterruptible power supply in Oakland ran into problems that impacted our traction power supply system and train control routing system.

Without both these systems, we are not able to safely dispatch trains for service.

Traction power sends power to the trains and our routing system is how we program which direction and route the trains should maneuver.

We have multiple crews trouble shooting the problem to get both systems up and working.

Riders can use our Trip Planner at bart.gov and our official app to plan alternatives trips.

Our Trip Planner integrates the transit systems that serve the region. Just put in your starting point and destination and in options/transit options turn off BART.

Bus agencies that serve parallel bus service between our stations have granted our riders mutual aid (which means you don't have to pay).

SamTrans is also providing a bus bridge between Millbrae and Daly City.

We will post updates here and on Twitter at @SFBART.


Posted at 5:42am Saturday March 9th

Updated at 6:37am to add info about mutual aid.

Updated at 7:50am with Millbrae-Daly City bus bridge info.

Updated at 9am to add info about limited service.

Updated at 10:25am with information about resuming normal service.

Updated at 10:50am with info about normal service.