Zero Genie

The program manager in charge of Boeing’s Starliner crew capsule program said Friday that additional checks would have uncovered problems with the spaceship’s software that plagued the craft’s first unpiloted orbital test flight in December, but he pushed back against suggestions that Boeing engineers took shortcuts during ground testing.

Boeing missed a pair of software errors during the Starliner’s Orbital Flight Test. One prevented the spacecraft from docking with the International Space Station, and the other could have resulted in catastrophic damage to the capsule during its return to Earth.

Both errors could have been caught before launch if Boeing had performed more thorough software testing on the ground, according to John Mulholland, vice president and manager of Boeing’s CST-100 Starliner program.

Mulholland said Boeing engineers performed testing of Starliner’s software in chunks, with each test focused on a specific segment of the mission. Boeing did not perform an end-to-end test of the entire software suite, and in some cases used stand-ins, or emulators, for flight computers.

“We are recommitting ourselves to the discipline needed to test and qualify our products,” Mulholland said Friday in a conference call with reporters. “The Boeing team is committed to the success of the Starliner program, and we are putting in the time and the resources to move forward.”

The Orbital Flight Test, or OFT, in December was intended to demonstrate the Starliner’s performance in space for the first time ahead of the capsule’s first flight with astronauts this year. The issues that plagued the OFT mission might force Boeing and NASA to plan a second unpiloted test flight before moving on to a crewed mission.

Officials have not decided whether another automated test flight might be required, or said when the Starliner might fly in space again.

Boeing developed the Starliner spacecraft under contract to NASA, which is seeking to end its sole reliance on Russian Soyuz crew capsules to ferry astronauts to and from the space station. NASA awarded Boeing a $4.2 billion contract and SpaceX received a $2.6 billion deal in 2014 to complete development of the Starliner and Crew Dragon spaceships.

The Crew Dragon completed a successful unpiloted test flight to the space station in March 2019, and then demonstrated the capsule’s in-flight launch abort capability in January. Final preparations are underway for the first Crew Dragon flight with astronauts on-board, which could take off as soon as May.

After the OFT mission exposed inadequate testing, Boeing’s engineers are examining every line of Starliner software to ensure teams did not miss any other errors that went undetected during the spacecraft’s December test flight.

“Hindsight uncovered a couple of the issues, but I really don’t want you or anyone to have the impression that this team tried to take shortcuts,” Mulholland said. “They didn’t. They did an abundance of testing, and in certain areas, obviously, we have gaps to go fill. But this is an incredibly talented and strong team.”

One of the software problems was immediately apparent after the Starliner’s otherwise successful ascent into space Dec. 20 from Cape Canaveral aboard a United Launch Alliance Atlas 5 rocket. A mission elapsed timer on the capsule had a wrong setting, causing the spacecraft to miss a planned engine firing soon after separating from the Atlas 5’s Centaur upper stage.

The orbit insertion burn was required to inject the Starliner capsule into a stable orbit and begin its pursuit of the space station. After the automated sequence failed due to the on-board timer setting, ground controllers at NASA’s Johnson Space Center in Houston had to uplink manual commands for the Starliner spacecraft to perform the orbit insertion burn, but the ship burned too much fuel during the process, leaving insufficient propellant to rendezvous and dock with the space station.

Ground teams in Houston also encountered trouble establishing a stable communications link with the Starliner when they attempted to send commands for the orbit insertion burn, further delaying the start of the maneuver. Boeing says ground teams had issues connecting with the spacecraft on more than 30 additional occasions during the Starliner’s two-day test flight.

With a docking to the space station no longer possible, mission managers cut short the Starliner test flight and targeted a landing of the capsule at White Sands Space Harbor, New Mexico, on Dec. 22.

After the mission timer problem, Boeing engineers reviewed other segments of the Starliner’s software code to search for other problem areas. They uncovered another software error that was missed in pre-flight testing, which could have caused the Starliner’s service module to slam into the craft’s crew module after the ship’s two elements separated just before re-entry into the atmosphere.

Controllers sent a software patch to the Starliner spacecraft to resolve the potential problem before it performed a deorbit burn to aim for landing in New Mexico.

Mulholland said Friday that more extensive testing before the Starliner test flight would have revealed the software errors.

Engineers traced the mission elapsed time problem to a coding error that caused the Starliner spacecraft retrieve the wrong time from the Atlas 5 rocket’s flight control system. The Starliner set its internal clocks based on a time captured from the Atlas 5’s computer hours before launch, when it should have retrieved the time from the launch vehicle in the terminal countdown.

Joint software simulations between Boeing and ULA focused only on the launch sequence, when the Starliner spacecraft is attached to the Atlas 5 rocket. The simulations ended at the time of the capsule’s deployment from the launcher, but testing would have revealed the timing error if the simulations continued through the time of the orbit insertion burn, which was scheduled to occur around a half-hour after liftoff.

“If we had run that integrated test for a number of minutes longer, it would have uncovered the issue,” Mulholland said.

“I think the sensitivity of this mission elapsed time was not recognized by the team and wasn’t believed to be an important aspect of the mission, so ideally we would have run that (software test) through at least … the first orbital insertion burn,” Mulholland said. “So from a hindsight standpoint, I think it’s very easy to see what we should have done because we uncovered an error.

“If we would have run the integrated test with ULA through the first orbital insertion burn timeframe, we would have seen that we would have missed the orbital insertion burn because the timing was corrupt,” he said. “When we got to that point in time, the software believed that the burn had happened many hours before, so it didn’t do the burn.”

Mulholland said Boeing teams thought it was more logical to break the Starliner mission phases into pieces, and run software testing on each segment of the flight.

“When you do a single run from launch to docking, that’s a 25-plus-hour single run in the computer,” he said.

“The team, at the time, decided that they would have multiple tests of different chunks of the mission,” Mulholland said. “It was not a matter at all of the team consciously shortcutting, or not doing what they believed was appropriate.”

Before every future Starliner mission, Boeing will run longer tests in software integration labs encompassing all events from launch through docking with the space station, then from undocking through landing, according to Mulholland.

Mulholland said more thorough testing could have also revealed the mis-configured software needed to safely jettison the Starliner’s service module before re-entry. Without a software patch, the service module, or propulsion element, could have rammed back into the crew module after separation, damaging the ship’s heat shield, or worse.

A propulsion controller is responsible for coordinating thruster burns on the service module to ensure it does not recontact the crew module after separation, which occurs after the Starliner’s deorbit burn and before re-entry.

The service module is designed to burn up in the atmosphere, while the reusable crew module descends back to Earth protected by a heat shield.

The propulsion controller on the Starliner service module is based on a design used by another program, and its software was improperly configured for the service module’s disposal burn after separating from the crew module, Mulholland said. The propulsion controller had a wrong “jet map,” which contains information about the service module’s thrusters and valves.

The Starliner uses two different jet maps: One when the entire spacecraft is connected — when the crew module computers command thruster firings — and another for the disposal burn after the service module is jettisoned.

“The only thing that was picked up was the one jet map for the integrated spacecraft, and we missed the jet map that was required for the service after separation,” Mulholland said.

He said software testing for the propulsion controller used an emulator, or a simulated component, rather than the actual controller intended to fly on the Starliner spacecraft. When Boeing ran the software simulation, the real propulsion controller was being used for test-firings of the service module thrusters in New Mexico.

“While that propulsion controller was outside supporting that other test was when they ran the qualification test of that section of the software, and because we had an incorrect emulator (and) it didn’t have the correct jet mapping, that issue was not uncovered during the qualification test,” Mulholland said. “Because that hardware was returned to the lab, we were able to, during the mission, re-run that sequence, identify the jet mapping issue and upload the software fix before we did the re-entry burn.”

One of many improvements Boeing says it is implementing is a requirement to ensure the proper hardware, avionics boxes and other components are included in future software tests.

“So if it is important to have a specific piece of avionics in the lab, we’ll be required to have that in there before we actually run the qualification test,” Mulholland said.

Another problem encountered during the Starliner test flight involved the ship’s communications link with NASA’s network of Tracking and Data Relay Satellites.

The spacecraft had trouble locking onto the TDRS network 37 times during the two-day test flight in December, according to Mulholland. Boeing engineers have identified the cause of one of the communications interruptions, which was caused by an explainable “false lock” condition, Mulholland said.

The other 36 instances of an unexpected communications outage all occurred over northern Europe and Russia, including on the Starliner’s first pass over the region minutes after launching from Florida. That’s when ground teams had trouble sending a command for the spacecraft to perform an orbit insertion burn after the mission elapsed timing error.

An independent review team chartered to investigate the problems that cropped up on the Starliner test flight is nearing the end of its inquiry. The results of the investigation will be announced next Friday, March 6.

But Mulholland said engineers are still looking into the communications issues, and a final verdict on the cause of the radio link interruptions is not expected next week.

Despite the problems in flight, the Starliner spacecraft safely returned to Earth and post-landing inspections show it can be flown again, Boeing says.

The ship’s heat shield and parachutes performed well, as did the Starliner’s life support systems, Mulholland said. Boeing was also able to test the functionality of the capsule’s docking system, but teams were unable to fully check the performance of the Starliner’s rendezvous and navigation sensors because the spacecraft did not dock with the space station.

Boeing technicians at NASA’s Kennedy Space Center in Florida are readying a second Starliner vehicle for the next test flight, whether it is a redo of the unpiloted OFT mission, or the first test flight with astronauts on-board.

Email the author.

Follow Stephen Clark on Twitter: @StephenClark1.