Mars Pathfinder Kept Resetting Itself on Mars. The Fix Was a Setting Its Engineers Had Missed.

In July 1997, soon after NASA’s Mars Pathfinder landed on Mars and began gathering weather data, the lander’s computer started resetting itself. Each reset pushed the rest of that day’s planned work to the next day, though data already collected was not lost.

The cause was in the software. A low-priority task that handled weather data was holding a lock that a high-priority task needed, while tasks of medium priority kept running. Engineers call this priority inversion.

The bug was not new. “We did see the problem before landing but could not get it to repeat when we tried to track it down,” wrote Glenn Reeves, who led the Pathfinder flight software team at NASA’s Jet Propulsion Laboratory (JPL). On Mars, the resets kept coming back, and this time the team’s trace data let it track down the cause.

Reeves later disputed the famous retelling, in which engineers fixed Pathfinder through a C interpreter that had been left on by luck.

How a weather task held up the lander’s data bus

Pathfinder’s flight software ran on VxWorks, a commercial real-time operating system from Wind River. Two tasks controlled the data bus and the instruments attached to it, including the lander’s weather instrument. The first, bc_sched, set up the transactions on the bus for each cycle, eight cycles a second, and had the highest priority in the system apart from one of VxWorks’s own tasks. The second, bc_dist, collected the results of each cycle and distributed the data. Every cycle, each task checked that the other had finished its work.

Tasks passed messages through VxWorks pipes and waited on them with select(), which relied on a mutex semaphore — a lock that only one task can hold at a time. The weather task, which had a much lower priority, still held that mutex when several medium-priority tasks preempted it. Then bc_dist tried to write to a pipe and blocked, waiting for the same lock. With medium-priority work able to run ahead of it, the weather task did not get to release the lock in time. When bc_sched next checked, bc_dist had not completed its cycle. bc_sched declared an error, and the error reset the computer.

A 1990 paper by Lui Sha, Ragunathan Rajkumar and John Lehoczky had analyzed this trap and a remedy for it, priority inheritance. While a low-priority task holds a lock that a higher-priority task is waiting for, the lock holder runs at the higher priority, so medium-priority tasks cannot get in between. VxWorks offered that option. On Pathfinder, the semaphores behind select() were created without it.

Finding the bug in the lab

A reset was not fatal for Pathfinder. The team had designed the software to survive resets: “We built in the ability (and tested it) to go through multiple resets while we were going through the Martian atmosphere,” Reeves wrote. Before landing, while the team was “concentrating heavily on the entry and landing software,” it considered the unexplained reset a lower-priority problem and ran out of time for such issues.

The team had deliberately kept a trace and log facility in the flight software. David Cummings, one of the software engineers, built it, and Lisa Stanley of Wind River instrumented VxWorks services such as pipes and select() for it. The software already stopped the trace and saved its data when this error occurred, so the team did not have to change any code to start testing. They reran the same activities in the lab. “In less than 18 hours, we were able to repeat the problem,” Reeves later wrote in Dr. Dobb’s Journal. His 1997 account describes what came next: “Once we were able to reproduce the failure the priority inversion problem was obvious.”

Why was priority inheritance off? Wind River had deliberately left it out of the default for its select library, Reeves wrote. The team had made no conscious decision to leave it disabled: “We just missed it.”

The fix: one global variable, sent as a patch

The fix was small. The team changed a global variable so that the semaphores select() created would include the priority-inheritance option. The change would apply to every select semaphore created afterward, so it was checked by analysis and test. Wind River concluded that the performance impact would be minimal and that select() would behave the same as long as only one task waited on any given file descriptor. “We tested the system extensively before we changed the software on the spacecraft,” Reeves wrote.

The patch went to Mars as a diff: the differences between the code on board and the version the team wanted. “This corrected the problem,” Reeves wrote.

What the famous retelling gets wrong

The story spread through a summary by Mike Jones of Microsoft that circulated by email as “What really happened on Mars?” and was posted to the RISKS Digest mailing list in December 1997. Jones based it on a keynote by David Wilner, chief technical officer of Wind River Systems, at the Real-Time Systems Symposium. In Jones’s version, JPL engineers “fortuitously” launched the spacecraft with the VxWorks shell — a C interpreter — enabled and fixed the bug by uploading a short C program through it. He also wrote that each reset caused losses of data and that one or two resets during preflight testing had been put down to a hardware glitch.

Reeves answered on December 15, 1997. “No, we did not use the vxWorks shell to change the software (although the shell is usable on the spacecraft),” he wrote. The debugging features “were not ‘fortuitously’ left enabled but remain in the software by design.” No data that had already been collected was lost. The earlier sighting had not been written off either: “It was not forgotten nor was it deemed unimportant.”

What Pathfinder’s resets show about locks and priorities

A priority is only as good as the locks a task waits on. The bus task bc_dist outranked the weather task, but once it needed the weather task’s lock, the medium-priority tasks decided how long it waited.

Know your platform’s defaults. Priority inheritance was off by default for select() in an off-the-shelf operating system. Reeves drew the lesson for anyone flying commercial off-the-shelf (COTS) software: “make sure you know how it works.”

Keep the instrumentation that shows you a failure. The bug had escaped the team before landing. With the trace facility already in place, the lab needed less than 18 hours to make it happen again — and once it did, the cause was obvious.

The fix did not make the weather task more important. It let the weather task borrow bc_dist’s priority just long enough to give the lock back.

Explore

Stay connected