Down To Earth Problems In High Earth Orbit

Ball Aerospace Depends on the Ada Programming Language to Isolate High-flying Hardware Problems and Keep a Classified Defense Satellite Operational

Product: TADS 1750A

s_success_i_0008Since Sputnik first turned the world’s eyes toward the night sky in 1957, satellites have become an integral part of our interconnected world. Today, defense applications like the global positioning system help people easily pinpoint their exact location on the planet surface, while commercial satellites bounce radio, television, and telecommunication signals around the globe in the blink of an eye.

Colorado-based Ball Aerospace & Technologies, a subsidiary of Ball Corporation, has been involved from the beginning, designing scientific rocket payloads soon after entering the industry in 1956. Ball’s distinguished history in space-based research and communication technology includes a long list of technical achievements, from building the second satellite in NASA history to flying the first scientific instruments aboard Skylab. More recently they completed optical repairs on the orbiting Hubble telescope.

“We specialize in satellites for commercial and defense contractors around the world, including onboard instrumentation,” says Bill Brown, the Principal Software Engineer at Ball initially responsible for repairing a problematic defense satellite lifted into orbit in February 1998.

Brown explains that the Ada programming language is used to write the software that controls the bird, from attitude control and internal monitoring to the reception of commands and transmission of telemetry. “Basically, we’re using the Tartan Ada Development System to control a satellite, and as part of the command language there’s a “patch upload” so the satellite software can modify itself,” he says.

The problems began when the classified satellite’s software began crashing every five days after initial deployment in April 1998. A lack of exception handling in the original code made spotting problems in space difficult from the ground, but the Ball team undertook a difficult and thorough “torture testing” of the software. They solved the dilemma when an Ada “storage error” flag pointed them toward a heap allocation problem, but after fixes were in place the satellite continued to crash.

From there, using a ground system with hardware identical to the satellite in space and a software simulator, they began testing theories on why the satellite software kept resetting and uncovered several hardware problems. Unable to duplicate the problems on the ground, it was while probing the internals of the Tartan Ada Development System itself, that an entirely different problem also surfaced.

With fundamental questions about the development system, and a software license that had lapsed after project development — largely because Tartan had been acquired by Texas Instruments — Brown had no direct route to answers. Certain he would need high-level information about the compiler and run time systems, as well as how their satellite hardware would react to particular aspects of the code produced by the Tartan Ada compiler, he contacted the Phoenix office of DDC-I.

An expert in Ada development for safety-critical embedded systems, DDC-I had purchased the Tartan product line from TI in early ’98 to round out their arsenal of Ada development systems. Left largely unsupported under TI ownership, DDC-I’s first goal for the mature Tartan Ada systems targeting the Intel i960, Motorola 68XXX, and MIL-STD-1750A processors was to beef up engineering services and support.

“We knew DDC-I had purchased the Tartan Ada Development System, so we approached them in April ’98 asking for help,” he says. “They immediately set us up on an annual maintenance contract as if we’d been with them from the start, sent over the latest version of the compiler, and helped us set it up.”

Headquartered in Lyngby, Denmark and Phoenix, Arizona, DDC-I is a long-standing developer of Ada development systems and tools, including compilers, run time systems, and debuggers. Beyond their products, DDC-I also offers consulting and support services to a distinguished list of commercial and defense contractors, including Boeing, Sikorsky, and Lockheed-Martin.

Beyond industry standard support packages, DDC-I will also tailor client-specific support packages for an unusual situation like Ball’s, containing the necessary elements from a list that includes: full consultation, installation and training, user documentation, custom software development services, and basic hotline or additional high-level engineering support.

Convinced that their satellite’s ongoing software problems were rooted in the hardware problems, Brown’s team set out to find software workarounds to resolve the situation. Enter Mike Gillmore, Senior Customer Support Engineer at DDC-I. “Mike has been our best resource for answering detailed questions about the safest course when it comes to our hardware and how it will likely react to any code changes,” says Brown.

With Gillmore’s insights into how the satellite hardware is likely reacting to specific constructs in Ada and the mechanics of the various pieces of the development system, the Ball team was eventually able to isolate specifically what they needed to know by looking at the post-hex dump assembly language.

Working back and forth with Gillmore and a team of DDC-I engineers on two continents, the Ball team resolved issues regarding the internal behavior of the Tartan Ada compiler and how the run time systems onboard the satellite reacts to hardware exceptions. With the continuing crash problem now occuring on a seven day cycle, an initial software patch was uploaded to reset the system every seven days.

“Basically, when we do a patch we’re only patching the ROM version of the code,” Brown says. They take the code running out of RAM and use it to modify the corresponding ROM image that’s brought into RAM when we boot. This doesn’t interfere with the running code, only with what comes up at the next boot up.”

Brown adds that sending software patches up to a multi-million dollar satellite is a touchy operation, and great care is required because the satellite in question doesn’t have a boot modifier that supports the ability to start up a small piece of the system to get things done. This all-or-nothing dilemma mandates great care in the patch upload process, as a single mistake in the new code sent up can hang the entire system.

The fact that the satellite was originally designed using Ada, before the Dept. of Defense (DoD) mandate to develop safety-critical embedded systems with Ada was lifted in 1997, has been a boon to their efforts in Brown’s mind. “I would never want to implement what we have in any language but Ada, and the fact that we are using Ada significantly improved our chances at figuring out what was wrong. It eliminated a whole host of possible problems just because of the nature of the language, so we could concentrate on the few things that could go wrong after it had been compiled.”

As a Ball Aerospace employee and DoD contractor Brown can enumerate the benefits of programming in the Ada language but can’t endorse products or companies. However, his final words on the level of support provided by DDC-I’s Gillmore are quite clear: “Mike has been very helpful.”