Real-Time Computing Report in Elephant Moraine, Antarctica
By: Mark Sibenac
Robotic Antarctic Meteorite Search
Year 2000 Expedition
2000-01-18 -- 2000-02-02

The Real-Time computer on Nomad consists of a VME cage with 7 boards. The main CPU board is a Pep Modular Computers VM62 with a 68060 at 40 MHz with 16 MB or RAM and 4 MB of Flash ROM. The other boards are various I/O cards. The CPU runs the VxWorks 5.3.1 real-time operating system.

So far there have been no problems with any of the processes running on the real-time computer. Not one of the processes crashed during uptime of the robot. The longest run-time was probably 12 hours.

Some of the code was modified to provide more telemetry and clean up minor problems that never got fixed before. A telemetry logger runs on another computer which archives most of the sensor data being produced by the real-time computer.

The new pendant is working extremely well in the Antarctic conditions. It is usually left sitting on the robot for easy access. It has not failed yet. The software that received commands from the pendant on the real-time system seems to have a bug which caused the robot to runaway out of control of the operator.

A problem was seen on the first day in the field where the Kill Switch I/O line seemed to fluctuate in software for no good reason. It did not affect the hardware at all. The connections were not checked even though the problem went away almost immediately. The problem could have been in the connectors, the solder connections to the switch, the I/O channel, or the CMOS chips that process the Kill Switch signal on a custom board inside of the motor amplifier cage.

One of the motor encoders gives a bad value every day or so. It is not always the same one. GPS messages get corrupted every few hours. Ded reckoning gets corrupted data every few days or so. None of these problems cause any disruptions.

On Jan 21, 2000, the robot was under control of the new pendant. A bad packet was received by the real-time system from the pendant, so the plug-n-play software switched to the old pendant code. A joystick calibration message went out to the driving task which screwed things up. The plug-n-play swtiched back to the new pendant, but the operator had no control over the joystick. The real-time system had to be rebooted to solve the problem. Shortly afterwards, the code was modified to prevent this problem from occurring in the future. This is probably the same problem that was seen on the first day of joysticking the robot in Elephant Moraine.

During the expedition and in McMurdo, the Flash PROM was reprogrammed about a half a dozen times. Three of the times, the Flash was corrupted which rendered the real-time useless. Human intervention is required to fix the problem by popping boards from the real-time cage, and installing jumpers. Luckily the days were good enough to pop the E-box to perform this task. The problem always occurred because the proper burning procedure was not followed. A hardware watchdog would reset the board once erasing and programming began. The board could not boot after that.

If the pendant is disabled before the robot comes to a complete stop, the robot will continue on at the old speed when the remote operator issues the command to enable the amps.

All compilation of the real-time system is now done on a fast linux laptop. The old sparc laptop (tire) is now out of the loop.

When a computer or the arm controller was power cycled via the relays connected to the real-time compture, something weird would happen to the wheel and steering motor amps. If they were enabled during a power cycling, no current would flow to the motor. The amps had to be disabled, and then reenabled to carry on. This was seen in Patriot Hills too.