Exploring Tekkotsu Programming on Mobile Robots:

The Storyboard Tool

Prev: Defining new types
Up: Contents
Next: ---

Contents: Overview, Installing the storyboard tool, Examining a storyboard, Modifying a layout, Generating an execution trace, Downloading a state machine model, Logging messages and images, Known bugs

Overview

The Storyboard Tool allows you to monitor the execution of a state machine and display a graphical execution trace called a "storyboard". The tool is described in a 2004 senior honors thesis by Akkarit Sangpetch of Carnegie Mellon. Click on the thumbnails below to see some representative screen captures.

 

The tool produces three types of files, all of which are encoded using XML representation:

  1. A model file (extension .tsm) defines a state machine by listing the name and type of each state node, and the name, type, sources, and destinations of each transition.

  2. A layout file (extension .tsl) specifies how the model should be laid out on the screen. You can have multiple layouts to display the same model in different ways.

  3. An execution trace file (extension .tse) is a log from a run of the state machine. It contains state activation and deactivation events, transition firings, sensor events, and other optional information, including user-generated messages. It is dependent on the model file, and can be displayed in different ways by using different layout files.

Installing the Storyboard Tool

There are three versions of the Storyboard Tool: for Linux, MacOS, and Windows. Since they're relatively large files, and are operating system-specific, they are not included as part of the standard Tekkotsu distribution. So if you do not have a Tekkotsu/tools/storyboard/ directory, you will have to download the version you need from the Tekkotsu web site's
Downloads page; see the "Optional Add-Ons" section.

Download the appropriate file for your OS, and unzip/uncompress/untar it. The resulting top-level directory will be named something like Tekkotsu_Storyboard_1.0. Move/rename this directory to Tekkotsu/tools/storyboard. Installation is now complete.


Examining a Storyboard

To begin using the tool, study the layout of the main window:


Several sample traces are included with the Storyboard Tool. In this section you will load a sample layout and model, and then use them to display a previously stored execution trace.

Loading An Example File

  1. Start the Storyboard Tool by running Tekkotsu/tools/storyboard/Storyboard. Note: you should run the tool from your home directory; do not cd to the Tekkotsu/tools/storyboard directory because this will confuse Eclipse when it tries to update your user configuration file.

  2. Click on the "Open Layout" icon in the upper left corner of the Monitor pane, or select File > Open Layout from the pulldown menu. (Note: all icons provide tooltip strings, so if you don't know what an icon does, just put the mouse over it and wait for the tooltip to pop up.)


  3. In the Open File dialog box, go to the Tekkotsu/tools/storyboard/samples/ directory, and select the drama/ subdirectory. Then load the layout file named drama.tsl. Note: if you choose a file whose extension is not .tsl, the program will treat it as a text file and open a viewer tab so you can browse the text. Close the tab and try again.

  4. Normally, loading a layout file automatically loads the corresponding model file. But due to a bug, this doesn't work for the sample files provided here. So when you load the layout file, you will get a new file browser dialog box whose title is "Cannot find model source". To proceed, manually select the file Tekkotsu/tools/storyboard/samples/drama/drama.tsm. You should then see a state machine displayed in the Monitor pane.

  5. Most of the bottom half of the window is taken up by the Storyboard pane. To load an execution trace, click on the Load Execution Trace icon in the upper right corner of the Storyboard pane. You could also right click inside the Storyboard pane to pop up a menu of operations; select "Load Trace" from that menu.


  6. Switch from the "Properties" tab to the "Runtime View" tab in the upper right corner of the window:


  7. Try placing the mouse over the various nodes, transitions, and icons that appear in the Storyboard pane. You will see tooltip windows pop up to describe each item. Clicking on an item will provide a more detailed description in the Runtime View tab. Notice that when you click on a state node in the Storyboard pane, the corresponding node in the Monitor pane is highlighted. When you click on a vertical line denoting a state transition, the source and destination nodes and the transition arrow are all highlighted.

  8. The vertical red line in the Storyboard pane is the time index. You can click and drag on this index to move through the execution trace. As you do so, the highlighting changes, as does the information in the Runtime View tab.


Modifying a Layout

When you first load a layout and model, the top left pane displays the Monitor tab, which is used for monitoring events on the AIBO. The model is displayed, but the diagram is grayed out, except for any nodes or links that fall under the red time index line.

To modify a layout, click on the Layout tab. Now the diagram is ungrayed, and you can click on the nodes and links. (To select a link you must click on the head or tail, not the middle portion.) Information about the node or link is displayed in the Properties tab in the upper right. (If you have the Runtime View tab selected, click on Properties to switch to that tab.)

The other two tabs in the top left pane, Model Source and Layout Source, allow you to view but not modify the contents of the .tsm and .tsl files you loaded.

Changing the LogTestMachine Layout

  1. Load the layout file samples/LogTestMachine/LogTest-layout.tsl.

  2. If it doesn't load automatically, load the model file samples/LogTestMachine/LogTest-model.tsm.

  3. Load the execution trace samples/LogTestMachine/LogTest-trace.tse.

  4. Switch to the Layout tab in the left pane, and select the Properties tab in the right pane. The model should look like this:


  5. Click and drag the Message node so that it is lower than the Image node. Then click on the Redraw Storyboard Trace icon to see the change take effect.


  6. Click on the Waiting node in the Layout tab, and observe its properties in the Properties tab. To display and modify the node's color and shape more easily, turn off Show Categories mode in the Properties tab. This causes all the properties to be listed alphabetically, instead of grouped into categories like "Info" and "Misc".


  7. Change the shape of the Waiting node from Rectangle to Ellipse. Note that after selecting Ellipse, you will need to click outside the selection box before the shape change takes effect in the Layout pane. Change the color of the node to brown by clicking on the "Color RGB [...]" text, then clicking on the icon that appears on the right.

  8. Right click on the Webcam node, and select Delete from the pop-up menu. The node and all its transitions should disappear.

  9. Right click on an empty spot in the Layout pane, and from the pop-up menu, select Add Node. Then select Webcam. Move the node to its previous position, but a little lower, so it is aligned with the Message node, and below the Image node.

  10. Right click on an empty spot again, and from the pop-up menu, select Add Transition. Then select one of the transitions and add it. Repeat this process to add the other missing transition.

  11. Save your modified layout in a file called mylogtest.tsl. You do not need to resave the model; mylogtest.tsl will point to the location of the original model, LogTest-model.tsm.


Generating an Execution Trace

You can use the built in LogTestMachine demo to practice generating a new execution trace. This will work using either the AIBO or the Tekkotsu simulator. Once you have this trace, you can replay it by dragging the time index across the storyboard, or save it to disk for later examination. (Although the behavior name is LogTestMachine, the menu item in the ControllerGUI is "Logging Test", so that is the name you must use in the Storyboard Tool's dialog box.)

LogTestMachine Execution Trace

  1. Boot your AIBO and open a telnet connection to port 59000. Or start the AIBO simulator.

  2. Start the ControllerGUI. Un-stop the AIBO.

  3. Go to Root Control > Status Reports > Event Logger, and turn on logging for buttonEGID.

  4. Start up a fresh Storyboard tool, and load the layout Tekkotsu/tools/storyboard/samples/LogTestMachine/LogTest-layout.tsl. If the model doesn't load automatically, then load LogTest-model.tsm as well.

  5. In the "Host" dialog box at the right side of the Monitor pane, type the hostname or IP address for the AIBO, or for your own computer if you're running the simulator.

  6. In the "Name" dialog box, type "Logging Test".

    Note: the name you type in the dialog box must exactly match the name in the ControllerGUI menu item that you use to start the behavior. If it doesn't match, the execution trace will still show event icons but there will be no state activations/deactivations.

  7. Click on the "New Trace" button. You will notice the time index in the Storyboard pane begin to advance.

  8. In the Controller GUI, un-stop the AIBO. Then go to Root Control > Mode Switch, then double click on "Logging Test" to start the behavior. You should see the state machine parent node "Logging Test" appear in orange in the storyboard. Note: if you activate the Logging Test behavior before beginning the new trace, the state activation event will not be sent to the Storyboard tool, so the node will never appear on the trace.

  9. Depress and hold one of the AIBO's buttons. You'll notice an event icon appear at the top of the storyboard, just below the timeline. Then release the button and you'll notice a different event icon appear. Place the mouse over the icon and a tooltip will pop up to describe the event.

  10. Type "!msg message" in the Controller GUI's Send Input window, and hit return. Notice the state transition that takes place in the storyboard. (You may have to scroll the storyboard to the right if the time index has advanced off-screen.)

  11. Type !msg image" and hit return, and the AIBO will take a picture with its camera, which then appears as a thumbnail on the timeline.

  12. The command "!msg webcam" is supposed to cause the Storyboard Tool to take a snapshot from an attached webcam and add it as a thumbnail on the timeline. This feature is not working yet, but you can still give the command and see the state transitions occur.

  13. In the ControllerGUI, click on "Logging Test" to stop the Behavior. Notice that the storyboard timeline continues to advance. Click on the Pause button in the Monitor pane to pause the storyboard logging.

  14. Click on the Save Execution Trace icon in the storyboard pane, (Note: this is not the same as the "Save" or "Save As" options in the File menu, which are for saving layouts. This save button appears on the storyboard pane itself, not the layout pane or pulldown menu.) Save your trace to the hard drive. Make sure the file has .tse as the extension.


Downloading a State Machine Model

Layout files describe how a state machine should be displayed. Model files describe the state machine itself: the node names and types, and the transition names, types, sources, and destinations. The model file is created by downloading the information from the AIBO: in other words, by asking a running state machine to describe itself.

Downloading the Howl/Blink State Machine Model

  1. Boot the AIBO, start the ControllerGUI, go to Mode Switch, and double click on DstBehavior to start the behavior running.

  2. Start a fresh Storyboard Tool.

  3. Click on the Create a New Layout icon.

  4. Type your AIBO's name or IP address in the Host dialog box, and DstBehavior in the Name dialog box.

  5. Click on Download Model. You should see the XML for the model scroll by in the log, and a dialog box will pop up asking if you would like to generate a layout for the model. Click "yes", and the model should appear. (If you instead click "No", you can manually add nodes and transitions by right-clicking in the Layout pane.)

  6. Switch to the Layout pane. Edit the layout until it looks roughly like this:

  7. Click on the Save Layout icon, or select "Save" from the File pulldown menu. Call your file bark.tsl. The tool will automatically save the model as bark.tsm.

  8. Go to the ControllerGUI and deactivate DstBehavior. Then go to Root Control > Status Reports > Event Logger and turn on logging for buttonEGID. Make sure the dog is un-stopped so it can receive button press events.

  9. Click on "New Trace" in the Monitor pane, and activate DstBehavior again. You are now generating a storyboard using the model and layout you created. If in response to a short bark, you push the AIBO's head button, you will cause the red {bark}--EventTrans->{wait} transition to fire, which will result in a vertical red line in the storyboard instead of the usual black line. See the third instance of the bark node below, and note the button event icons that appear just above the transition.

  10. When you've completed the trace, stop the behavior in ControllerGUI, and hit the Pause button in the Monitor pane. Then click on the "Save Execution Trace" icon in the Storyboard pane and save your trace to the hard drive.


Logging Messages and Images

You can log messages to the storyboard under program control, like the Message node did in the LogTestMachine example. You can also send AIBO camera images to the storyboard, or tell the Storyboard Tool to snap a picture with a webcam (presumably pointed in the general direction of the AIBO) and log that to the storyboard. The following is a simplified version of LogTestMachine that illustrates these features.

#ifndef INCLUDED_DstBehavior_h_
#define INCLUDED_DstBehavior_h_
 
#include "Behaviors/StateNode.h"
#include "Behaviors/Transitions/NullTrans.h"
#include "Behaviors/Transitions/TextMsgTrans.h"
#include "Behaviors/Controls/EventLogger.h"
#include "Events/EventRouter.h"
#include "Shared/ProjectInterface.h"
#include "Sound/SoundManager.h"
#include "Vision/JPEGGenerator.h"

class Groucho : public StateNode {
public:
  Groucho() : StateNode("Groucho") {}
  virtual void DoStart() {
    StateNode::DoStart();
    std::string const msg = "Hooray for Captain Spaulding!";
    EventLogger::logMessage(msg,this);
  }
};

class Harpo : public StateNode {
public:
  Harpo() : StateNode("Harpo") {}
  virtual void DoStart() {
    StateNode::DoStart();
    EventLogger::logImage(*ProjectInterface::defColorJPEGGenerator,ProjectInterface::fullLayer,0,this);
    sndman->PlayFile("camera.wav");
  }
};

class Chico : public StateNode {
public:
  Chico() : StateNode("Chico") {}
  virtual void DoStart() {
    StateNode::DoStart();
    EventLogger::logWebcam(this);
  }
};

class DstBehavior : public StateNode {
private:
  StateNode *startnode;

public:
  DstBehavior() : StateNode("DstBehavior"), startnode(NULL) {}

  virtual void setup() {
    StateNode::setup();
    StateNode *brothers = new StateNode("brothers");
    Groucho *gmarx = new Groucho;;
    Harpo *hmarx = new Harpo;
    Chico *cmarx = new Chico;
    addNode(brothers); addNode(gmarx); addNode(hmarx); addNode(cmarx);

    brothers->addTransition(new TextMsgTrans(gmarx,"groucho"));
    gmarx->addTransition(new NullTrans(brothers));

    brothers->addTransition(new TextMsgTrans(hmarx,"harpo"));
    hmarx->addTransition(new NullTrans(brothers));

    brothers->addTransition(new TextMsgTrans(cmarx,"chico"));
    cmarx->addTransition(new NullTrans(brothers));

    startnode = brothers;
  }

  virtual void DoStart() {
    StateNode::DoStart();
    startnode->DoStart();
  }

protected:  // Dummy functions to satisfy the compiler
  DstBehavior(const DstBehavior&);
  DstBehavior& operator=(const DstBehavior&);

};

#endif

This behavior uses a TextMsgTrans to look for messages from the ControllerGUI and transition to the appropriate state node. If you run this behavior and do "!msg groucho" it will log a message to the storyboard, while "!msg harpo" will log a camera image.


Known Bugs

Version 1.0 of the Storyboard Tool is a bit fragile. A variety of things can corrupt the program state, leaving it unable to load saved execution traces or generate new ones. When that happens, simply kill the program and start it again.

Prev: Defining new types
Up: Contents
Next: ---


Last modified: Thu Feb 14 01:29:01 EST 2008