Exploring Tekkotsu Programming on Mobile Robots:

Sketch Primitives

Prev: VisualRoutinesStateNode
Up: Visual Routines
Next: Shape primitives

Contents: Types of sketches, SketchGUI tool, Constructors, Copying and assignment, Indexed access, Summary of operations, Exercises, Data structures

Types of Sketches

Several types of skeches are built in to the system. More types may be added by editing some source files as described at the end of this section. The built in types are:

Sketch<bool>  Binary pixels.
Sketch<uchar> Unsigned char. Useful for storing color-segmented images, since the color index easily fits within one byte.
Sketch<uint> Unsigned int. Used to represent array indices for subscripting sketches (except on the AIBO, where short ints are used). Also used to represent areas and Euclidean distances.
Sketch<usint> Unsigned short (2-byte) int. Used to represent array indices on the AIBO, for subscripting sketches. (This assumes there are fewer than 65,536 pixels. A full-resolution ERS-7 camera image has 208*160 = 33280 pixels.)
Sketch<float> Floating point values; for experimentation.

The easiest way to get started with sketches is to grab a segmented camera frame from the vision stream and turn it into a Sketch<uchar>. The sketchFromSeg() function does this for you. The NEW_SKETCH macro is used to declare new variables that point to sketches. We'll call our first sketch "camFrame".

NEW_SKETCH(camFrame, uchar, sketchFromSeg());
Now we will locate the orange pixels in camFrame. The result is a Sketch<bool>, where each pixel is true if the corresponding pixel in camFrame is orange:

NEW_SKETCH(orange_stuff, bool, visops::colormask(camFrame,"orange"));
The "neighbor sum" function counts how many of a pixel's neighbors are non-zero. We can use either four-way or eight-way connectivity; the latter includes the diagonal neighbors.

NEW_SKETCH(neighbors, uchar, visops::neighborSum(orange_stuff, visops::EightWayConnect));
We can use the neighor sum to do edge detection. True pixels with fewer than 8 true neighbors are edge pixels:

NEW_SKETCH(edges, bool, orange_stuff & neighbors < 8);
The example above illustrates how the standard C++ arithmetic comparison and logical operators have been overloaded to accept sketch arguments. The operations are performed in parallel on all pixels. First, each element of the sketch neighbors is compared against the constant 8. The result is a Sketch<bool>. Next, each element of orange_stuff is logically and-ed with that result, producing another Sketch<bool> that becomes the value of edges.

Another way to look at images is to extract just the intensity information, which is the Y-channel in the AIBO's native YUV encoding. The function sketchFromRawY() returns a Sketch<uchar> where the values from 0 to 255 are intensity values (brightness).

NEW_SKETCH(camY, uchar, sketchFromRawY());
Let's turn this whole sequence into a running program and try it out:

#include "Behaviors/StateMachine.h"
using namespace DualCoding;

$nodeclass DstBehavior : VisualRoutinesStateNode : doStart {
  NEW_SKETCH(camFrame, uchar, sketchFromSeg());
  NEW_SKETCH(orange_stuff, bool, visops::colormask(camFrame,"orange"));
  NEW_SKETCH(neighbors, uchar, visops::neighborSum(orange_stuff, visops::EightWayConnect));
  NEW_SKETCH(edges, bool, orange_stuff & neighbors < 8);
  NEW_SKETCH(camY, uchar, sketchFromRawY());
}

This behavior does not produce any printed output. But we can use the SketchGUI tool to examine the results of the computation.


The SketchGUI tool The SketchGUI tool is used to examine the sketches and shapes your program constructs. There are three versions, for camera, local, and world space. Right now we are only concerned with camera space.

Using SketchGUI

  1. Compile and run DstBehavior, either on the simulator or on the AIBO. If you're using an AIBO, make sure the camera is pointing at something orange, or change the example code to use another color. If you're using the simulator, you can place this test image in your images directory:

  2. Click on the "C" button in the ControllerGUI window to launch the camera space SketchGUI.

  3. In the SketchGUI menu, click on "camFrame". The results should look like this:

  4. Now click on "edges", and the results should look like this:

  5. Click on the other sketches to see what they look like.

  6. By holding down the Control key and clicking on a sketch, you can select or deselect it without affecting the other menu items. Use this technique to superimpose the sketches camY and edges.

  7. Right-clicking on a sketch summons a pop-up menu. Try inverting camFrame and then superimposing edges on top of it.

From this example you can discern the basic structure of a "sketch space":

Explore more:
  1. Try switching the order of the arguments to the & operator in the definition of edges. What effects do you observe in the listing displayed by the SketchGUI?

  2. Write a program to find all the pixels that have both a blue neighbor and a green neighbor.


Constructors

We often talk about "sketches" as if they were arrays of pixels. The reality is more complex. The pixels are stored in a valarray, which is in turn stored in a SketchData object, which is managed by a SketchPool, which resides in a SketchSpace. Objects of type Sketch are actually only a few bytes long. They are smart pointers that provide a convenient way of referencing SketchData objects, hiding the bookkeeping details from the user. We will use the lowercase word "sketch" to refer to the pixel array and associated data structures, and the capitalized word Sketch to refer to the smart pointers that point to these structures.

NEW_SKETCH is the easiest way to construct a sketch. NEW_SKETCH_N can be used when you don't want the sketch to be visible. But it is sometimes convenient to call one of the Sketch constructor functions directly. There are two constructor functions plus one copy constructor. The first Sketch constructor requires a SketchSpace and a sketch name as inputs. We will use camSkS, the camera sketch space:

Sketch<bool> foo(camSkS, "foo");   // first Sketch constructor
foo = false;                       // clear all the pixels
for ( int i=50; i<90; i++ )
  foo(i,i) = true;                 // draw a diagonal line
foo->V();                          // make foo visible to the SketchGUI

If you run the above example and look at the result with the SketchGUI, you will see that the sketch named "foo" has no parent; it is a root node in the set of derivation trees. Also, since we did not specify a color for the sketch, it was assigned the default color 1, meaning the first color in the color list given in the .col file the robot is using.

The expression foo = false makes use of the overloaded assignment operator; it clears all the pixels in sketch foo. This is necessary because the sketch constructor simply allocates space for the sketch; it does not bother initializing the pixels, since this may be unnecessary.

The second Sketch constructor is used when we want to create a child of an existing sketch. The first argument is the new sketch name, and the second argument is the parent sketch. (We don't have to specify the sketch space because the parent's space must be used.)

Sketch<uchar> bar("bar", foo);     // second Sketch constructor
bar = (Sketch<uchar>)foo + 2;      // element-wise addition 
bar->V();

While foo is a Sketch<bool>, bar is a Sketch<uchar>, so the "color" it inherits from foo is ignored (but preserved), and its pixel values are treated as indices into the default color map. Whereas foo's values were 0's and 1's, bar's values will be 2's and 3's.

NEW_SKETCH doesn't use either of the above constructor functions. It uses the copy constructor to create a new smart pointer to reference an existing sketch that has already been created by some expression. So when we write this:

NEW_SKETCH(orange_stuff, bool, visops::colormask(camFrame,"orange"));
it expands into this:
Sketch<bool> orange_stuff(visops::colormask(camFrame,"orange"));    // copy constructor
orange_stuff->V("orange_stuff");                                    // rename and make visible
The function visops::colormask creates a new sketch that is a child of camFrame (it uses the second sketch constructor function). This sketch is initially given the name "camFrame==scalar", and the Sketch object is returned as the value of colormask. The copy constructor is then used to create a local variable orange_stuff whose value is a copy of this Sketch object. On the next line, the call to the V() function changes the name of the sketch to "orange_stuff", and makes the sketch visible to the SketchGUI. Note: the name is stored in the SketchData object, not in the Sketch object that points to it, so a renaming operation effects all sketch objects pointing to that sketch data.

In summary: NEW_SKETCH does not make a new sketch, only a new Sketch.

As a smart pointer, the Sketch class overloads the arrow operator. Thus, when we want to call a sketch method, we use arrow rather than dot. Here are some example operations that can be performed on sketches:

camFrame->setViewable(true);

cout << edges->getColor() << endl;

int const width = orange_stuff->getWidth();

Copying is Shallow; Assignment is Deep

The Sketch copy constructor is a shallow copy, i.e., it copies only the smart pointer, not the pixels. But the assignment operator is deep: it alters the pixels the Sketch points to, not the Sketch itself. Example:

Sketch<uchar> A(camSkS, "A");    // new sketch named "A"
A = 1;                           // all A's pixels set to 1

Sketch<uchar> B(camSkS, "B");    // new sketch named "B"

Sketch<uchar> C(B);              // shallow copy: C points to the same sketch "B" as B does

C = A;                           // deep assignment: now all B's/C's pixels are 1,
                                 // but they are not the same pixels as A's pixels

B = A + 1;                       // now all B's/C's pixels are 2,
                                 // while A's pixels are still 1

Why is copying shallow. but assignment deep? Copying is commonly used when a function returns a Sketch value; a deep copy would be wasteful, and defeat the purpose of having sketches be smart pointers. Assignment is deep because some assignments only make sense as deep operations, e.g, A = 1. Since this must be deep, B = A+1 and C = A should also be deep, for consistency.

For shallow assignment, use the bind function. It changes the sketch data to which a Sketch object points, without altering any pixels. For deep copy, use visops::copy to copy the pixels.

Sketch<uchar> P(camSkS,"P");      // new sketch named "P"
P = 1;                            // all P's pixels set to 1

Sketch<uchar> Q(camSkS,"Q");      // new sketch named "Q"

NEW_SKETCH(R, uchar, Q);          // new sketch R points to same pixels as Q
Q = 2;                            // all Q's/R's pixels set to 2

R.bind(P);                        // make R point to same pixels as P

NEW_SKETCH(S, uchar, visops::copy(P));    // new sketch named "S" is copy of P's pixels

P = 3;                            // all P's pixels set to 3; S is unaffected

At the conclusion of the above sequence, P's and R's pixels are 3, Q's pixels are 2, and S's pixels are 1.

Note: for the std::vector class, both copying and assignment of vectors are deep. Sketches do not behave the same way as STL vectors.


Indexed Access

The simplest way to access individual elements of a sketch is to treat the sketch as a vector. The operator[] function has been overloaded to support this. Here is an example of creating a sketch with only the center pixel turned on:

Sketch<bool> mysketch(camSkS,"mysketch");
mysketch = 0;
int const midpoint = mysketch->getHeight()/2 * mysketch->getWidth() + mysketch->getWidth()/2;
mysketch[midpoint] = 1;

The other way to access sketch elements is via (x,y) coordinates. The operator() function has been overloaded to support this mode of access. Here is sample code that draws a salmon-colored triangle, after using a different method to initialize the sketch to zero:

NEW_SKETCH(tri, bool, visops::zeros(camSkS));
tri->setColor(rgb(252,110,15));
for ( int x=0; x<20; x++ )
  for ( int y = -x; y<=x; y++ )
    tri(x+50,y+50) = 1;

For reasons of speed, these operators do not do bounds checking. You can use the at() function to do bounds-checked array access, either by pixel number or by (x,y) coordinates:

tri->at(x+50,y+50) = 1;
To be added: SketchIndices and Region


Summary of Sketch Operations

Macros NEW_SKETCH(name, type, value)
NEW_SKETCH_N(name, type, value)
GET_SKETCH(name, type, space)
Constructors Sketch<T>(SketchSpace &sketchspace, string const &name)
Sketch<T>(string const &name, SketchRoot &parent)
Name Sketch->getName()
Sketch->setName(string name)
Color Sketch->getColor()
Sketch->setColor(rgb color)
Arithmetic ops:
     +, -, *, /
Sketch op Sketch
Sketch op constant
Logical comparison:
    <, <=, >, >=, ==, !=
Sketch op Sketch
Sketch op constant
Boolean ops: &, |, ^, !
Aggregate Sketch->max(), Sketch->findMax()
Sketch->min(), Sketch->findMin()
Sketch->minPlus(), Sketch->findMinPlus()
Sketch->mode(), Sketch->modePlus()
Sketch->sum()
Empty test Sketch->empty()
Assignment:
    =, +=, -=, *=, /=
Sketch op Sketch
Sketch op constant
Logical assignment:
     &=, |=, ^=
Sketch<bool> op Sketch<bool>
Binding Sketch.bind(other_sketch)
Indexing Sketch[i]
Sketch(x,y)
Sketch->at(i)
Sketch->at(x,y)
Visual routines See the summary of visops:: functions below.

Summary of visops:: Functions

Connected components labelcc(Sketch<bool>)
oldlabelcc(Sketch<bool>)
areacc(Sketch<bool>)
areacc(Sketch<uint>)
inArea(Sketch,minval)
Min/max max(Sketch,value), max(Sketch,Sketch)
min(Sketch,value), min(Sketch,Sketch)
Conditional assignment ifNot(Sketch,Sketch)
maskedAssign(Sketch<T>,Sketch<bool>,Sketch<T>)
maskedAssign(Sketch<T>,Sketch<bool>,T value)
Region filling fillexterior(Sketch<bool> boundary)
fillinterior(Sketch<bool> boundary)
seedfill(Sketch<bool> boundary, unsigned int index)
Edge and symmetry detection edge(Sketch)
horsym(Sketch), versym(Sketch)
skel(Sketch)
Distance bdist(Sketch,maxdist)
edist(Sketch)
Miscellaneous colormask(Sketch,color_index)
colormask(Sketch,"colorname")
copy(Sketch)
neighborSum(Sketch,connectivity)
fillin(Sketch,minNeighbors,maxNeighbors,iter)
zeros(Sketch), zeros(SketchSpace)


Exercises

  1. Remove speckle noise from a Sketch<bool> by generating a new sketch containing only those true pixels with more than one neighbor.

  2. Mark the region at a distance of 20 to 30 pixels from the edge of the largest region in the image. (Hint: use Euclidean distance operator, edist.)

  3. Find all regions with area greated than 50 pixels. Calculate the average area of these regions. Then generate a new sketch containing only those regions whose area is above average.

  4. Generate a Sketch<bool> containing the outlines (edges) of all yellow blobs greater than 50 pixels in area.

  5. Generate a Sketch<uchar> containing the outlines of all yellow and all blue blobs greater than 50 pixels in area. Each outline should be of the appropriate color (that's why we're using uchar instead of bool.)


The Sketch Data Structures

Textual explanation.

To extend the set of sketch types, modify the following files:

  • SketchTypes.h
  • SketchSpace.h (which functions)
  • SketchSpace.cc (which functions)
  • Sketch.h (which functions)
  • SketchData.h.(which functions)
Prev: VisualRoutinesStateNode
Up: Visual Routines
Next: Shape primitives


Dave Touretzky
Last modified: Wed Jan 26 01:14:20 EST 2011