Calypso Module 11: Speech and Hearing

Learning Goal: In this module you will learn more about how to make characters speak, and how to make them respond to speech, including your own. After completing this module you will be able to control Cozmo with voice commands.

Speaking

You already know that you can use the "say" tile to make Cozmo speak. You can also use this tile when you program the cubes or the charger. Although the cubes and charger do not have speakers, their speech is displayed in a speech bubble in the world map, just as Cozmo's speech is. A future release of Calypso will include synthesized voices for the cubes and charger that play through your laptop's speaker.

When you make a new "say" tile, a window pops up where you type what the character should say. The text that a character speaks should fit on a single line. If you type multiple lines in the "say" text box, one line will be picked at random each time the "say" action fires. This is an easy way to generate variability in the character's behavior, which makes it more interesting.

Suppose Cozmo is waiting for a cube to come into view. We might decide that every 10 to 20 seconds he should remind the human that he's waiting for a cube. Rather than say the same thing every time, we can give him a collection of similar phrases to choose from:

Please show me a cube.
Can you show me a cube?
I'm waiting to see a cube.
I need to see a cube!

You can edit the text of a "say" tile by putting the pencil on the tile and then pressing the Y button on the game controller or hitting "Y" on the keyboard to bring back the text box. Here is what the text box looks like:

Hearing Other Characters

When characters speak, all the other characters hear what they say. You can listen for particular phrases by writing a "WHEN hear" rule. In the text box, type the phrases you want to listen for, one per line. If any of those phrases match what is said, the rule can run.

A phrase must match exactly, except that spacing and capitalization are ignored. So "hello there" will match "HELLO There" or "hello there", but it will not match something with different punctuation, such as "hello there!", or a phrase with extra words such as "well hello there". Later in this module we'll look at techniques for accommodating optional words.

Here's an example of using the "hear" tile. Let's program cube1 to say "meow" when we tap it:

Character:

Now we want Cozmo to "chase the cat" by moving to the cube when it meows. In order to remember the cube across the many rule cycles it will take to chase it, we'll mark it in purple when we hear the meow, and then pursue the purple thing:

Character:

You can listen for a particular character's speech by mentioning the character in the rule. If you don't mention one, the rule can match any character, with one exception: a character cannot hear its own speech unless the rule includes the "me" tile.

Hearing Human Speech

Calypso includes built-in speech recognition using the Google speech API. In order for this to work, your computer must have a microphone. There is no micrphone built in to Cozmo. Many laptops have microphones built in, but if you're using a machine without one, simply purchase an inexpensive USB microphone. These are available from many vendors, including Amazon.

In addition to having a microphone, your computer must have a working Internet connection so it can communicate with Google's speech recognition service.

When your speech is recognized by Calypso, the text appears as an orange speech bubble at the top of the world map, like this:

If you want a rule to match only human speech, not speech from any other character, include the "face" tile in your "WHEN hear" rule. There is currently no way to tell different human speakers apart, so you cannot specify a name with the face.

When matching human speech, capitalization, spacing, and punctuation are all ignored. The system also performs certain spelling corrections, such as changing "cosmo" to "cozmo", and "cube to" to "cube2".

Controlling Cozmo With Your Voice

You can easily write rules to control Cozmo or the cubes with your voice. It's a good idea to begin these voice commands with "cozmo" or "cube1" so it's clear who you're speaking to.

Here are some examples of common voice commands:

Character:

Matching Complex Patterns (Advanced Topic)

For more complex matching, such as specifying synonyms, we can use a special notation called regular expression syntax. For example, to match any of "cozmo turn", "cozmo spin", or "cozmo rotate", we would write:

cozmo (turn|spin|rotate)

Optional words can be provided for by specifying the empty string as an alternative. For example, to include an optional "please" in the command "cozmo drop it" we would write:

cozmo (please|) drop it

The above pattern will match either "cozmo drop it" or "cozmo please drop it".

It is also possible to write a "wildcard" pattern that will match any sequence of characters. The wildcard pattern is "(.*)". This can be useful if we want to look for certain keywords in the input while ignoring anything that comes before or after. For example: to match any string that ends with the words "pick it up" we could write:

(.*) pick it up

This will match "Cozmo please pick it up" or "Now you can pick it up" or anything else that ends with those three words.

If we also want to ignore words at the end of the input, so that the only requirement is that "pick it up" appears somewhere in the input, we would write:

(.*) pick it up (.*)

Note that the parentheses are required when writing regular expression patterns.

Review and Discovery

Do the Module 11 Review and Discovery activity to review what you've learned.

Next Module

In the next module you'll learn about measuring time in Calypso.

Back to Calypso Curriculum overview.