Roger B. Dannenberg

I recently started playing with MicroPython running on an ESP32 Thing microcontroller from SparkFun. Online materials helped me get started, but I ran into trouble dealing with Wi-Fi and multiple files in my project. I will describe my development process in this blog, and next time show some fun applications of O2 and the new O2lite library for MicroPython (that I developed using the techniqued described here).

Passwords are a problem because you frequently reboot your microcontroller and have to provide a Wi-Fi name and password to get connected. Most online tutorials show how to connect by wiring your secret password into code, but there are two problems with that approach: (1) anyone that could pocket your microcontroller could probably read your password, and (worse) (2) if you share your code or your code is unprotected in any way, the password could be disclosed.

Multiple Files are a problem because writing files, at least to my ESP32 flash memory, is slow. With typical development in C++, you compile a (hopefully small) program and upload a monolithic program to the device. With MicroPython, you may have many source files that you need to upload individually, but only if the file has changed. So what tool or process will save time by uploading only what has been edited since the last upload?

The Goal is to have a single simple command that uploads any changed files that are needed onto my ESP32 flash memory, then runs the "main" program directly using ampy run, which is faster than writing to flash.

The Solution in brief is to use make to keep track of when files are uploaded to the ESP32 and upload them again when they have been edited. For a modicum of password security, I use a shell script to append Wi-Fi setup code to the program I want to run (via ampy). This keeps the password out of my source code and out of flash memory on the ESP32. Details of all this follow.

Keeping Passwords (Relatively) Safe

My approach with passwords is I will keep Wi-Fi network setup code in a place that is only readable by me from my laptop. I will use a script to append the Wi-Fi setup code to the front of the program I really want to run, constructing a temporary file that I run with ampy. The program I really want to run can be stored with project files and shared or published, while the small bit of Wi-Fi setup code and the temporary file that is actually run on the ESP32 is kept in a “secret” place on my laptop, avoiding even accidentally sharing or publishing my password. The password may exist in RAM on the ESP32, but it will be lost when the computer is powered off or reset.

You will need to create a shell script and run it, so if you do not have a personal bin directory create one:

mkdir ~/bin

You will also want to put your bin on your shell PATH, e.g. in my ~/.zshenv, I have something like

export PATH="$PATH:/Users/rbd/bin"

but shell initialization on Unix (Mac, Linux, whatever) is a mess with many shells, options and conditions, so you'll have to figure out how to get ~/bin on your path.

The script we want is called uprun (short for MicroPython run):

#!/bin/sh

BINPATH=/Users/rbd/bin
SERIAL=/dev/tty.usbserial-D306EBLS

make upload SERIAL=${SERIAL}

rm ${BINPATH}/secrets/main.py
touch ${BINPATH}/secrets/main.py
chmod og-r ${BINPATH}/secrets/main.py
cat ${BINPATH}/secrets/wifiprefix.py ${1:-main.py} >> ${BINPATH}/secrets/main.py
ampy --port ${SERIAL} run ${BINPATH}/secrets/main.py
  

Looking at this uprun script, you can see that it defines BINPATH and SERIAL, but you will want to change these according to your environment.

Next, you see that uprun invokes make upload to upload any changed files to the ESP32. We will see how to implement this in the next section. Now for the Wi-Fi setup...

The sequence of rm, touch and chmod commands simply create a protected empty file in ~/bin/secrets/main.py. By creating an empty file first, we can set the permissions to be read-only (by the owner) so that it will be more protected when we write our Wi-Fi password there.

Next, the cat command concatenates ~/bin/secrets/wifiprefix.py to “${1:-main.py}” and writes the result to our protected ~/bin/secrets/main.py. What is ${1:-main.py}? It means the argument that you pass to uprun, if any, and otherwise main.py. So this will combine the Wi-Fi setup with the program you want to run.

Finally, uprun uses ampy to run the constructed main program secrets/main.py. You should put the script in ~/bin/uprun and make it executable:

chmod +x ~/bin/uprun

The next thing you need is a file ~/bin/secrets/wifiprefix.py. This will be MicroPython commands to set up Wi-Fi. Here is what my ~/bin/secrets/wifiprefix.py looks like:

WIFI_NAME = "NETWORKID"
WIFI_PASSWORD = "PASSWORD"

import globals
import time
import network

sta_if = network.WLAN(network.STA_IF)
sta_if.active(True)
sta_if.connect(WIFI_NAME, WIFI_PASSWORD)
while not sta_if.isconnected():
    time.sleep(1.0)

globals.internal_ip_address = sta_if.ifconfig()[0]
print("IP Address", globals.internal_ip_address)

You will have to replace NETWORKID with your actual network name and PASSWORD with your actual network password. For safety, you should make this file read-protected from any other users:

chmod chmod og-r ~/bin/secrets/wifiprefix.py

Notice that this code stores globals.internal_ip_address for any other module to access. I need this for my code, but it requires module globals. You will see later than I have an empty (!) file named globals.py to implement this trivial module. But if you do not need that, you can just delete every line in ~/bin/secrets/wifiprefix.py with the word global.

Now we have a file that will be prepended to MicroPython programs you want to to run with uprun.

And with uprun, you have the ability to run any program with MicroPython after first automatically setting up a Wi-Fi connection, all without storing your Wi-Fi password in your source code under development. Of course, if you want to detach from your USB connection and run your ESP32 on batteries, you will have to put main.py into flash memory and your password will be readable by anyone with access to your ESP32, so it’s not perfect, but better than typing your Wi-Fi password into every program you create.

Working with Multiple Files

The next problem we will deal with is developing with multiple files. In my case, I started testing code for O2lite on my ESP32, but sometimes it was hard to remember what changed and needed to be uploaded to the ESP32 from my laptop where I store and edit the code. I could write a script to upload everything, but uploads to flash are slow, so I want to upload only what is necessary.

The basic tool for this kind of “process only when something is out of date” is make. Make usually relies on timestamps of derived files, but here the uploaded files are on the ESP32 and do not have accessible timestamps. As an alternative, I use files that record when the corresponding file was uploaded. If the timestamp file is up-to-date, then the ESP32 is up-to-date as well.

Here is a prototype of what the Makefile will look like:

upload: timestamps/mysrc.time timestamps/mysrc2.time

timestamps/mysrc.time : src/mysrc.py 
	ampy --port $(SERIAL) put src/mysrc.py src/mysrc.py 
	touch timestamps/mysrc.time 

timestamps/mysrc2.time : src/mysrc2.py 
	ampy --port $(SERIAL) put src/mysrc2.py src/mysrc2.py 
	touch timestamps/mysrc2.time 

When you run make upload, note that upload depends on two files in timestamps/. These, in turn, depend on actual Python source files. If the Python source files are older than the timestamps, nothing happens. But if they are newer (e.g. edited), then the source file is copied to the ESP32 with ampy and the timestamp file is updated to the current time using touch.

We could create the whole Makefile in this manner, but it is more compact if we expand filenames in rules using some features of make. Here is the result:

# Makefile - copy changed files needed by MicroPython to esp32
default: help

SHAREDFILES = o2lite.py o2lite_disc.py o2lite_handler.py \
              byte_swap.py ip_util.py

UPYFILES = $(SHAREDFILES) globals.py upydiscovery.py upyfns.py

# produce o2lite.time globals.time ...
TIMENAMES = $(addsuffix .time,$(basename $(UPYFILES)))

# produce timestamps/o2lite.time timestamps/globals.time ...
TIMESTAMPS = $(addprefix timestamps/,$(TIMENAMES))

timestamps/%.time : src/%.py
	ampy --port $(SERIAL) put $< $<
	touch $@

upload: $(TIMESTAMPS)


clean:
	rm -r timestamps/*


help:
	@echo "make upload SERIAL=/dev/tty.usbserial-D306EBLS -- upload files to ESP32"
	@echo "make clean -- remove timestamps to force uploading everything"

This Makefile defines SHAREDFILES, a list of source files common to MicroPython and Python3 (I am creating code that runs on both), and UPYFILES, a list of MicroPython-specific files. Together they are used to create the list TIMESTAMPS which is “built” by the upload target. Note that no file name upload is ever built, so it always forces make to “build” the TIMESTAMPS.

Summary

Now we know how to work with multiple files, editing them locally and then automatically copying them to the ESP32 when uprun is called. We also know how to automatically add code to the main program that intializes Wi-Fi, avoiding the need to store Wi-Fi passwords with the source code.

The next installment describes O2lite and introduces our first O2lite with MicroPython example: Connecting physical controls to a software synthesizer.

Complete Sources

• uprun
• wifiprefix.py