Creating Multimodal Smartphone Automation by Demonstration
SUGILITE is a new programming-by-demonstration (PBD) system that enables users to create automation on smartphones. SUGILITE uses Android's accessibility API to support automating arbitrary tasks in any Android app (or even across multiple apps). When the user gives verbal commands that SUGILITE does not know how to execute, the user can demonstrate by directly manipulating the regular apps' user interface. By leveraging the verbal instructions, the demonstrated procedures, and the apps' UI hierarchy structures, SUGILITE can automatically generalize the script from the recorded actions, so SUGILITE learns how to perform tasks with different variations and parameters from a single demonstration. Extensive error handling and context checking support forking the script when new situations are encountered, and provide robustness if the apps change their user interface. Our lab study suggests that users with little or no programming knowledge can successfully automate smartphone tasks using SUGILITE.
Our follow-up system EPIDOSITE extends SUGILITE to support users to leverage smartphones as hubs for smart home and IoT automation, and to create automation for smart home and IoT devices by demonstrating the desired behaviors through directly manipulating the corresponding smartphone apps. EPIDOSITE also supports using the smartphone app usage context and external web services as triggers and data for automations, enabling the creation of highly context-aware smart home and IoT applications.
Toby Jia-Jun Li, Amos Azaria and Brad A. Myers. SUGILITE: Creating Multimodal Smartphone Automation by Demonstration, Proceedings CHI'2017: Human Factors in Computing Systems, Denver, CO, May 06-11, 2017. Best Paper Honorable Mention Award.
Toby Jia-Jun Li, Yuanchun Li, Fanglin Chen and Brad A. Myers. Programming IoT Devices by Demonstration Using Mobile Apps, End-User Development. IS-EUD 2017. Lecture Notes in Computer Science, vol 10303., Eindhoven, The Netherlands, June 13-15, 2017. Best Paper Award.