Thursday, February 24, 2022 4:00pm
“Compositional generalization in minds and machines”
Brenden M. Lake, NYU
Abstract: People learn in fast and flexible ways that elude the best artificial neural networks. Once a person learns how to “dax,” they can effortlessly understand how to “dax twice” or “dax vigorously” thanks to their compositional skills. In this talk, we examine how people and machines generalize compositionally in language-like instruction learning tasks. Artificial neural networks have long been criticized for lacking systematic compositionality (Fodor & Pylyshyn, 1988; Marcus, 1998), but new architectures have been tackling increasingly ambitious language tasks. In light of these developments, we reevaluate these classic criticisms and find that artificial neural nets still struggle when systematic compositionality is required. We then show how people succeed in similar few-shot learning tasks and find they utilize three inductive biases that can be incorporated into models. Finally, we show how neural nets can acquire compositional skills and human-like inductive biases through meta learning.
Video of the day
2022: Course Format.
This is a project course.
One goal is for projects to interoperate and persist beyond the end of the course.
A great outcome would be a set of web pages telling others how to do the
We will use class time for presentations by you and the instructors,
discussions, work, and group meetings with instructors.
We will let projects drive the presentations.
There is no fixed schedule or syllabus.
Let our interests shape the course.
There will be some assignments intermixed with working on the projects.
The TA says:
The extended deadline for homework 1 is Wednesday, February 2nd. Please have slides about your work, a demo, or something else you can show over zoom prepared. I'll also ask you to submit a version of whatever you show just for my reference. This will be submitted over Gradescope (information to come). In the meantime, if you have any questions, please submit them here. I encourage you to make them public if you are comfortable with that, as someone else may have the same question!
This is meant to be a fun assignment, so try not to stress out over it too much!
Assignment 2: Due Feb 23.
Get a computer to learn something. Do something cool and fun. Examples of
software and demos on the web:
I googled "demo face recognition" and selected a few. Many more ...
OpenVino: Face detection
OpenVino: Face recognition
Google AIY Vision
Defeat face recognition
Maybe only Dragon learns
Neural net learning demos
A list of possible projects
Software available to you for free, that I highly recommend.
Matlab and its neural networks toolbox
The TA provides a way to do Assignment2 in python This is a good option if you are less familiar with machine learning, or are looking for a more structured assignment. The assignment covers learning the kinematics of a two-link arm and we have provided code to get you started. As always, you are free to work on an original project too! Send me an email or come by my office hours if you have any questions!
Video: Passcode: P4*?DiWh
Introduction to the course.
What are some big questions about humanoids?
Should robots be humanoid?
Is that the most useful form for a robot that stamps out widgets? Or harvest wheat? Or picks apples? Or takes care of your grandmother? What about giant spiders?
The superhuman sensing argument
How should we program robots? The intellectual pendulum, and the exponential improvement of computers.
Inverse Physics for perception?
Model-based optimal control for action?
Black Box Learning?
Jan 24: Video: Passcode: iW5V@knm
Continue above discussion.
Let's look at a specific problem: reaching out and touching someone (kinematics).
Jan 26: Video (Passcode: ^upNYf1.)
Discuss possible projects.
Discuss and present Assignment 1 in class for those who are ready.
- Get a graphical conversational agent
to show emotion and personality. Can we find existing software that allows
an agent to look angry, and gesture and talk angrily? Look happy, and
gesture and talk happily? Sadly? Disgutedly? Upset? Annoyed? ...
- Develop facial behaviors for a graphical humanoid. Face and torso. Upper body (add arm and hand gestures). Full body.
Develop an "AI" player for a physics puzzle game like Humans: Fall Flat
or any game that implements physics.
Use a robot kit to do something.
For example, we have some balancing robot kits (small segways).
Do something cool with 1 or more desktop arms.
Help Mrinal get robots to pick up monkeys and solve physics puzzle boxes
Do something cool with 2 arms using the Baxter robot.
Video of the day
Jan 31: Video (Passcode: ?qttrAE4)
Reach out and touch someone:
Inverse Kinematics: how could robots and people do it.
Feb 2: (Student presentations, so no video)
Assignment 1 presentations and project discussions.
Feb 7: Video (Passcode: =41kWcQA)
Finish up Assignment 1 presentations, Discuss projects, start next topic.
Feb 9: Video(Passcode: 6DVFS@zT)
Learning, representations, and
Very nice tutorial on neural networks
A basic tutorial for vectors and matrices,
Redundant inverse kinematics:
Unconstrained optimization: Using Matlab's fminsearch and fminunc.
Unconstrained optimization with soft constraints: Using Matlab's fminsearch and fminunc, with
Constrained optimization: Using Matlab's fmincon.
Feb 14: Classification
Computer Scientists Prove Why Bigger Neural Networks Do Better
Feb 16: Video (Passcode: S6=0rsaX)
Debate which way AI research should go
Feb 21: Video (Passcode: VWJ1=5QT)
Mrinal's HW2 comment
Finish up reinforcement learning
Feb 23: Video (Passcode: DRwwr7==)
on making humanoids more human-like.
Feb 28: Present Assignments
March 2: Present Assignments
March 7: Spring Break
March 9: Spring Break
March 14: No class
March 16: Project presentations
March 21: Project presentations
March 23: Video (Passcode: Lw*5jCk9)
Where am I: Probabilistic reasoning: Bayesian reasoning, Kalman filter, SLAM (where am I?)
March 28: no class, work on projects
March 30: Mrinal tutorial on training function approximators, Legged locomotion
April 4: Thinking about water: handling deformable and granular materials,
liquids, and gas.
April 11: Conversational agents, Personality and Emotion
April 18: What could go wrong?
April 25 and 27: Final presentations, and draft writeup (web page) due.
May 16 (May 11 for graduating students): Final project writeup (web page) due.
Robot Companions: Humanoids: Benefits and Risks
eldercare-bots, nanny-bots, teacher-bots, sex-bots, www.robotcompanion.ai
How can robots help us now?
Rise of robots hastened by coronavirus,
CORONAVIRUS OUTBREAK: BEST TEST TO ANALYZE THE FUTURISTIC CAPABILITIES OF TECHNOLOGY
- Connecting people.
Belgian video-calling robots to keep elderly connected during coronavirus
- Care for older adults and people with disabilities living alone.
- Home logistics, grocery delivery, ...
- Automate lab tests
- Clean equipment for re-use.
- Clean and sterilize hospital rooms, other contaminated areas.
Coronavirus: Robots use light beams to zap hospital viruses,
These robots are fighting the coronavirus in China by disinfecting hospitals, taking temperatures, and preparing meals
- Automate simple medical operations. Measure people's temperature during daily life.
"The machine consists of a robotic arm on wheels that can perform ultrasounds, take mouth swabs and listen to sounds made by a patient's organs, usually done with a stethoscope.",
Thailand hospitals use 'ninja robots' to fight coronavirus
- Fully automate some types of hospitals, quarantine areas.
What America can learn from China's use of robots and telemedicine to combat the coronavirus,
Coronavirus: Hospital ward staffed entirely by robots opens in China
Coronavirus care at one hospital got totally taken over by robots
- Robots And Drones Are Now Used To Fight COVID-19
IEEE Spectrum: Coronavirus Pandemic: A Call to Action for the Robotics Community
- Drug discovery and testing
- Public education, quarantine and other restriction enforcement, public surveillance
- News filtering
- If Robots Steal So Many Jobs, Why Aren't They Saving Us Now?
- Robotic supply chains for a post-pandemic world
- The most ridiculous ways companies are trying to profit from the coronavirus outbreak
- 'Naturally antibacterial': Sex doll companies trying to cash in on coronavirus
Manipulation: What do we do with our hands?
Dynamics: Understanding physics.
Simulation: Modeling and predicting what will happen.
Planning: Deciding what should happen.
Design and Building: Creating capabilities.
Sensing: Knowing what is happening.
Where am I: Sensor processing and fusion,
Control: Reacting to what actually happens.
Learning: Getting better with experience.
Recognition: object, face, and gesture recognition
Communication: speech recognition, dialog,
Skills: reinforcement learning. Legs.
Personality and Emotion.
Personal agents: Siri, Alexa, Google Home.
Books, movies, TV shows, and games:
What inspires us and shapes our thinking about robots?
Artificial people and playmates. Entertainment robots.
Ethics and policy: What are the implifications of robotics?
What should we do about them?
Jobs, weapons, social and behavioral, ...
See course format (above) and the deadlines in the schedule (above).
We will work out the project topics together.
You can work in groups or alone.
The "deliverables" include
a web page along the lines of an instructables web page
explaining how others could do your project and improve on your results.
You will also present your project, and ideally the presentation should
be made public as part of your web page. There will be intermediate
deliverables including draft web pages and practice presentations.