The point of this assignment is to:
We would like you to implement a component of a personal agent. We expect you to use publicly availble software to do it. Trying to write your own component is fine (but ambitious). Recycling something you have done before is fine. Components include conversational agents (chatbots), something that recognizes speech, something that generates speech, something that sees (recognizing faces is a good component here), something that gestures or moves, or some hardware component (talk to the profs about trying to do hardware). Below we give you pointers to components.
You can work in groups or alone. Create a web page describing what you did and email the URL to the professors (one web page per group). Be sure to list the names of all the members of your group. You choose the format of your writeup. The writeup is more important than the code or what performance is achieved. What did you do? How and why did it work? What didn't work and why?
We encourage you to create and post a YouTube video showing a demo. Be sure to put "CMU 16-264" in the text describing the video, so we can search for class videos.
--------------------------------------------------------------------- Get started building your own conversational agent! --------------------------------------------------------------------- Highly recommended: Get Python!: https://www.python.org/ The Google Assistant SDK makes google's speech recognition and conversational APIs available on other platforms. Overview and walkthroughs: https://developers.google.com/assistant/sdk/overview Official SDK Github repository: https://github.com/googlesamples/assistant-sdk-python See if you can get the Python library demo up and running, and then try the GRPC flavor (more low-level and hackable). "Hey Google" is boring. Snowboy lets you train your own custom hotwords. Snowboy offline hotword detection: https://snowboy.kitt.ai/ Snowboy Github repository: https://github.com/kitt-ai/snowboy Can you train a custom hotword and use it to do something? (Flashing an LED or printing "Hello World" is a good start.) Google and AIY have a convenient board for adding a microphone and speaker to a Raspberry Pi. (What's better is they have lots of example code, regardless of if you use their hardware.) AIY's microphone/speaker Voice Hat: https://aiyprojects.withgoogle.com/voice#project-overview Voice Hat Github repository: https://github.com/google/aiyprojects-raspbian Especially if you're using a Raspberry Pi, just have a look and see if there's anything relevant to your interests. Mallow is based on this code to combine Snowboy and the Google Assistant GRPC SDK... plus some hacking around. Combining Snowboy hotword detection with Google Assistant: https://github.com/warchildmd/google-assistant-hotword-raspi If you can get this running, you're in great shape to really start customizing your project. Extra challenge A: Get your assistant to recognize a few specific phrases or questions and respond accordingly. These ought to be questions that the assistant can't answer out of the box, like "Who is Chris's favorite robot" Extra challenge B: Google wants you to use it's Assistant voice. Find a clever way to get around that. Modifying the voice in pitch, cadence, etc. is good too! Extra challenge C: What else is out there besides Google? Definitely Alexa, maybe some others. See what you can do with them!
--------------------------------------------------------------------- Do something with faces --------------------------------------------------------------------- FACE RECOGNITION https://cmusatyalab.github.io/openface/ OpenFace OpenFace is a Python and Torch implementation of face recognition with deep neural networks and is based on the CVPR 2015 paper FaceNet: A Unified Embedding for Face Recognition and Clustering by Florian Schroff, Dmitry Kalenichenko, and James Philbin at Google. Torch allows the network to be executed on a CPU or with CUDA. code: https://github.com/cmusatyalab/openface http://openbiometrics.org/ OpenBR OpenBR is supported on Windows, Mac OS X, and Debian Linux. https://github.com/ageitgey/face_recognition ageitgey - Face Recognition The world's simplest facial recognition api for Python and the command line Python 3.3+ or Python 2.7 macOS or Linux (Windows not officially supported, but might work) https://realpython.com/blog/python/face-recognition-with-python/ OpenCV Face Recognition with Python, in Under 25 Lines of Code Collections http://blog.rapidapi.com/2017/11/10/top-10-facial-recognition-apis-of-2017/ Top 10 Facial Recognition APIs of 2017 https://www.quora.com/What-are-the-best-face-detection-APIs-Could-I-use-it-to-determine-whether-there-were-no-faces-in-the-image Quora: What are the best face detection APIs? Could I use it to determine whether there were no faces in the image? https://www.kairos.com/blog/face-recognition-kairos-vs-microsoft-vs-google-vs-amazon-vs-opencv Face Recognition: Kairos vs Microsoft vs Google vs Amazon vs OpenCV ************************************************************************* FACE RECOGNITION ON RASPBERRY PIs https://www.hackster.io/gr1m/raspberry-pi-facial-recognition-16e34e https://learn.adafruit.com/raspberry-pi-face-recognition-treasure-box https://trello.com/c/niAl9sps/74-raspberry-pi-facial-recognition-door-greeting-system http://www.instructables.com/id/Who-Is-at-the-Coffee-Machine-Facial-Recognition-Us/ http://www.instructables.com/id/RasPi-OpenCV-Face-Tracking/ https://www.youtube.com/watch?v=Fggavxx-Kds https://electronicsforu.com/electronics-projects/face-recognition-using-raspberry-pi https://www.linkedin.com/pulse/face-recognition-using-opencv-python-raspberry-pi-abhishek-mulgund https://devpost.com/software/face-recognition-raspberry-pi-tqp95a https://www.udemy.com/facial-recognition-using-raspberry-pi-and-opencv/ http://www.knight-of-pi.org/opencv-primer-face-detection-with-the-raspberry-pi/ ************************************************************************* RECOGNIZING EMOTIONS USING VISION Intraface IntraFace - Human Sensing Laboratory www.humansensing.cs.cmu.edu/intraface/ http://www.paulvangent.com/2016/04/01/emotion-recognition-with-python-opencv-and-a-face-dataset/ https://nordicapis.com/20-emotion-recognition-apis-that-will-leave-you-impressed-and-concerned/ 20+ Emotion Recognition APIs That Will Leave You Impressed, and Concerned https://www.matec-conferences.org/articles/matecconf/pdf/2017/39/matecconf_cscc2017_02067.pdf ************************************************************************** RECOGNIZING EMOTIONS IN TEXT Synesketch: An Open Source Library for Sentence-Based Emotion ... ieeexplore.ieee.org/document/6589580/ https://github.com/collab-uniba/Emotion_and_Polarity_SO EmoTxt : A Toolkit for Emotion Recognition from Te x t ************************************************************************** FACE GENERATION http://ascl.cis.fiu.edu/hapfacs-open-source-softwareapi-download.html
Google "open source personal agents" http://lucida.ai/ https://github.com/huginn/huginn https://www.quora.com/What-are-some-open-source-programs-for-creating-an-Intelligent-Virtual-Agent https://www.predictiveanalyticstoday.com/top-intelligent-personal-assistants-automated-personal-assistants/ http://meta-guide.com/software-meta-guide/100-best-github-personal-assistant