16-421: Vision Sensors, Spring 2008

General Information

Time : Tuesdays and Thursdays, 1:30 pm -- 2:50 pm

Location: Wean Hall 5312

Credits : 12

Pre-requisites : Linear algebra, Calculus, or Instructors Permission

Instructor

Srinivasa Narasimhan

http://www.cs.cmu.edu/~srinivas

Email: srinivas@cs.cmu.edu

Office: NSH 4117

Office Hours: By appointment

Overview

This course covers the fundamentals of vision cameras and other sensors - how they function, how they are built, and how to use them effectively. The course presents a journey through the fascinating five hundered year history of "camera-making" from the early 1500's "camera obscura" through the advent of film and lenses, to today's mirror-based and solid state devices (CCD, CMOS). The course includes a significant hands-on component where students learn how to use the sensors and understand, model and deal with the uncertainty (noise) in their measurements. While the first half of the course deals with conventional "single viewpoint" or "perspective" cameras, the second half of the course covers much more recent "multi-viewpoint" or "multi-perspective" cameras that includes a host of lenses and mirrors.

Lectures and Readings

Lecture 1: History of Cameras and Images

Lecture 2: Camera Obscura and View Camera [Readings]

Lecture 3: Optical Elements of Imaging

Lecture 4: Lab demonstration - Lens undistortion

Lecture 5: Camera Calibration: Geometric (Thanks to Martial Hebert)

Lecture 6: Camera Calibration: Radiometric

Lecture 7: High Dynamic Range Imaging [HDRShop] [Fiat Lux] (Thanks to Paul Debevec)

Lecture 8: Programmable Imaging I (Thanks to Shree Nayar and CAVE Lab)

Lecture 9: Programmable Imaging II (Thanks to Shree Nayar and CAVE Lab)

Lecture 10: Lab Demonstration (Beam Splitters, NSH 2204)

Lecture 11: Structured Light and Range Imaging

Lecture 12: Lab Demonstration (Minolta Range Scanner, Martial Hebert's Lab)

Lecture 13: Super-resolution using Jitter Camera and Hybrid Imaging (Thanks to Shree Nayar and CAVE Lab)

Lecture 14: Midterm Review

Lecture 15: Camera Arrays (Thanks to Stanford Graphics Lab)

Lecture 16: Catadioptric Cameras (Thanks to CAVE Lab)

Lecture 17: Stereo with Catadioptric Cameras with Planar Mirrors (Thanks to Joshua Gluckman)

Lecture 17: Polarization Imaging (Thanks to Yoav Schechner)

Lecture 18: Photo Tourism Google TechTalk (Thanks to Steve Seitz's group)

March 4: Student presentations of assignments

March 6: Midterm

April 15-May 1: Student presentations of papers

Light Field Photography with a Hand-Held Plenoptic Camera
Prakash: Lighting-Aware Motion Capture Using Photosensing Markers and Multiplexed Illumination
Rendering for an Interactive 360 Light Field Display
Compressive Imaging: A New Single Pixel Camera
Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing

Assignments

Correcting Lens Distortions: Display Gray-code patterns on a flat LCD screen and capture images with a camera. Then, find the mapping between corresponding pixels of the camera and LCD. A new distorted image is corrected by using the inverse mapping. (10 points) (Due March 4)

Geometric Calibration of a camera: Use the matlab calibration toolbox and a printed checker-board pattern to calibrate the camera intrinsic and extrinsic matrices. The re-projection errors must be very small. (10 points) (Due March 4)

Radiometric Calibration of a camera: Capture images of a color chart under uniform lighting with different exposures. Plot the response function of the camera. (10 points) (Due March 4)

The World in Eyes: Capture a high-res image of the cornea of the eye and compute the environment map and the retinal image. (10 points) (Due April 28)

Split-shot Camera: Build a split-shot camera with a planar mirror. Capture rectified stereo image pairs with the setup. Set the disparity of the farthest scene point to zero and translate the image to create a red-blue stereo pair. (10 points) (Due April 28)

List of Topics

PART 1: Perspective Sensors

Basic Principles of Still and Video Cameras
- CCD, Film, CMOS Sensors
- Electronics (A/D conversion, integration time, sampling, etc)
- Noise
- Device response
- Focus and Depth of Field
- Scheimpflug Photography
Basic Principles of Optical Elements: Filters and Lenses
- Filters (Polarizers, Neutral Density, Linear Interference)
- Fish Eye Lenses
- Single Lens Reflex
- Lens Distortions, Vignetting, Chromatic Abberations
- Autoexposure, Autofocus, Optical Stabilization
- Optical Transfer Function (OTF/MTF)
- Diffraction-limited Imaging
Camera Calibration
- Projection Fundamentals and Image Formation
- Geometric Calibration
- Radiometric Calibration
CCD Demosaicing
High Dynamic Range Imaging
- Sequential change in exposures
- Spatially varying exposures
- Real time exposure control over space and time
Other sensors
- Range Finders (structured light, time-of-flight)
- Gated Imaging

PART 2: Non-Perspective Sensors

Light Field
- Plenoptic function
- Light Field Parameterization
Single Viewpoint Catadioptric Cameras
- Omni-cameras
Multi-viewpoint systems
- Mathematical background on Caustics
- Refracting/Reflecting Caustics
- Multi perspective Images
Mosaicing
- Panoramic/Stereo Mosaicing
- Pushbroom cameras
Projectors and Displays
- Stereoscopic Displays
- 3D displays
- DMD for cameras, displays, projectors
- Projector-Camera Systems

Grading

Five Assignments 60%
Midterm 20%
Final Presentation 20%

[Acknowledgements] A significant part of this course is similar to the courses offered at Stanford (Pat Hanrahan, Marc Levoy, Ron Fediw), UC San Diego (Henrik Wann Jensen), Columbia (Shree Nayar, Peter Belhumeur, Ravi Ramamoorthi), UW Madison (Chuck Dyer), UWash (Steve Seitz), Utah (Pete Shirley), Rutgers (Kristin Dana), Cornell (Steve Marschner, Kavita Bala), Technion (Yoav Schechner), Princeton (Szymon Rusinkiewicz), MIT (Ted Adelson), Drexel (Ko Nishino), TU Berlin and Deutsch Telecom (Rahul Swaminathan) The instructor thanks the instructors of these courses for the materials (slides, content) used in this course. In addition, several photographs and illustrations are borrowed from internet sources. The instructor thanks them all.

[Permission to use/modify materials] The instructor gladly gives permission to use and modify any of the slides for academic and research purposes. Since a lot of the material is borrowed from other sources, please acknowledge the original sources too. Finally, since this is a continuously evolving course, all suggestions and corrections (major, minor) are welcome!