CMU 15-869
Graphics and Imaging Architectures
Kayvon Fatahalian
Tues/Thurs 3:00 - 4:20pm (Fall 2011)
GHC 4101

Visual computing tasks such as 3D graphics and image processing are increasingly important to the capabilities and overall user experience delivered by computer systems ranging from high-end workstations to sensor-rich smart phones. The aim of this reading and project-based course is to examine key ideas, trends, and challenges associated with the design of architectures and systems responsible for efficiently executing these workloads.

This course begins with an in-depth study of the real-time graphics pipeline architecture and the efficiency of its modern GPU implementations. Key topics include the design of GPU processing and communication resources, graphics pipeline components and their scheduling on heterogeneous, parallel hardware, and how current abstractions balance conflicting needs for both efficiency and programmability. The second part of the course will address system design challenges in a broader array of emerging visual computing topics including: image processing architectures for mobile computing, programmable camera platforms, alternative graphics pipelines, and GPU-accelerated interfaces for application domains beyond graphics.

This course is intended for systems/graphics students interested in architecting future graphics or image processing platforms and for students interested in gaining experience designing applications or programming frameworks for emerging heterogeneous, parallel systems.

Sept 13:
Readings: (due for class Sept 15)
  • D. Blythe, Rise of the Graphics Processor. Proceedings of the IEEE, 2008
  • T. Akinene-Moller and E. Haines, Real-Time Rendering, Chapter 2 (handed out in class)
Sept 15:
(system inputs, outputs, entities, and operations... a.k.a., real-time graphics from a systems perspective)
Readings: (due for class Sept 20)
Sept 20:
(characteristics of the graphics pipeline workload, Molnar's "sorting" taxonomy for mapping the pipeline to parallel execution resources: trade-offs between parallelism, communication, and locality)
Readings: (due for class Sept 22)
Sept 22:
(clipping, tessellation, workload scheduling challenges introduced by tessellation)
Readings: (due for class Sept 27)
For the curious: (suggested additional readings on parallel tessellation)
Sept 27:
(fixed-function implementation, precision issues, work efficient algorithms vs. parallel algorithms, hierarchical and early Z, Z-buffer compression)
Readings: (due for class Sept 29)
For the curious:
Sept 29:
(anti-aliasing using the mip-map, fixed-function filtering, texture prefetching and caching policies)
Readings: (due for class Oct 4)
Supplemental material:
At this point in the course we have covered many of the basic algorithms implemented by graphics pipelines before the introduction of application-programmable shading stages. We have also discussed key factors in the graphics pipeline's implementation. The following papers describe graphics systems of historical significance that may be of interest to the class. Given course lectures so far, you should be able to digest most of the material in these papers.
For the curious:
Oct 4:
(implicit vs. explicit SIMD execution, large-scale multi-threading, thread scheduling, modern GPU case studies: NVIDIA Fermi and AMD Cypress)
Review reading: (combines topics up to, but not including, today's GPU core lecture; due for class Oct 6)
Supplemental material on GPU processing cores:
Oct 6:
(accessing data, the role of caches, bandwidth-limited applications)
Readings: (due for class Oct 11)
Supplemental material:
Oct 11:
Oct 13:
Guest Speaker:
Henry Moreton, NVIDIA
Oct 18:
(Part 1: an extended discussion of how to implement SIMT execution. Part 2: the rendering equation and why expressing shading/lighting functions requires programmability)
Homework Assignment 1 [Solution] (due for class Oct 20)
Supplemental material on SIMT/SIMD (also see material from Oct 4th lecture):
Oct 20
(key primitives, level of abstraction decisions [mimic hardware or align with domain?], Renderman Shading Language, modern GPU shading languages)
Supplemental material on shading languages:
Oct 25:
(history of GPGPU leading to CUDA (and then OpenCL), its good and not-so-good ideas, looking forward to CPU/GPU integration, GRAMPS)
Supplemental material:
Oct 27:
Guest Speaker:
David Blythe, Chief Graphics Software Architect, Intel
Nov 1:
(deferred shading costs/benefits in terms of coherence, operation count, and bandwidth; anti-aliasing in deferred rendering systems, tile-based deferred shading)
Supplemental material:
Nov 3:
(design constraints, differences between the Reyes pipeline and the real-time OpenGL/Direct3D graphics pipeline)
Motion blur demos:
Required reading: (due for class Nov 3)
Supplemental material:
Nov 8:
(modern packet-based approaches, ray reordering, ray-tracing vs. rasterization)
Required reading: (due for class Nov 10)
Supplemental material on modern ray packet tracing:
Supplemental material on global ray reordering for increased locality:
More good ray tracing reads:
High-quality, open-source ray tracing implementations:
Nov 10:
(light-field theory, how image-based rendering has found its legs in rendering massive data: e.g., Google Street View, Microsoft Photosynth)
Suggested readings:
Nov 15:
(sensor basics, noise sources, the early stages of the image processing pipeline)
Suggested readings:
  • The Stanford CS178 course notes are an incredible source of detail about digital cameras.
  • The demos shown in class (and more) are available on the Stanford CS178 pages.
  • The brave may want to check out the description of the image processing pipeline in Sections 12.4.6 through 12.4.8 of the TI OMAP35x Application Processor Technical Reference Manual (now getting dated). The OMAP4 version is in Section 8.3 here.
Homework Assignment 2: Related paper links are below (due for class Nov 22)
Nov 17:
(later stages of the image processing pipeline up through JPG compression, preshot basics: auto-focus, auto-exposure, hardware implementations: TI OMAP, NVIDIA Tegra)
Suggested readings:
  • (Please see suggested readings from the previous lecture.)
Nov 22:
(computational challenges of acquiring light fields; applications of light-field photography: synthetic aperture, post-shot refocusing)
Suggested readings:
Nov 29:
(case study of how the Microsoft Kinect system works. (Fixed-function processing and new algorithms to become practical on XBox 360)
Suggested reading:
Dec 1:
(trends in camera phones, Stanford's Frankencamera project, challenges of 'always-on' cameras)
Required reading: (due for class Dec 1)
Suggested readings:
  • Willow Garage is now supporting the FCam project. See the project page.
Dec 6:
Suggested readings:
  • I. Sutherland, Technology and Courage. SunLabs Perspectives Series, 1996 (This essay is based on a famous talk Sutherland gave at CMU in 1982.)
Dec 8:
No class: project work day
Assignments / Grading

Students will be expected to complete and analyze required course readings as well as perform a semester long research project of their choosing (groups of 1 or 2). One to two short assignments will be given during the semester (involving literature presentation and architecture design/thought experiments).

See the project details page for expectations and deadlines for projects.

Reading summaries should be submitted on the reviews web site. (In the collections list, look for the collection called CMU 15-869: Graphics and Imaging Architectures -- Fall 2011):


Knowledge of real-time 3D rendering as presented in an introductory graphics class (e.g., 15-462) is strongly recommended. Background in computer architecture (at the level of 15-213) or parallel computing is also recommended. If you are unsure about your preparedness for the material in this course, just ask!

Other Courses With Similar Topics