Notes on Mullet & Sano, 1995

Kevin Mullet and Darrell Sano. Designing Visual Interfaces: Communication Oriented Techniques. Prentice Hall, 1995.

These are notes on the Mullet & Sano book from the perspective of someone designing a personal universal controller. The information that is extracted is relevant for that purpose, and doesn't necessarily reflect the book as a whole. For example, the sections of imagery and style were not included, because the controller is unlikely to generate icons or choose a style in which to present an interface. These will be chosen by the appliance and the device respectively.

The book is divided into eight chapters, half of which seem useful for the controller research.

Chapter 2: Elegance and Simplicity

This chapter is about simplifying interfaces so that complexity is avoided and the user is not distracted or confused by parts of the interface. As they write, "elegance cannot be easily summarized by rules of thumb. It depends heavily on taste." Of the four chapters, this one is the least relevant. The techniques described do seem interesting however, and it would be interesting to see if we could implement them in some way.

Chapter 3: Scale, Contrast, and Proportion

This chapter covers visual variables, such as those described in its title. Certain variables are better than others for describing certain things. The use of these variables effectively will be important for the controller. The techniques in this section are unfortunately somewhat difficult to implement. The knowledge of variables may be more useful for integrating in some other way (i.e. not rule-based).

Chapter 4: Organization and Visual Structure

This chapter and the next are the most relevant to our purposes. This chapter emphasizes the need for well thought-out organization in any interface. Grouping and hierarchy are important components of establishing organization, which underscores the necessity for these to be well-represented in the specification language.

Chapter 5: Module and Program

Module and Program are design terms that refer to aspects of a design that can be used to build consistency across a family of designs for a related cause. The main application of this methodology will be in describing the interface conventions of the controlling devices. The gridding technique will be useful for the controller in general however. Several techniques in this chapter can be applied to creating good grids and these should be implemented in the controller.

The beauty of this book is that each chapter is divided into three sections: principles, common errors, and techniques. The techniques sections are divided into several conceptual sections, all of which end with a set of design rules that we can attempt to implement in computer form. The rules aren't specified for a computer, but many do not appear to require human creativity to follow. I believe the next step after reading this book will be to implement a simple layout engine that takes as input a set of widgets and groupings and creates screens based on the concepts in the techniques sections of the book.

The notes for each chapter focus on the techniques section, although knowledge from other sections has also been extracted where appropriate.

The important points of this section are 1) complexity must be avoided and 2) avoiding complexity is often an iterative process in which the value of each design element is evaluated. If the element is found to be unnecessary, then it must be removed. This is the first of three techniques that are described in this chapter:

Reducing a Design to its Essence

The purpose of this technique is to reduce the design to its essential elements. The reinforces the message by not distracting the user from what is important. Trial and error seems to be only way to effectively reduce a design however, which could make it difficult to use this in our automatic layout method. The best we can hope for is to create the minimal number of elements to begin with... Some kind of evaluation technique would be necessary to do more. This is made explicit when reduction is seen as a three-step process:

1. "Determine the essential qualities that should be conveyed by the design"
2. "Critically examine each element in the design and ask yourself why it is needed, how it relates to the essence of the design, and how the design would suffer without it."
3. "Try to remeove the element from the design anyway. What happens?"

Clearly, it is not clear how the second steps could be implemented, although the third might be possible if a design could be evaluated for functional or aesthetic value.

Regularizing the Elements of the Design

A predictable rythym can make the design easier to comprehend. The regular columns in the NeXTStep browser is an excellent example. The interface has five columns, which are easy to read because they are all the same width. A figure that shows the same figure with the columns constrained to their maximum width is nearly impossible to read. The controller must create designs with this kind of regularity, which fortunately will probably be a by-product of gridding, discussed later.

There are four strategies that can be used to achieve regularity in designs.

• "Use regular geometric forms, simplified contours, and muted colors wherever possible."
• "If multiple similar forms are required, make them identical, if possible, in size, shape, color, texture, lineweight, orientation, alignment, or spacing.
• "Limit variation in typography to a few sizes from one or two families"
• "To reap the benefits of regularity, make sure critical elements intended to stand out in the display are not regularized.

The first two points should be addressed by the interface conventions of the controlling device, although they may need to be described to our layout software. The third point may be addressed by interface conventions, but should be taken into account in any case. The last item must be handled by the interface generator, although I guess that interface conventions may take advantage here as well (e.g. default button borders).

Combining Elements for Maximum Leverage

What is leverage? This was initially a point of confusion for me. Leverage means using parts of the design for multiple purposes, thus making the design visually efficient. The example given is the title bar in most GUI windows, which use the bar to display a label for the window and window manager controls. This kind of leverage can't be obtained by the PUC, because it exists at the level of the controlling device. Leverage at higher levels is more difficult because it requires task knowledge that we might not have. The steps for finding places to leverage the elements in the design are:

1. "Review the functional role played by each element in the design. (This information should be a natural product of the reduction phase.)"
2. "Look for situations where multiple elements are filling (or partially filling) the same role.
3. "Question whether an element's role could be filled as well by an adjacent component, possibly after minor modification."
4. "Combine redundant elements into a single, simpler unit or replace the lot with a common higher-level idiom from the target environment designed to address the situation"

These points are about as difficult to implement as those for reduction. The fourth point is reminiscent of the process that we may need to use to substitute a fancier component for a set of simpler components, except that it's backwards. Ideally, we'll be trying the fancy component first before resorting to constructing our own.

This section describes how the visual relationships of a design can be altered to produce an effect that does not overly dominate the design in an undesired way. This discussion is based on an idea of visual variables (size, value, hue, orientation, texture, shape, and position) and different styles of perception (associative, selective, ordered, and quantitative). See the book (pp. 54-56) for further explanation of these variables and styles. Hopefully most of these issues will be handled by the built-in widgets of the controlling device.

An odd tidbit from the section is somewhat relevant. The golden ratio has been shown to the be the most pleasing proportion. As an aside, the ratio is used internally by Kim's system to judge which objects are vertically vs. horizontally oriented.

Four techniques are described in this chapter, with the caveat that "mastery of these techniques requires extensive practice." It may be interesting to explore a neural net to describe these techniques, if it in fact they cannot be represented formally as rules.

The Squint Test

This is a less a technique than a trick. Simply squint at a design. The essence of the trick is to view the design out-of-focus and with reduced light. Elements that are visible when squinting will be visible at a glance by the users.

It may be possible to implement this technique is some way, but it is not clear that other methods of judging symmetry and balance would not work better. Furthermore, this is an evaluative technique. It is not clear how such techniques will fit into the system.

Establishing Perceptual Layers

Similar elements in a design can be grouped into visual layers for easy association by users.

1. "Group each item of information into a small number (7+/-2) of categories according to its origin or intended use. A group must be established for any group of items that will need to be processed independently. Each item must be assigned to at least one group."
2. "Determine the rank or importance of the various groups and organize them into an even smaller number (e.g., 3-5) of echelons based on this ranking."
3. "Use appropriate perceptual variables to establish the layering effect. Size and value can establish clear perceptual hierarchies, while hue is most effective for non-hierarchical grouping."
4. "Maximize the perceptual difference between groups while minimizing the difference within groups."
5. "Use the squint test to ensure that elements in the same layer group together as a unit."

Although our layouts could probably benefit from some layering, the cost of implementing a system that could do it seems high.

Sharpening Visual Distinctions

Make the distinctions between different visual layers very clear. This is similar to the technique above.

• "Identify the rankings that need to be established across the groups of information (see analysis undering Layering, above)."
• "Determine the range of variation (e.g., minimum and maximum values or sizes, numbers of colors, etc) available and use as much of it as possible in the resulting code."
• "Use logarithmic rather than linear scaling across groups."
• "Use the squint test to ensure that [the different] echelons can be easily recognized."

Hopefully most of this work will be handled by the interface conventions of the controlling device.

Integrating Figure and Ground

The figure is the primary formal element and the ground is the visual context within which the figure appears. The figure and the ground must be proportionally sized for the design to visually integrate correctly. This design rule must be implemented by the automatic layout tool.

1. "Determine the overall size of the figure/ground combination."
2. "Equalize the visual weight of figure and ground.
3. "Provide enough space around the margins of the figure to eliminate unwarranted visual tension."
4. "Position the figure correctly within the ground."

Since the above steps are vague, we will need to find some heuristic numbers to use in our implementations. These may need to be supplied in a definition for the conventions of the device.

The structure of the design must lead the user to systematically discover the elements of the interface. There are four principles of structure: grouping, hierarchy, relationship, and balance. The techniques described are the ones that will be the most important for the PUC to implement correctly.

Using Symmetry to Ensure Balance

Symmetry is one of the easiest ways to get good looking interfaces and is very effective for simple layouts. Balance comes for free in symmetric designs.

1. "Identify the axes along which symmetry will be established. In visual interface design, this decision usually boils down to whether horizontal and/or vertical symmetry."
2. "Symmetry about the vertical axis is more prevalent in human perception and is generally more useful in visual displays."
3. "Center the information on the axis of symmetry by carefully balancing the amount of information on each side of the axis. The information need not form a literal mirror image provided the mass and extent are equalized."
4. "Make sure the axis of symmetry is itself centered within the overall display context (i.e., the window or icon in question)."
5. "Use the squint test to verify the results."

It will be interesting to see how we measure issues of "mass" and "extent."

Using Alignment to Establish Visual Relationships

Alignment is important for displays to be coherent... Elements that are not aligned can be distracting and confusing.

• "Identify the major boundaries in the existing layout, and look for ways to enhance them by moving additional elements into alignment with them."
• "Look for elements and margins - both internal and external - that almost, but not quite, align with one another and bring them into alignment by altering the size or postiion of one or both elements."
• "Look for free-standing elements and make sure they are aligned with something else in the display - either a major margin or some other element to which they are related."
• "If an element cannot be related to anything else in the display, try to relate it to the proportions of the display itself by positioning the element to correspond to a regular division of the space."

Optical Adjustment for Human Vision

It is sometimes necessary to make changes to the design to account for pecularities in human visual perception. Circular elements should extend slightly beyond margins set by linear elements. Spacing should be determined by area in addition to distance, as in kerning for fonts.

• "Determine the true point of alignment, dimension of extent, or unit of spacing required. Translate this into the "normal" margin that would be occupied by a rectangular element in the same position."
• "Extend elements beyond the margin according to the sharpness of their adjacent angle. The greater the acuteness of the angle, the farther it will need to extend beyond the "normal" margin."
• "Use a "close-up" version of the squint test encompassing only the elements in question to verify alignment with the intended margin or visual equivalence of the relevant intervals."

We will need functions in order to make some of these adjustments... Particular for extending elements outside of the margin based upon the acuteness of their angle or the curvature of their shape. This information will probably need to be obtained through studies with designers.

Shaping the Display with Negative Space

Negative space, or white space, is important for differentiating different elements and groups within a design. Good use of white space will direct the viewer's eye to the regions of interest.

1. "Review the organization of the information into a prioritized set of chunks of manageable size. Note: the groups identified for techniques described previously serve this purpose as well."
2. "Ensure spatial separation of independent units of information by adding extra white space between chunks. Extra space is needed even if explicit boundary delimiters are used!"
3. "Determine which elements - and these often include individual elements used as labels for larger, less important chunks - require additional visual emphasis."
4. "Increase the white space surrounding critical elements by moving them into the margins, by moving other elements away from them."
5. "Always remember that white space is not wasted space! Its role is to direct the viewer's attention to adjacent regions containing critical information."

Hopefully we can make these steps generative instead of refining. These rules are also supposed to be difficult to apply...

This chapter discusses mechanisms that bring regularity and structure to families of designs. While the module and program of our interfaces will be mostly specified by the controlling device, it will be necessary to specify the module and program, i.e. the interface conventions, of each controlling device to maintain consistency.

THe chapter focuses mostly on grid-based design. It will probably be necessary to specify a grid for each controlling device.

Reinforcing Structure through Repetition

Repetition is already used in most interfaces simply by following user interface conventions. The PUC may want to create certain conventions of its own however, because it will be creating a number of interfaces that the user will interact with, and these interfaces are likely to more have more in common than arbitrary applications on the controlling device.

The following steps may need to be carried out for each controlling device.

1. "Start with rough sketches of the series of layouts to be produced."
2. "Look for common margins or functional units that must be clearly perceived across displays. Individual displays should be adjusted to ensure consistent positioning of major structural elements."
3. "Look for widely-spaced elements that should be visually related but which cannot be positioned next to one another."
4. "Look for paths the user's eye needs to follow through the display. Repeated structural elements can serve as landmarks and guides that help users with the nagivation task as they become familiar with the program."
5. "Use standard locations and a consistent presentation style for rules, text, or images introduced to visually reinforce the naturally repeating elements of the design."

Probably should also consider doing this on the current designs, just to see what we find. The results could be useful for the specification.

Establishing Modular Units

Modular units are the created by the spacing in a grid. The following steps illustrate how to specify a grid. Most of these steps will have to be performed for each controlling device.

1. "Determine the vertical unit. In GUI design this means defining the standard control height and the spacing between controls. You will need to do this once for each GUI toolkit at each standard size you intend to support."
2. "The vertical unit should allow any two controls to be placed next to one another, and should provide for proper spacing of multi-line controls, multi-control groups, and separations between groups. If labels are to be placed above rather than beside controls, the vertical unit must provide for correct positioning of labels as well."
3. "Determine the horizontal unit. The unit should be large enough to accommodate most of the one-word labels in the design space and should be at least three times as wide as the vertical unit."
4. "The optimum horizontal unit should provide 5-7 divisions of the typical display width and should be easy to factor evenly into partical or multiple units."

The specification of each controlling device will need to at least include partial values that can be used to calculate the vertical and horizontal units. The conventions of the device may influence how these units are chosen.

Creating Grid-Based Layout Programs

They say in the book, "Effective grid-based design requires both careful planning and a willingness to adjust the content intself when appropriate." Both comments are relevant and we must focus on the second one in particular. Requiring multiple labels in the specification will help with this, as will adding the ability to scale images, etc. Fortunately, "applying the canonical grid is largely mechanical."

1. "Determine any size restrictions on the area to be laid out."
2. "Determine the basic vertical and horizontal modules. The vertical module is determined by the widget set, while the horizontal module is determined by the number of controls (and sometimes by the lengths of their labels). These parameters (row height, column width, and number of columns) define precisely the canonical grid that will be used for this layout problem."
3. "Develop a rough layout sketch that approximates the sizes, positions, and orientations of the relevant control elements."
4. "Use the canonical grid to adjust the sizes and positions of elements across rows. Short elements are extended to begin and end on grid boundaries, while long elements are allowed to span multiple grid units or are shortened to fit within the standard unit. In this way, the grid is merely being used to help establish consistent alignment relationships of the type described in the previous chapter. Symmetry-enhancing adjustments to control ordering and arrangement can also be introduced anywhere in the display."
5. "For dynamic layouts, identifying the minimum size that can be accommodated by the layout is usually a better solution than trying to recompute the layout for arbitrarily small display sizes."

The trickiest part of this is deciding on the number of columns to put into the grid. Do you use a six-column format if your longest component is a string of six radio buttons? What about three columns with two rows of radio buttons? What metric do we use to decide this? Probably a trade-off between the length of the label and the space in the layout.

I think that the best way to see how these things work will be to try them out. Won Chul Kim did some things to improve the balance and symmetry of his generated interfaces. I will read up a little on what he did and then experiment with it myself.