Lecture notes from Scott Hudson, Guest Lecture for 05-830 on January 25, 1999 Constraints A general technique useful for describing and maintaining relationships several important UI tasks presentation geometry management automation of feedback aspects of presentation (this gate should turn red when its fan-out limit is exceeded) dialog management maintaining highlight & enable status application interface and connections between application data and interface objects multiple views support for constructing interactive objects by aggregation A constraint system consists of a set of variables - in our case these variables are typically embedded inside objects things like: size & position choice of color or icon visible/invisible enabled/disabled value of scrollbar etc. - in this treatment we will ignore this and assume that attrs just "float" but always remember that these are values embedded in interactor tree (or appl data structures) hence they affect display (and possibly how input is handled) a set of relationships that are to hold between the variables the system is responsible for making sure constraints hold under changes to the variables under changes to the constraints => a declarative approach say "what" is to happen leave "how" to the system if the system can do it, this is a lot less work on programmer Unfortunately, general case of constraint solution is unsolvable However restricted, but very useful restricted versions can be handled User interfaces typically use one of two restricted classes "one-way" constraints A more general class of "multi-way" constraints - don't have time to say much about these. (For us...) Relationships normally expressed by equations (expressing equality) A = B + C In the general case, if we change A, B, or C system is responsible for finding changing values of one or both of the other vars in order to maintain constraint - note we normally have multiple constraints that must hold, so job is a bit trickier than it might appear In first restricted case, we only allow the information to flow in one direction: - one way constraints A <- B + C under changes to B or C, system will find a value of A that satisfies constraints Details of restrictions: variables can only be the "target" of one constraint - one defining constraint variables which have a defining constraint may not be directly assigned no cycles in dependency graph - not everybody agrees on that part Brad's systems make use of cycles An algorithm to efficiently update one-way constraints Some terminology: will need to build a dependency graph indicates which variables are dependent on which others - note will often call variables "attributes" Node in the graph for each attribute Edge from A to B indicates that B is dependent on A A = B + C yields two edges B -> A C -> A Note this is data flow direction sometimes see this in other ("depends on" relation) direction When we make a change to a value (lets set aside changing the constraints for now) This may affect other values "down stream" in the dependency graph Use "grid-shaped" dependency graph for intuition <> if we change a value the set of values that might change fans out in a wedge Simple update algorithm: when I change value, recompute everybody that depends on me (if they change, they cause others to recompute, etc.) => data-driven algorithm Problem: /--->(B1)---\ /--->(B2)---\ ... / v / v / (A1)------------>(A2)------------>(A3) ... (An) \ ^ \ ^ \ \--->(C1)---/ \--->(C2)---/ ... When we change A1: eval B1, A2, and C1 B1 changes eval A2 C1 changes eval A2 Notice A2 gets several values before it settles down on final value In fact its worse than several A2 gets 3 values A3 gets 9 (3 for each different value of A3) An gets 3^n values!! (that can take a while) both breadth first and depth propagation are exponential in this case breadth first is *only* 2^n, but others are worse What do we do? Need to respect topological order of the graph Never compute the value of an attribute until *all* the attributes it depends on have final values Following works: Topologically sort the graph For each attribute A in topsort order ("lowest" to "highest") evaluate A topsort guarantees that things attribute depends on are all evaluated before the attribute "What you don't remember the topsort algorithm!?" basically just a depth first traversal But we are still doing too much work <> (again) only some of the attributes in the system could get new values but above approach reevaluates everything would like an incremental solution just do the things inside the wedge In fact, that may be more work than we need incoming values of some attributes might change, but same value results example: a = b || c b == true and c changes a still has same value <> Would like to update only "influenced" attributes the ones that change value, plus one more downstream (if a parameter changes we can't tell in advance if value stays the same or changes, so we have to evaluate one past point of actual change) not update all of "maybe_influenced" But that is still more than we need may not actually use all values would like to be able to selectively enable/disable expensive features consider: a = if do_expensive_op then expensive_op(x,y,z) else 0 If we go back to the wedge, we have an analogous situation use of a value ("demanding" that value) may require a other attributes again, fans out but may not require all attributes in wedge - don't need x,y,z or anything they depend on from above <> would like to just evaluate "needed" not "maybe_needed" In fact what we want is intersection of "influenced" and "needed" - actually a bit more complicated if you requested and modify different attributes over time, but that gives the intuition So, how do we do this? Add a bit of bookkeeping Put a boolean ("out_of_date"/"OOD") on each attribute - indicates that value *might* be out of date wrt its equation (will correspond essentially to the "maybe_influenced" set) And a boolean ("pending") on each edge - indicates that there is a changed value which has not been propagated across that edge (represents the "high water mark" of the "influenced" set) For each change of an attribute (lets assume a static dep graph for now) { do a recursive walk and mark each reachable attribute as OOD (if we reach an attribute that is already OOD stop) set pending mark on all outgoing edges } For each requested value if the value is OOD { set OOD false recursively request values it depends on if any incoming edges are marked pending { clear all incoming pending marks evaluate the equation to get a new value if the value has changed mark each outgoing edge as pending } } return the value Actually, we only request attributes that the equation needs example if-then-else never needs more than 2 of its values => intertwine evaluation and testing for pending This algorithm is "partially optimal" executes the minimal set of equations after any given change may do slightly more total bookkeeping work than optimal in rare cases - happens in mark OOD phase which is cheap - lower bound result says it is optimal in all cases where maybe_influence < 2^|influenced| Multi-way constraints Relaxing the restriction on information flow direction A = B + C can change any variables - more powerful/expressive - e.g. difficult to do some centering constraints in one-way - e.g. regain symmetry. system is responsible for finding other values - this isn't always possible (can be over constrained) note: this can never happen in one-way's - may be possible, but particular algorithm can't do it most troubling: - may be more than one way to do it (under constrained) arbitrary choices may be surprising i.e., constrain a point to be at the midpoint of a line segment can solve constraint after any change by making all points coincident => solves constraint, but not what you want - in general much less understandable behavior than one-way (but you have to give up power for understandability) Best solution to this so far: "constraint hierarchies" - odd name, better thought of as "constraint strengths" - don't have time to consider algorithms to update, but they exist and are efficient within certain limitations Opus A more general visual notation for layout Based on an underlying constraint-based system (Penguims) Supports a nice general notation for layout based on constraints Allows other aspects of appearance to be driven by constraints - provides another way to do application interface establish & automatically maintain relationship between ` appl data structures and displays four basic constructs in notation: frames constraints reference lines interactor objects Interactor objects are the only parts that appear in the final interface - reasonable subset of the Artkit interactor library is supported Each object is represented by a bounding box (+ "other properties") each edge of bounding box represents one coordinate - top, bottom, left, right pretty much all geometry is covered this way (only things like polygons and splines need more) As before, layout task comes down to size and position of these rectangles each edge can either have a constraint that controls it or be "free" visual convention: constrained edges are drawn double - makes it easy to see where you can and can't add constraints some objects have an intrinsic size => internal constraints - in this version of system, bottom left is always constrained - later we dynamically add second constraint once first established non-geometric aspects of object controlled by values or equations entered in property sheet for the object constraints are represented by arrows <> one-way constraints so directional always goes from horiz to horiz or vert to vert small box holds equation $ in equation represents value at source can put application objects/function into constraints => application interface => possibilities for semantic feedback also a special un-anchored form of constraint with no source Frames <> - don't appear in the final interface used as a grouping unit also the unit of composition (& reuse) make hierarchy by "plugging" one frame into another - composite frames vs. open frames Frames support "reference lines" analogy to reference lines used by draftsmen very thin line drawn to align other things to, later erased reference lines in Opus do the same don't appear in final interface, but used to align things reference lines come in several varieties "free" just floating, defined by a constraint like any other edge min, max, average do a computation over any number of incoming constraints proportional a fixed proportion of the frame they are within - provides effect of Cardelli's attachment points Examples of how to put this all together Fig. 8 fixed size, stick to top left Fig. 9 expand to fill all the available space Fig. 10 stretch proportionally Fig. 11 centered under Notice that neither "outside-in" (top-down) or "inside-out" (bottom-up) is sufficient for everything you want to do e.g. Fig. 12 needs both The path of least resistance is very important... what does OPUS not do (well)? doesn't handle number independent layout - must know/specify all objects in advance - can't talk about objects not created until run-time