Weblog

Pittsburgh, PA: 18:41 EST

Well, it's been a good six months since I've written in my weblog. At this point I have implemented an algorithm that infers structure from dependency information. This is useful for controlling devices that have modes. The inferrence algorithm is only the first part of multi-phase system. There are at least two other phases to consider, back-tracking and global aesthetic touch-up. I have done a little work on the third "touch-up" phase already, using James Fogarty's gadget system.

That bring me to my issues of today, which are as follows:

• Back-tracking involves some sort of negotiation architecture/command object setup. I need to find a general way to build this architecture, such that I can specify a set of decisions to made as the interface is constructed, rules for making a particular decision, and additional rules for picking which decisions to revisit when there are problems with the layout.
• The gadget system uses an optimization algorithm to solve a variety of problems. I am applying this system to aesthetic layout of dialog boxes. To do this, I need to develop a set of rules for making dialog box layouts aesthetic. My main goal is to make the algorithm do grid-based layout, and then maybe add some additional rules to sync layouts between "stacked panels." Stacked panels are those that always one on top of another, like in a TabControl. It seems reasonable that these panels could share some common design elements, and I would like to figure out what these are. Before I can get there though, I need to figure out some good rules for doing grid layout, which has become more difficult than expected.

I think I made each of those points a little too expansive, because I thought I had more than two points when I started, but I can't think of anything else now.

It occurs to me that one thing I need to do is go back to my notes from HCIC. I had some ideas about how to proceed then, that I've lost track of.

Pittsburgh, PA: 13:48 EDT

Design Studio is just beginning, but I'll try to give a short update before it does.

I have started working now on the opposite end of the implementation of the controller. I am coding the implementation of the concrete interaction objects (CIOs) and their associated functionality.

I have separated CIOs into two broad groups: state-linked CIOs and indepedent CIOs. This represents the distinction between widgets in the interface that change things on the controlled device, and widgets that control organizational mechanisms, such as widget panel is displayed or whether a dialog box is open. This set of widgets is nearly disjoint, although there is some crossover (buttons in particular).

An interesting piece of this discussion is that I have not separated these widgets into input vs. output categories, because in many cases state-linked widgets are used for both input and output. The input vs. output distinction is one that I have seen in previous work (need to find this again...).

Although independent CIOs are used for managing organization, linked CIOs may be used for organization as well. In addition, the type of organization that these widgets control may independent of the widget. e.g. a button could control which panel is visible, or whether a dialog box is shown. To implement this nicely, I made the organizational functionality pluggable, in order to allow any widget to control any organizational mechanism.

I am now working on a state-linked TextField and Button CIOs for use with our system now. With the addition of a network component, this should allow the current code base to control a device that exports our specification. e.g. the stereo that Maya is currently building.

More details on this system as I get further along.

Pittsburgh, PA: 14:11 EDT

Yet again, I'll spend a few paragraphs catching up on yesterday, and then I'll document what I've done today.

Yesterday I tried to divide my tasks up, just so I could be better organized and feel like I was making some progress. (If research is the only item on my to-do list, I can never check it off...) The list I came up with was this:

1. Find which nodes should be organized
2. Figure out how to organize those nodes
3. Develop the back-tracking system, for adding organization when widgets do not fit on the screen
4. Build the decision trees for choosing widgets for each state and command
5. Design and build a generic architecture for managing the widgets on the screen.

At the time, I thought that the first two tasks could be separated, but I have since changed my mind. My initial plan was to find which nodes should be organized, and then use a decision tree to determine which type of organization should be used. I have since scrapped this because the process of adding a screen within the structure is very different from adding a panel, and these elements are added for very different reasons. As far as I have determined, the dependency graph information will only suggest the addition of screen in one situation: the existence of a power button upon which everything else depends.

Panels and frames on the other hand, are used in about the same situation: whenever there is mutual exclusiveness within the dependency graph. Frames seem to be used when the exclusive dependency is boolean and the set of widgets is small (all can fit on the screen). Panels are used when dependency is more than boolean or when more space is needed for widgets.

Because of these restrictions on screens and panels, it would seem to make more sense if there was a initial check for the "power button" special case, and then further work was done to find situations in which panels or frames are needed.

Handing the special case is easy. Finding panel and frame locations is not quite as easy, but I think that I have found an algorithm that will work. To start, all of the states that other objects depend on will be placed in an ordered list by the number of objects that depend on them. A loop will iterate through this list, each time selecting the remaining state with the most items that depend on it. The algorithm will then find every occurence of that state within the interface tree. A new organization node will be created at the same level within the tree. Any groups in the same group with the state, that also completely depend on that state, will be placed in the new organization node. The type of the state will determine whether this organization node is given a "frame" or "panel" designation. If "panel" organization is used and the groups are mutually exclusive, then the groups will be left as children of the "panel" node. If they are not, a new organization node will be created as a child of the "panel" node, and all of the groups will be placed under this node.

It is also possible for there to be two mutually exclusive groups under a "frame" node. In this case, the two groups will be placed under their own "frame" nodes which will be children of the original node. The original node will be given no organization.

At some point, a rule will need to be added that limits the depth of these panel and frame trees... New screens will have to be created at some point to keep the UI legible.

At this point I would like to code my solution and try it out, but there is no test bed for me to do so. I have decided to take a detour for at least the rest of this week, and work in reverse order from the task list I showed above. I will design and code the basis of the system first, so that I can test the trickier parts of my system later.

Working in this order also has the benefit that it will create a system that will be usable by Maya for the control of their stereo, when they get that finished.

I will document the design of my UI architecture later tonight.

Pittsburgh, PA: 21:05 EDT

I'm noting that it's been quite awhile since I used this log. That's really too bad, since I often find that it helps me think more clearly and directly about what I am doing. Oh well.

Fortunately, I do work, even when I'm not writing about it in my weblog. I am a bit further along than I was when I last wrote here. The question before was how to link the dependency graph and the group tree. For the most part, I think that I have solved that problem now, although there are still some issues that I am hoping to resolve tonight.

Since last time, I have constructed a data structure that will be the final output of whatever structure algorithm that I come up with. I feel that this structure contains all of the information that is necessary to feed to the layout engine in order to make a UI. The data structure is basically a tree in which the abstract interaction objects (AIOs) are leaf nodes of the tree, and the non-leaf nodes are called organizational nodes. The organizational nodes can represent one of several types of organization that are typically found within user interfaces: screens, panels, frames, or nothing. There are more details to this structure, but I won't write them here because they're already in my notes.

Since the group tree is already the same structure as this final data structure, it seemed best to use the group tree as the initial starting point for constructing the interface tree. Dependency graph and type information can be used to determine where the tree needs to be changed, and then I think that decision trees can be used to determine what particular type of organization should be used.

This is essentially where I am now... Trying to figure out exactly how to use the dependency and type information to determine where and how to modify the group tree.

Pittsburgh, PA: 09:24 EDT

I've been distracted by my speaking requirement talk and the trip to New York the past few days, and so I haven't been able to spend much time working on this. It makes me realize that I need to write a little more often in my weblog because I no longer have the full context of what I was thinking about five days ago. Fortunately, I think that I've recovered most of what I was thinking about.

To start with, I am going to find all of the states that other objects depend on. Depending on what type of variable they are and the grouping of the objects that depend on them, I will decide whether they should define a structural break in the interface. Then I need to infer a tree of these breaks, and then implement them within the interface.

The last bit is a little vague, and I think that is where I am unsure how I was planning to implement this. Basically there are two issues that need to be addressed:

• How to be sure that the dependency graph will resolve to a nice structural tree. Or at least, how to be sure that whatever manipulation that we do to the graph will result in a tree.
• How to link the group tree, which may be completely disjoint, with whatever tree we create from the dependency graph.

A question immediately jumps into my head, which is whether these two parts can be combined into a single algorithm which uses the group tree and the dependency graph data together to build an interface tree, which describes the structure of the interface. The optimal branches would be generated initially, and then some form of backtracking would be done if the widgets specified for one screen did not all fit.

I am now going to work on this algorithm on paper... I will update the weblog when I get something.

Pittsburgh, PA: 23:15 EDT

Ugh... I'm realizing that representing the dependency graph with arbitrary dependency equations is hard. I *think* that its sufficient it's sufficient to maintain two pieces of information about the dependency graph:

• Top-level and statements, for determining structure.
• All variable dependencies, for constructing the mini-group trees.

The mini-group trees are beginning to worry me too. I am making an assumption that the dependencies on a particular variable will group nicely and not overlap. This is not necessarily the case...I'm not sure how to handle this situation either. Perhaps I ought to go back to the tree-node idea. Which means that I need to figure out the and vs. or thing.

Pittsburgh, PA: 10:14 EDT

Well, I thought that I would continue my earlier discussion last night, but instead it has taken me until this morning.

The biggest problem that I feel I have right now is getting bogged down in thinking too much about the problem without trying things. After a little bit of thinking, here is an outline of what I am going to try to get started on automatically designing interface structure.

• The first important issue is that dependency graph information is strictly more important than group information. This is because it is not generally a good idea to keep large groups of disabled controls visible on the screen, but it is reasonable to keep different groups of controls on the screen at the same time. This indicates that the basis of our interface structure should come from the dependency graph.
• Of course, the dependency graph is rarely sufficient to determine all structure, especially in situations where it is flat. The telephone is an excellent example of a flat dependency graph. About the only dependency of note is the whether the answering machine's power is on or off. In these cases, more information is needed to decide interface structure. I suggest a mechanism that I had considered earlier: Widgets are inserted into their structured pane until either all the widgets are used or no more widgets will fit. If the latter case prevails, then the interface layout tool backtracks. All widgets are removed from the pane and a new structural element is added (a tabbed component or a new dialog box, for example). The widgets are divided based on the group tree.
• An important aside: Groups must have labels! This comes from my dicussion in the last point, where some interface structure may be based upon the group tree. In these cases, we may want to display a tabbed interface with different groups on each pane. How do we label these tabbed panes? Group labels would help us do this.
• The only problem with this algorithm so far is that the dependency information and the group information are not related. I assume that somehow I will be able to figure out how to associate a set of groups with each element of the structure created from the dependency graph. Perhaps I can just create mini-group trees from the main group tree, within the context of the dependency graph structure. In other words, each piece of the graph structure would have its own group tree, which is identical to the main group tree except that none of the variables and commands that did not have the proper dependencies would not be included in the tree. If a branch had no variables or nodes, it would be removed. Any node of the tree would be removed if it had only one branch remaining. This would leave useful trees that could be used for creating further structural elements if necessary.
• One issue that I have a minor problem is the difference between and and or elements. I get a little confused when I try to think of how to use or elements in the organization process. They can certainly be used in creating the mini-group trees discussed in the above point, but I am not sure if I should use them when generating the dependency graph structure. Perhaps the problem here is because I haven't thought enough about the exact algorithm that I will use for finding structure in the dependency graph. I think the general idea was to take all of the variables that other variables depend on. Then we must look for hierarchy among these variables (do any depend on another). Variables at the same level within the hierarchies should be ordered by the number of variables that depend upon them. Now, having written all that, I'm not sure how we extract the structure from the hierarchies.

While I think about that, I'm going to write the portions of the parser that still aren't finished.

Pittsburgh, PA: 17:21 EDT

After thinking a little about the decision trees, I decided that developing a structure for the interface was important than picking which widget to use. I believe that structure is more closely related to the dependency graph than widget selection.

I think the basis of structure in a remote-control interface are the variables that other variables depend upon. My observations suggest that a relatively small percentage of the variables have other variables that depend on them. The dependent variables can be grouped by what value activates them, with an additional group for variables that are not dependent at all. I think that some structure can be deduced by examining these groups and comparing them with the group hierarchy.

More on this later tonight...

Pittsburgh, PA: 10:30 EDT

Today I am thinking trying to develop the decision trees that will help me pick widgets for each of the different value spaces. I also want to spend some time thinking about the dependency graph and the representation that I want to use for this project.

I'll start with the decision trees, because I think this will help give me an idea of which dependency graph representation I should use. My approach for developing the decision trees will be this:

1. Start with the decision trees described in the DeBaar 92 paper.
2. Go through each value space and look for examples where the DeBaar decision trees fail. I already know that this happens for the boolean space, and I suspect that it happens elsewhere as well.
3. Document these examples and come up with new decision tree attributes that can be used to describe them. Mark attributes that have something to do with interface structure (the next thing that I need to think about it).

Pittsburgh, PA: 18:42 EDT

A pressing question right now is whether or not the group tree contains as much information as the dependency graph...i.e. is the dependency graph worthwhile? So far, I see two differences between the two trees.

1. The group tree is specified, to some extent, by the designer rather than by factors related to the device. The dependency graph is directly specified by the device with no designer interference (presuming there are no human errors in the dependency graph).
2. The branches in the group tree do not come with any indicator of significance. Some branches are strictly different and should be organizationally separated at a high level, whereas other groups are only present to help separate large groups of components into smaller sub-sections on small devices. The dependency graph is useful for determining the significance of branches in the group tree.

I think the second point is more significant than the first.

Pittsburgh, PA: 22:15 EDT

Here are the papers that I found that cite the earlier paper. I follow this list with discussion.

• Francois Bodart, Anne-Marie Hennebert, Jean-Marie Leheureux, and Jean Vanderdonckt. "Towards a Dynamic Strategy for Computer-Aided Visual Placement," Advanced Visual Interfaces, 1994. pp. 78-87

• Jean Vanderdonckt. "Knowledge-Based Systems for Automated User Interface Generation: the TRIDENT Experience," Technical Report RP-95-010. Namur: Facultes Universitaires Notre-Dame de la Paix, Institut d'Informatique 1995.

• Francois Bodart and Jean Vanderdonckt. "On the Problem of Selecting Interaction Objects," Cockton G., Diaper S.W., Weir G.R.S (eds), People and Computers IX (HCI `94), Cambridge University Press, 1994. pp. 163-178

• Jean Vanderdonckt. "Advice-Giving Systems for Selecting Interaction Objects," User Interfaces to Data Intensive Systems. 1999. pp. 152-157

• Siegfried Schreiber. "Specification and Generation of User Interfaces with the BOSS-System," In Blumenthal B., Gornostaev J., Unger C. (eds.): Proceedings of EWHCI'94. Berlin: Springer-Verlag 1994 (Lecture Notes in Computer Sciences, vol. 876, pp. 107-120).

• Andrew Sears. "Layout Appropriateness: A metric for evaluating user interface widget layout," IEEE Transactions on Software Engineering 19(7), July 1993. pp. 707-719

• Jean Vanderdonckt and Francois Bodart. "Encapsulating Knowledge For Intelligent Automatic Interaction Objects Selection," in Proceedings of ACM Conference on Human Aspects in Computing Systems InterCHI'93, S. Ashlund, K. Mullet, A. Henderson, E. Hollnagel, T. White (eds.) Addison Wesley, vol., pp. 424-429

The Analysis...

Pittsburgh, PA: 19:42 EDT

Widget Selection, Continued...

There is still some question of when to assign a checkbox vs. a button whose title changes. e.g. In the phone interface I used a button with two different titles as the power button for the phone. I chose a button instead of a checkbox because a checkbox did not feel right in that situation. This should be investigated further... Is there some rule that can be gleaned from this choice?

Side Note  Can the decision tree that represents the choice of widgets be learned with a decision tree learning algorithm? Could we collect data that would be meaningful? Of course, before we can collect data we would need to pick attributes of each state to learn from. Picking these attributes is decidedly non-trivial.

The paper referenced yesterday uses different attributes for each type. For the numeric types (integer and real) there are four attributes: content, limits, range, and precision. NOTE: content is always in the set {editable, read-only}. These are calculated from raw information about the possible contents of the variable. The boolean type is automatically assigned to be a checkbox, as discussed elsewhere. The string type has two attributes: content and length. The enumerated type is the most complicated, using five attributes: items, set size, label, min, and max. As I mentioned yesterday, these attributes, and the rules based upon them, make a lot of sense. Something does seem to missing though.

To get a little more insight into the problem and what others have done, I did a search for papers that cited the earlier paper. I found several. I reference them in a later entry, and discuss what additional information that I've learned.

Pittsburgh, PA: 22:48 EDT

It's been quite awhile since I've written in my weblog... I will be trying to do a little better in the future, because I feel that this venue really helps me organize my thoughts and keep my direction clear. I feel that's something that has been lacking over the last month or more.

Anyway, I have reached the point where I need to think about widget selection algorithms. I think that it is pretty clear that some sort of decision tree algorithm will need to be a part of this process. There are other issues though...Such as how to treat elements that have a standard layout. I thought that I would try to find at least one or two papers on previous widget selection so that I could get the problem in perspective. Below is a reference to the most accessible paper that I could find on the subject.

• Dennis J.M.J. de Baar, James Foley, and Kevin Mullet "Coupling Application Design and User Interface Design," In Proceedings of CHI'92, May 3-7, 1992. Monterey, CA. pp. 259-266.

This paper cleared up one important issue for me...Specifically that the rule systems that have been described in the literature were basically decision trees with a different name. This adds a litle to my confidence that at least I'm thinking in a way that's similar to people who have worked on this problem in the past. However, while reading the paper, I was constantly struck with the thought that something is different with the personal universal controller.

The main thing that bothers me is the choice of treating the Power variable as a button instead of a checkbox or something else. Under the rules presented in the above paper, it would have become a checkbox. Clearly more is going on in our case and this needs to be investigated. More on this tomorrow.

Pittsburgh, PA: 22:39 EDT

It's been a little while since I last updated this page. Since last time, I've been thinking about using dependency and grouping information to creating interfaces, especially about methods that can be used to separate components across different panels. I have some ideas, which I will elaborate on in a separate entry later tonight.

I have also just finished reading four papers, which have been of questionable relevance... I will list them here for the record.

• James Foley, Won Chul Kim, Srdjan Kovacevic, and Kevin Murray. "Defining Interfaces at a High Level of Abstraction," IEEE Software, January 1989. pp. 25-32

• Mark A. Linton, John M. Vlissides, and Paul R. Calder. "Composing User Interfaces with InterViews," IEEE Computer, February 1989. pp. 8-22

• John M. Vlissides and Mark A. Linton, "Unidraw: A Framework for Building Domain-Specific Graphical Editors," ACM Transactions on Information Systems, Vol 8, Issue 3. 1990. pp. 158-167

• John M. Vlissides and Steven Tang. "A Unidraw-Based User Interface Builder," In Proceedings of the fourth annual ACM symposium on User interface software and technology, November 11 - 13, 1991, Hilton Head USA. pp. 201-210

The first paper is a UIDE paper that I hadn't read, which was I had thought might be interesting because I was thinking about the knowledge representation architecture in the PUC research. Unfortunately, I didn't learn much from this paper that I didn't already know from other sources.

The remaining papers come from the InterViews research. Unfortunately, I can't remember why it was suggested that I read these papers... They were somewhat interesting, although I didn't think that they were directly relevant to my project. Perhaps they will be of use later.

Gotta do more work tonight in preparation for the Maya meeting tomorrow... A rudimentary algorithm will hopefully be discussed here later tonight.

Pittsburgh, PA: 17:16 EDT

Haven't accomplished much so far today, but I just wanted to briefly discuss my plans for the rest of the day. I started reading the InterViews paper last night and today, and I'd like to finish reading all four of the papers tonight. I'll put complete references here with descriptions of interesting points raised by the papers.

I also want to talk a little bit about ideas for testing the semantic snarfing mechanism vs. laser pointers and other distance pointing devices. I'll put a little about that in here later tonight. Now it's time for ultimate...

Pittsburgh, PA: 01:05 EDT

I was just trying to go through a grid-layout algorithm on my whiteboard and I realized a few things that I wanted to get down in textual form.

• I was thinking of using some kind of "importance" queue in the layout algorithm. Basically, a queue of items would be created by reading through the interface element tree and pushing the items onto the queue in order of most importance. A screen would be created by taking items off the queue until there is no more room on the screen. Some kind of backtracking would probably be necessary if only half of a really important group could be placed or something... There are a couple problems with the important queue though... Items with the same importance have to occupy the same queue slot, and then other decision metrics need to be applied to determine which element to pick first. My initial tests have been with the stereo's speaker setup screen, which has some interesting grid and symmetry uses. It's not clear how the importance queue would help us find and exploit symmetry in our components.
• One thing that occurred to me while I was thinking about "finding" symmetries was the concept of choosing widgets but keeping their parameters unchosen until later in the layout phase. Perhaps by doing an iterative search of choices for these parameters, symmetries could be found. This certainly might be possible with the scroll bars in the speaker setup dialog, because setting the parameters the same, and placing the scrollbars next to each other would create a nice symmetry.
• I am also thinking about two-dimensional layout and or analysis algorithms. It seems that humans have an advantage over computers for this task because of our two-dimensional visual perception capabilities... Kim does some 2D analysis in his work, but I'm not sure if that would help a generative algorithm such as ours...
• I also thought of using an abstract grid initially, placing objects within it, and then pruning the grid until the simplest layout is obtained. In other words, as we removed elements from the "importance queue", we would only place them into rows. No column matching would be done. Then we would inspect down the rows, finding the minimal columns, or perhaps fitting the smallest element to the canonical grid. Then we would fit the larger element into the grid, rearranging as necessary for symmetry and size considerations. The biggest problem I see with this method is understanding whether a rearrangement is good or not. It seems that some might not be. Either the arrangment into rows needs to be done more intelligently, or we'll need some knowledge of what can and can't be done maintained (the information should be there somewhere so this might not be hard).
• There might also be some way of inspecting the importance queue for possible symmetries, and other design groupings, before placing items onto the grid. It would only be necessary to check for graphical relationships between objects that are adjacent, or at least nearby, in the queue. It might also be better to do these checks as the decision maker for whether to place an item in the current row or the next row (horizontal vs. vertical symmetry).
• Hints are used at just about every level of the interface generation. Hints are needed to guide the choice of the appropriate parameters and even widget for each variable. Still need to do more work on where they come in and how they are specified. We also need to worry about the effects of a hint on more than one interface type. It is very conceivable that a spec. author might add a hint when an interface doesn't come out quite right that could have negative effects in other situations.

I seem to think better when I write in the weblog, so I think that'll be good motivation for using this in the future.

Pittsburgh, PA: 16:53 EDT

I haven't been writing consistently in this log since I got back to Pittsburgh. I'll try to improve my performance in that area from here on out, although I suspect that the baseball trip in a week and a half will screw things up again.

Anyway, I've been working on a few things in the past week.

• I've thought a bit about laying out components using a global grid, rather than the tree algorithm that was used by Won Chul Kim. I think that this is at the point where I need to implement something or at least diagram on a whiteboard.
• I've been thinking about organizing components based upon the dependency graph and other "hint" information. This kind of information would tell us that there are standard arrangements for the "Play" buttons, etc.
• I've been helping Rishi reset the semantic snarfing experiment and improve it. I am currently working on finding a new task to use in the experiment that is more in line with the kind of interactions we would expect from the semantic snarfing idea.

Brad also found a magazine called Electronic House (Vol 16, No. 5, May 2001), see sidebar for link, which deals with topics about home automation. The magazine is mainly about technology that is available now. It was interesting to see products that might be replaced with a functional PUC. One site that I found a reference to in the magazine was LiveManuals.com which provides access to electronic copies of the manuals for most consumer electronic products. This might be useful in the future if we decide to prototype more devices. We may also contact the Pittsburgh home automation products installer listed in the magazine, to see what kind of products are available for automating the home that could conceivably be controlled by the PUC.

Pittsburgh, PA: 00:27 EDT

I finally finished reading Won Chul Kim's thesis on the plane ride home yesterday. I found several things of interest:

• Layout is done from a tree-like data structure, which limits the quality of some of the interfaces that can be created. For example, some interfaces can be simplified by using a tabular layout with labels on both the rows and columns. A tree is not capable of representing the dual relationship of the items in a tabular layout, preventing Kim's engine from building an interface of that type.
• The bottom-up layout approach that Kim uses does not work well with a grid-layout such as the one recommended by Mullet & Sano. No grid is used by Kim, and his designs suffer as a consequence. It seems to me that grids require some global knowledge of the layout, whereas the bottom-up/tree approach only allows local knowledge. In the bottom-up/tree approach, you don't know about the layout of other components in different tree branches. This means that components do not always end up aligned with each other across functional units. Kim does provide a column layout feature, but the items in each column do not always line up horizontally, which can be hard to look at. Clearly a key element of our work will be using a grid more effectively than Kim.
• Kim has separate systems for creating menus and dialog boxes. This is something that we haven't talked about, but will want to do. I am curious if the designs for cellular phones can be generated exclusively from the menu system, or whether some additional work will need to be done especially for that system.
• Kim has implemented several design evaluation techniques, but doesn't try to iterate on them to pick a design. He relies on the designer to pick the best work and then modify that. This keeps with their philosophy of building an assistive tool instead of trying to do all the work. Once in the dissertation Kim says that there is no layout algorithm that will do a perfect job. Human intervention is apparently required.

Another thing that I realized, while reading Kim's thesis, is that we have focused very little on the specification of the style component of our generation engine. In using style, I am keeping with the terminology used by Kim, the ITS projects, and others, which refers to the interface conventions used by the designer. In our case, this refers to a couple of related knowledge bases: interface conventions for the controlling device and cultural knowledge of the user. There may be other categories...I'm still working this out.

To get a handle on these other categories, I've started examining the hand-designed interfaces in order to extract the knowledge from these categories. I am hopeful that this will help me understand the form of this knowledge and what the best representation for it will be. This should help the construction of the interface generator. More on that here, when I finish.

San Diego, CA: 11:18 PDT

I need to stop being so ambitious with my goals... Reading the thesis is taking more time than I expected, although I think that the extra time spent is well worth it. Kim thought about most of the issues that I need to consider, although from a slightly different point of view, and it has been beneficial to read about his design decisions. Based on the reading that I've done so far, it's looks like it will be difficult to start an implementation by the end of the week, but that's still the goal I am working towards...

Today's Goals:

• Read Won Chul Kim's Thesis
• Decide what to implement

I'm hoping that the thesis will give me some insight into what I should concentrate on for the rest of the week. I may simply want to reimplement some layout algorithms described in the thesis.

San Diego, CA: 14:41 PDT

I have three weekly goals and a couple of daily goals that I want to write about today. First of all, this week I will be trying to write in this log more regularly and also earlier. I will be trying to write the first entry of my daily log by noon in my current timezone. My third, and most ambitious weekly goal, is to have something interesting implemented by the end of the week. My goal right now is to give my program a list of widgets and groupings and have it output a reasonable layout from the input. I will be testing the program will a number of screens that are currently in the hand-designed stereo and phone interfaces, just to see how the designs compare.

To get started on my third weekly goal, today I will be exploring how Won Chul Kim implemented some of these ideas and thinking about the difference between generative vs. iterative refinement design. A key question for our implementation will be whether our architecture will have an iterative improvement loop or whether it will be able to design good interfaces after one pass through the design phase. It seems that most design rules assume iterative improvement...How can we work this into our system, given that iterative improvement requires evaluation as one of the critical steps.

Today's Goals

• Read Won Chul Kim's thesis
• Brainstorm some basic architectural ideas for implementing the PUC
• Compare and contrast these ideas

The later steps will be documented in the form of notes posted to this site at the end of the day today or beginning of the day tomorrow. Hopefully I will be able to use them as a basis for starting an implementation tomorrow.

Before I finish this entry, I just remember that there is another design book I have looked at that I wanted to briefly mention. It is not as well organized as the Mullet & Sano book, nor is it targeted at GUI design in particular. Jodi referred it to me as a resource for grid-based design. I have only skimmed the book so far, and haven't found anything interesting enough to mention here. The reference follows:

• Philip B. Meggs. Type & Image: The Language of Graphic Design. John Wiley & Sonss, Inc.: New York, NY. 1992

San Diego, CA: 01:27 PDT

I have finally finished writing down my notes for the Mullet & Sano book. I finished the book on Thursday as intended, but the notes took awhile to write up. See the sidebar for a link. Tonight I am going to look over Won Chul Kim's thesis some more, and finish looking it over tomorrow. More about that tomorrow in Today's Goals.

M&S Notes

San Diego, CA: 11:22 PDT

Good news today... NSF accepted our personal universal controller proposal! So now I have even more motivation to get working... Here are my goals for today:

Today's Goals:

• Finish reading Mullet & Sano
• Write up some notes on the relevant material in the book, to be posted here.
• Find two new outfielders and a shortstop for my fantasy baseball team.

San Diego, CA: 01:09 PDT

I didn't quite accomplish my goals for today. I came up with a plan for the for the baseball trip, but only read the first four chapters of Mullet & Sano. I can already tell that the book is going to be really useful, however. Chapter 4 is particularly useful, as far as design rules go... The material about symmetry and alignment seemed directly applicable. The next chapter should be the most important however, because it covers gridding, which will probably be the foundation of the layout engine.

Here are the references to what I've been reading in the last two days:

• Kevin Mullet and Darrell Sano. Designing Visual Interfaces: Communication Oriented Techniques. Prentice Hall, 1995.
• Won Chul Kim and James D. Foley. "Providing High-level Control and Expert Assistance in the User Interface Presentation Design," In Proceedings of INTERCHI'93. 24-29 April 1993. pp. 430-437
• Won Chul Kim. Knowledge-Based Framework for An Automated User Interface Presentation Design Tool, Ph.D. Dissertation, The George Washington University, February 14, 1993.

I read the second paper yesterday, and got a reasonable idea of the system that Kim built. I am looking forward to getting more detail from the dissertation, although I imagine that it will be a long read. I did note that Kim used the Mullet book as a basis for some of the rules that are a part of his system. I also noted that he used an existing reasoning package to implement his system. This makes me wonder how easy it will be to implement our system straight out in Java, or some other language. Things to think about...

In fact, it seems to me like there could be a number of modules in the system. There is a rule-solving module, some kind of graph manipulation module, a layout module, a widget-selection module, etc. I am worried about the amount of infrastructure that will need to be created in advance...

San Diego, CA: 15:58 PDT

I just finished formatting this page in a way that I kinda like. I will now leave my web page alone for another six months to a year... :-)

This weblog will be my research journal...especially for when I am away from Pittsburgh. I am hoping it will motivate me to get things done instead of loafing. The general format should include about two entries per day... One like below, with a goals section. The second entry should cover what I actually accomplished and any interesting ideas I had during the day.

Today's Goals:

• Read Mullet & Sano, "Desiging Visual Interfaces"
• Plan summer baseball trip