For this study, we have chosen an architecture that resembles the DASH multiprocessor , a large-scale cache-coherent machine that has been built at Stanford. Figure shows the high-level organization of the simulated architecture. The architecture consists of several processing nodes connected through a low-latency scalable interconnection network. Physical memory is distributed among the nodes. Cache coherence is maintained using an invalidating, distributed directory-based protocol. For each memory block, the directory keeps track of remote nodes caching it. When a write occurs, point-to-point messages are sent to invalidate remote copies of the block. Acknowledgment messages are used to inform the originating processing node when an invalidation has been completed.
We use the actual parameters from the DASH prototype wherever possible, but have removed some of the limitations that were imposed on the DASH prototype due to design effort constraints. Figure also shows the organization of the processor environment we assume for this study. Each node in the system contains a 33MHz MIPS R3000/R3010 processor connected to a 64 Kbyte write-through primary data cache. The write-through cache enables processors to do single-cycle write operations. The first-level data cache interfaces to a 256 Kbyte second-level write-back cache. The interface includes read and write buffers. The write buffer is 16 entries deep. Reads can bypass writes in the write buffer if the memory consistency model allows this. Both the first and second level caches are lockup-free , direct-mapped, and use 16 byte lines. The bus bandwidth of the node bus is 133 Mbytes/sec, and the peak network bandwidth is approximately 120 Mbytes/sec into and 120 Mbytes/sec out of each node.
The latency of a memory access in the simulated architecture depends on where in the memory hierarchy the access is serviced. Table shows the latency for servicing an access at different levels of the hierarchy, in the absence of contention (the simulations done in this study do model contention, however). The following naming convention is used for describing the memory hierarchy. The local node is the node that contains the processor originating a given request, while the home node is the node that contains the main memory and directory for the given physical memory address. A remote node is any other node. The latency shown for writes is the time for retiring the request from the write buffer. This latency is the time for acquiring exclusive ownership of the line, which does not necessarily include the time for receiving acknowledgment messages from invalidations, since the release consistency model is used .