/section{Overhead of the local AAPC switch}

The router in the iWarp computation agent permitted us to implement a
fast protype of the local switch hardware through its program acces to
the communication state. All overheads are discussed in iWarp clocks
@20Mhz or 50ns. Two clock of overhead add a 4 bytes or 1 word to the
minimal message size for 50% perfomance.  Accesses to the control
state of any of the 20 iWarp router queues take 1 cycle for reads and
3 cycles for writes. Bookkeeping takes a few more cycles in the local
switch which performed at each node every phase.  Speed in this
routine is of great importance since the phase switches have to
propagate through the network and therefore the overheads are
multiplied with the network diameter.

We examined the overheads for the standard one-source/sink per cell
pattern as well as for a modified two sink/source per cell pattern
that left half of the cells idle in each phase. The latter pattern
permitted us to work around a hardware implementation problem which
originally limited our bandwidth to a half of the specified channel
limits. The overheads for the two source/sink patterns is higher since
two routes must be generated by a single processor in each phase.
The implementation documented in this report automatically selects
the more effincient one source/sink implementation for small messages
and the asymptotically opimal two source/sink pattern to reach the
limits on long messages. We limit the discussion of the overhead
to the more efficient single source/sink patterns.

The 64 cell, 8x8 mesh is the smallest configuration in which we can use
our systematically derived patterns and this is also the largest system
available to the authors at the time their experiments were carried out.

The measure switching time for an empty AAPC amounts to 21'300 clocks
or 333 clocks/per phase. The time includes the propagation times of
the message headers which is 2-4 clocks through each link and 2 clocks
through each mesh node. With a maximal network diameter of 8 this
accounts for 32-48 clocks. Seeting up a message, generating a route
and putting the harware into sending state at the sender and reveiving
state at the receiver takes about 120 clocks, in the message passing
implementation as well as in the phased AAPC. The remaining 25 clocks
per hop are delays due in the software implementation of the
synchronizing switch.  This overhead would be eliminated if the queues
are interlocked and released by a hardware ``and'' gate, rather than by
software.

The iWarp architecture can source and sink the message 40 Mbytes/sec
by either systolic or memory communication \cite{IEEE-computer-paper}.
While systolic communication can supply and process data right out of
a computation, memory computation is relying on eight built in DMA
controllers (spoolers) to move the data from the memory of the sender.
Unlike in message passing, in phased AAPC the message arrival is fully
deterministic and therfore the data could be merged directly into a
computation with the AAPC algorithms.

The subsequent experimental rsults are all obtained for memory
communicateion. Starting the DMA agents (spoolers) at the sender and
the receiver and testing for completion of them costs about extra 120
clocks to the overheads. The total overhead for 64 phases memory to
memory including DMA was measured at 28900 clock or 452 clocks per phase.

