\subsection{Overheads of synchronizing switch on iWarp}

Here we outline the overheads in the synchronizing switch prototype
implemented on an $8 \times 8$ iWarp system. We relate the overheads
in cycles for a system running at 20 MHz. The speed of the
synchronizing switch is of great importance, since the overheads of
the local switches propagate through the network, so the overheads are
multiplied by the network diameter. Every 2 cycles of overhead
increases the message size needed for half peak bandwidth by 4 bytes.

By measuring the time to perform an AAPC with no data, we determined
that the synchronizing switch overhead is 333 cycles/phase. This time
includes the time to propagate the message headers. The header takes 2
cycles per node and 2 to 4 cycles per link. For a network with
diameter 8, this propagation delay accounts for 32 to 48 cycles in
overheads.

Both the phased and the message passing implementations include overheads
to start sending a messages. On iWarp this includes setting up the message,
generating a route, and putting the router in the proper state to send and
receive messages. This requires an additional 120 cycles.

The remaining 25 cycles/link are delays due to the software
implementation of the synchronizing switch. We anticipate that this
overhead would be eliminated if the queues enforced the local
constraint in hardware as described in Section~\ref{hw-add-sect}.
Figure~\ref{break-fig} shows a breakdown of processing overhead
incurring in all iWarp implementations of AAPC.

\begin{figure}[htb]
\epsfxsize=5in
\centerline{
\epsfbox{\figpath overheads.eps}
}
\begin{center}
\begin{minipage}{\caplen}
\caption{Breakdown of the per message processing overhead on a 64 cell 
iWarp system. The setup overheads apply to both, phased AAPC and
e-cube message passing. The synch-switch overhead and the
network-delay apply only to phased AAPC. The graph does not show
congestion delay for the e-cube message passing, since it is not a
constant processing overhead.}
\label{break-fig}
\end{minipage}
\end{center}
\end{figure}

In addition to the synchronizing switch overheads, each phase incurs
overheads to start sending data. Each iWarp router has 8 DMA
controllers to move data to and from the node's memory, and our
implementation of phased AAPC uses these DMAs. Starting the DMA agents
and testing for the DMA completion adds an additional 120 clocks to
the per phase overhead. Therefore, the complete per phase overhead on
an $8 \times 8$ iWarp system is 453 cycles.

In principle the iWarp architecture can source and sink messages at 40
MBytes/s by either systolic or memory communication
\cite{IEEE-computer-paper}.
% will appear in december 1994
Systolic communication reads and writes data directly from the
computation; network communicaiton looks like another register access.
Memory computation relies on DMA agents (spoolers) to move data
between the node's memory and the network without additional action by
the computation agent.  The phased AAPC routines can be implemented
with either communication mode; but message passing must use memory
communication and DMAs to handle the non-deterministic message arrival
of long messages.

