\section{Communication mechanisms}
\label{sec:commmech}

All of our test applications use the PVM system for communication.
PVM~\cite{PVM,PVMBOOK} is a message-passing and utility package which
provides a presentation layer interface which has the syntax and
semantics of message passing interfaces on distributed memory parallel
supercomputers.  In addition to message passing, PVM also provides
mechanisms for managing a dynamic, heterogeneous pool of machines as a
single ``parallel virtual machine.''  This support is implemented in a
user-level daemon process which is run on each machine.  The daemons
talk to each other via UDP in order to maintain information about the
global state of the virtual machine, as well as to handle user
requests such as sending signals to remote user processes.  Each
machine may run multiple user processes.  A user process can
communicate with another user process on the same machine or on a
different machine using the same interface.  Intramachine
communication is done via a local IPC mechanism.  Intermachine
communication can be done in two distinct (user selectable) ways.  By
default, the message is copied via IPC to the daemon, which sends it
to the daemon on the destination machine via a protocol built on top
of UDP.  The receiving daemon then delivers the message to the
destination process via IPC.  This mechanism has the advantage of
better scalability, but tends to be somewhat slow.  In the alternative
mechanism, the messages are sent directly from the sender process to
receiver process via TCP.  All of the Fx kernels and AIRSHED use this
mechanism.

PVM messages can contain arbitrary data collected from arbitrary
memory locations.  Data is ``packed'' into a message using a variety
of API calls.  However, the data is not necessarily appended into a
contiguous memory buffer.  Instead, it is stored as a list of
fragments which are sent independently.  This distinction is important
to understand the behavior of one of the Fx kernels, T2DFFT.  All the
other kernels (and AIRSHED) assemble their messages in a copy loop
{\em before} using PVM.  The result is that each message is sent as a
single, large fragment by PVM.  The copy loop is an artifact of other
(older) Fx implementations for message passing systems which only support
sending contiguous buffers.  T2DFFT, however, tries to avoid the
intermediate copy step by performing multiple packs per message.  The
result is that each message is passed to the socket layer as a series
of fragments.