[dw92.abs]  

T. Gross and S. Hinrichs, "Debugging a Parallel Program: Capturing
Inter-Processor Communication in an iWarp Torus", Proc. Supercomputer
Debugging Workshop '92, Dallas, October, 239-276.

Abstract: To understand or improve the execution behavior of a program
on a parallel system, it is often necessary to consider the
interaction between the processors in the system.  Since communication
is important for all parallel programs, obtaining information about
the inter-process communication of the program is an important aspect
of understanding program execution.  If the user's concern is the
performance of the parallel program, then the user must be able to
capture the dynamic aspects of inter-processor communication.
Unfortunately, timing information about inter-processor communication
is often not easy to obtain.  Building a special-purpose hardware
performance monitor is too costly in most scenarios, and the use of
VLSI to integrate communication and computation on a single component
often provides only a few externally visible measurement points.  A
software monitor on the other hand is often too slow to allow
execution of the monitored program without serious perturbation.

Although the integration of communication and computation engines on a
single component provides some challenges to monitoring, it also opens
the opportunity to program the processors in a parallel system so that
they can monitor communication traffic.  That is, during monitoring
some processors execute the user program (to be monitored) while other
processors execute a special monitor program (which captures
information on the inter-processor communication).  With adequate
software tools, this information then can be analyzed to present a
picture of the communication between the user processes.  This paper
discusses the benefits and difficulties of an implementation based on
this idea for the iWarp system.  We conclude that a programmable
processor that integrates communication and computation is also suited
to serve as a hardware monitor at a fraction of the cost of a
special-purpose design.