\section*{Code Generation}

Archimedes programs are written in C, augmented with high-level operations
specific to finite element methods.  Archimedes' output is also C code,
with FEM libraries and machine-specific communication code added.
There are several reasons why
parallel applications should be automatically generated from an augmented
language.  First, few
scientists or engineers want to invest the time necessary to
learn how to program a multicomputer.  Second, Archimedes' high-level commands
allow one to quickly change code and experiment with different
linear solvers, preconditioners, ODE
integrators, and so on.  Third, 
a general-purpose finite element code generator ensures
that the effort spent writing optimized parallel linear solvers and
other application code can be
easily reused for different simulations.  Fourth, every application
written in the language can be
ported to new architectures by porting the code generator.

We illustrate Archimedes' language with an excerpt from code written
to solve the heat equation.

In addition to the usual C data types,
Archimedes has several distributed data types,
including vectors and sparse matrices.
Consider the inner loop of a simplified conjugate gradient
code~\cite{shewchuk94b}
to solve ${\bf Ku = f}$.  Below, the {\tt MATRIX} keyword declares a
distributed sparse matrix, and the
{\tt NODEVECTOR} keyword declares distributed
vectors. (We omit a detailed description of the implementation of these
distributed data types; the approach used is similar to that described 
earlier in the paper, under the heading 
``Parallelizing Sparse Matrix-Vector Products''.)

Informally, the code fragment below performs the following conjugate
gradient inner loop: 
(JONATHAN: HELP! I DON'T KNOW WHAT TO SAY HERE.)
\\
\\
{\tt MATRIX} ${\bf K}$; \\
{\tt NODEVECTOR} ${\bf u, d, q, r}$; \\
{\tt float} dq, norm, firstnorm, oldnorm, alpha, beta; \\
\\
(Setup code is omitted here.) \\
{\tt while} (norm $>$ EPSILON $*$ firstnorm) \{ \\
\verb.  .beta = norm / oldnorm; \\
\verb.  .${\bf d}$ = ${\bf r}$ + beta $* {\bf d}$; \\
\verb.  .{\tt MVPRODUCT}(${\bf K, d, q}$); \\
\verb.  .{\tt DOTPRODUCT}(${\bf d, q,}$ dq); \\
\verb.  .alpha = norm / dq; \\
\verb.  .${\bf u}$ = ${\bf u}$ + alpha $* {\bf d}$; \\
\verb.  .${\bf r}$ = ${\bf r}$ - alpha $* {\bf q}$; \\
\verb.  .oldnorm = norm; \\
\verb.  .{\tt DOTPRODUCT}(${\bf r, r,}$ norm); \\
\} \\

% JONATHAN: I THINK A SHORT PARAGRAPH HERE DESCRIBING AT A VERY HIGH LEVEL
% WHAT THE CODE ABOVE IS DOING WOULD BE VERY HELPFUL HERE.  UNFORTUNATELY,
% I HAVE NO IDEA WHAT THAT WOULD BE.  COULD YOU PLEASE ADD SOMETHING HERE?
Several lines of code above perform mixed scalar/vector arithmetic; these
lines can be executed without any communication.
The {\tt MVPRODUCT} operation forms
the product ${\bf q = Kd}$, and the {\tt DOTPRODUCT} operation forms the
products dq ${\bf = d \cdot q}$ and norm ${\bf = r \cdot r}$,
each requiring communication.
Because many simulations can be implemented with only these two
communication primitives, a simple version of
Archimedes is easy to port to new architectures.
Although Archimedes includes a variety of other commands that implement
stiffness matrix formation, preconditioning, and other operations,
these do not require communication and are trivially portable.

In addition to ordinary iterative methods like conjugate gradients,
Archimedes includes support for a numerical technique known as
{\em domain decomposition}, which can greatly improve the
solution time for time-dependent simulations.  Domain
decomposition methods combine direct methods for solving linear systems
with iterative methods, in such a way that they are easy to
parallelize (recall that
pure direct methods are not).
Briefly, each processor uses standard Gaussian elimination to eliminate
the interior nodes of its subdomain; then an iterative method is used
to solve the linear system for the shared nodes on the boundaries of
subdomains.  Further details are given in the survey by
Chan~\cite{chan94} and the article by Shewchuk and Ghattas~\cite{shewchuk94a}.
