Proceedings of the InfoJapan '90 (Oct 1-5, 1990)
---------------------------------------------------------------------------
    Condela-3 : A Language for Neural Networks

Nikolaus Almassy, Monika Koehle,  Franz Schoenbauer

	Abstract

Condela-3 (Connection Definition Language) [1] is a high level
programming language, specifically designed for the development and
modeling of neural network applications. It is a procedural and
general purpose language, that allows parallelism via the concept of
selections, i.e. groups of units or connections to which actions can
be applied. Units and connections can be created dynamically at any
point in the program flow. The parallelism expressible in Condela-3 is
independent of the underlying Hardware. Condela-3 is easy to teach, as
it has few language constructs, yet allows the expression of arbitrary
network topologies and learning paradigms due to its powerful
statements and its two levels of abstraction. It is easily portable to
other operating systems, its open design allows simple interfacing
to existing applications.

	1. Motivation.

Our exploration of neural network models was greatly restricted by the
absence of a practicable way to describe various topologies of neural
networks. The existing tools are neither easy to read nor comfortable
to work with. Also it was difficult to describe dynamic creation of
units and links. A tool suited for teaching as well as for research
was needed. So the decision was made to define and implement an
enhanced  version of Condela [2,3]. 

	2. Required Properties of Neural Network Languages.

A programming language for neural networks should comprise all the
elements needed to comfortably define a network topology and
behavior. Clear statements and mnemonic keywords have to give an idea
of their meaning to inexperienced users. In addition, the parallelism
of the underlying computational model for neural networks should be
incorporated into the language design [4]. As some neural network
models attempt to simulate the human brain, a neural network -language
should permit dynamic creation of units and links.

	3. Description of Condela

Condela is a high-level programming language providing a built-in
parallel concept and is especially dedicated to neural network models.
It is not tied to any operating system or machine and it could also
be used as a general purpose language. It is procedural and to any
degree parallel when describing the learning and evaluation of the
model. Condela is not based on any single neural network paradigm
(such as error back propagation [5] in the sample program in 6.) but
rather enables users to simulate a wide variety of neural network
models. Condela can be approached by inexperienced users using
predefined unit types and learning paradigms, although real life tasks
may require definition of more complex units and learning algorithms.

	3.1 Data types

The data types of Condela are real, integer, boolean, vector, unit,
and unit-selection, the data types real, integer and boolean behave
in the usual manner. The data type vector may be regarded as a dynamic
array of real and objects of that type can be used with range-lists.
For example if v is a vector you can use v[0..3,7..vmax] as well; the
resulting vector consists only of those elements whose indices are
within the range list. The data type unit is used to access the
components of a single unit. The data type unitselection is used to
access groups of units.

                 unit-selection-A
             ,-----,
 layer_1   O | O O | O O
             |     '-----,
             |       ,---+-------,
 layer_2   O | O O O | Q | Q Q Q |  unit-selection-B
             |       '---+-------'
             '-----------'

             Fig. 1

Components of the marked units in Fig. 1 could be modified by the
following statement:

        { netid.layer_2[4..7] } : component_name := vector_2[6..9];

The colon after the unit-selection indicates that you are using the
components as a vector. There are no restrictions concerning the
unit-selections, they can overlap and they may consist of units of
different layers or networks and may even be of different types.

	4. Program structure

A Condela program consists of two successive parts. The first part
defines data objects and constants and declares network topologies and
symbolic types. The second part consists of procedures that modify
data. You can define local variables that are only visible in a
particular procedure. As Condela is not a block-oriented language,
nested procedures are not possible. After the keyword TOPOLOGY various
network topologies can be defined.
It is possible to arrange units in a linear order in a FIELD of units
with several dimensions, or to specify the absence of any order with a
POOL of anonymous units. A POOL is a set of units, i.e. you can not
explicitly select a certain unit in that POOL, you can only access a
number of units in a POOL. The order of access to the units in a POOL
is random.

[Fig 2 deleted...]

This is a sample structure of a NETWORK named mynet. See the syntax of
the description of this structure in the following paragraph.

	4.1 Syntax

The syntax of Condela is similar to Modula-2. A FIELD is an array of
units and a POOL is a set of units. As many sub layers as appropriate
can be specified, and of course every LAYER can have sub layers. There
is no restriction concerning the  number of dimensions of LAYERs,
FIELDs or NETWORKs. Upper case words designate keywords of Condela,
lower case words denote user definable identifiers.

    TOPOLOGY mynet =	FIELD[20];
			POOL(4);
			LAYER input OF FIELD[10]; END;
			LAYER hidden<3> OF	FIELD[10];
						LAYER dummy OF special_net;
					END;

In this example the layer hidden has one dimension with indices from 0
to 2 and a sub layer named dummy with a user-defined topology
"special_net" that has to be defined earlier (Please note that the
identifier special-net does not require the END- Keyword in this
definition). The POOL in the top level of mynet consists of 4 units
and can be accessed with p(i) where i is the number of units (not the
index!). Note that the order of units in a POOL can not be predicted.
In the following the description of the statements in EBNF that are
important for network update. Uppercase words denote keywords of
Condela and symbols that are not part of the meta-language of EBNF are
written in single quotes.

        <Statement> :==	FOR <Ident> IN <UnitSelection>
				DO { <Statement> ';' } END
			| 	APPLY <ProcedureCall> TO <UnitSelection>
					[ TO <UnitSelection> ]
			|	CONNECT <UnitSelection> TO <UnitSelection>
					[ TYPE <Ident> ] INIT <Expr>
			|	CREATE <Ident> { '<' <Expr> '>' }
			|	EXTEND <Ident><LayerId> BY <Size>
			|	<UnitSelection> ':' <Ident> ':=' <Expr>
			|	<Ident> ':=' <UnitSelection> [ ':' <Ident> ].
	<LayerId> :==	{ '.' <Ident> | '<' <RangeList> '>' }.
	<Size> :==	'[' <Expr> ']' { '[' <Expr> ']' }
			|	'(' <Expr> ')'.
	<UnitSelection>:=='{' <UnitSet> { <UnitOp> <UnitSet> } '}'
			|	<Ident>.
	<UnitSet> :==	<Ident><LayerId><UnitId>.
	<UnitId> :==	{ '[' <Expr> ']' }
			|	'(' <Expr> ')'.
        <UnitOp> :==	INTERSECT
			|	JOIN
			|	LESS.
	4.2 Statements

The following statements refer to the sample topology in 4.1. Before
a network is used, it must be created. A network variable has to
be defined with a specific network type (with optional dimensions):

	VAR     p : NETWORK OF  mynet;

        CREATE p;
After this statement internal memory is allocated to the network.
After network creation certain units may have to be connected to
others:

	CONNECT { p.input } TO { p.hidden<0..2> }
		TYPE mylink INIT random();
With this statement a special type of connection named "mylink" is
established. Every unit in the layer "input" of the network "p"
(including all sub layers of "input") is connected to all units in
all three "hidden" layers (with their sub layers "dummy"). A special
component defined by the description of mylink  is initialized by the
expression after the INIT keyword.

Now user-defined functions coded in Condela or C or any other
language (that allows the usage of pointers) and implemented on the
host machine can be applied to unit-selections or weight-selections:

	APPLY feed_forward(k) TO { p.hidden<0,2> };
With this statement the layer p.hidden<1> will remain unaffected.
A weight-selection can be expressed by a second "TO" -term: 

	APPLY myfunc() TO { p.input } TO { p.hidden };
This notation encourages thinking parallel irrespective of the actual
implementation. The dynamic creation of links is done by the CONNECT
statement, and dynamic unit creation (after the creation of the net)
is done by the EXTEND statement:

	EXTEND p.hidden<1> BY [5];
After this statement five units have been added to the field component
of the layer. If round brackets had been used, the units would have been
added to the pool component of the layer. Of course the new units
aren't connected yet.
Beside the concept of the unit-selection there is a useful
implementation of a vector processing facility. Vector expressions are
used to access components of the units within a unit-selection:

	{ p.input_a } : component_name := vector_x;
or
	vector_y := { p.input_a } : component_name;

In this example the components of a unit-selection form a data object
that is similar to a vector and therefore they are
assignment-compatible. The only restriction is that unitselections
with their components cannot be used in conjunction with other
vectors.
Symbolic unit-selections can be used to aid code optimization and
yield a significant speed up of program execution: 

	il := { p.input[0..1] };

After this statement the identifier "il" can be used like any other
unit-selection. The difference is an implementation that makes the
code faster. In the present implementation explicit assignment of a
single unit to a unit variable is only allowed within a FOR loop.
This loop iterates as many times as there are  units in the
unit-selection. The control variable subsequently refers to all units
in the selection and can be referenced within the FOR block as if it
where a unit. To access components of a unit variable you  can use a
Modula - like component selection:

	my_unit.net := 0.32;

This syntax is also used to access the appropriate sublayers of a
network.

	5. Implementation

To make it easier to use Condela on different operating systems it is
implemented as a pre-compiler to standard C with the well known
UNIX-tools lex [6] and yacc [7]. A direct translation into into
assembly  language while speeding the translation would prohibit the
use of other machines.

   ,----------,
   | Condela  |  User defined topologies, program-flow, I/O, ...
   '----------'

   ,----------,
   | C        |  Predefined unit-types and learning rules, user-
   '----------'  defined unit-types, ..

   ,----------,
   | Assembly |  Executable code
   '----------'

         Fig. 3

In the present implementation, sophisticated unit-types and propagation
rules must be coded in C. Future versions of Condela will allow the
usage of pointers, such that every non-predefined unit-type or
learning-rule can be coded in Condela. This will simplify the
development of new applications.
After compilation of the Condela-program (by the Condela-compiler) and
after compilation of the resulting C code, all the necessary
procedures (with the runtime library) are linked together. The result
is a executable file that is ready to run (See Fig 4). 

	myprog.con					Condela 
            |
        ,---------,
	| condela |
	'---------'
	    |
	 myprog.c	mylib.c		condela.c	C
            |		   |		   |
	 ,----,		,----,		,----,
	 | cc |		| cc |		| cc |
	 '----'		'----'		'----'
	    |		   |		   |
        myprog.o       mylib.o         condela.o
            |		   |		   |
	    '--------------+---------------'
			   |
			,----,
			| ld |
			'----'
			   |
			   V
			 a.out				Assembly

	Fig. 4


	6. Performance

The sample program in the following section performs on a DECstation
3100 one million updates in less than two minutes, that are about 8474
learning steps per second. On a mVAX II the same program needs about
39 minutes of CPU time.

	7. A sample program

The following sample program demonstrates the classical XOR learning
problem using error back propagation.

TOPOLOGY
	xor =	LAYER input OF FIELD[2]; END;
		LAYER hidden OF FIELD[2]; END;
		LAYER output OF FIELD[1]; END;
		LAYER dummy OF FIELD[1]; END;
VAR	p : NETWORK OF xor;

PROCEDURE main();
VAR	output_vec, input_vec : VECTOR;
	input_layer, hidden_layer, output_layer : USEL;

BEGIN
	CREATE p;
	input_layer := { p.input[0..1] };
	hidden_layer := { p.hidden[0..1] };
	output_layer := { p.output[0] };
	CONNECT input_layer TO hidden_layer INIT random();
	CONNECT { p.dummy } TO hidden_layer INIT random();
	CONNECT hidden_layer TO output_layer INIT random();
	CONNECT { p.dummy } TO output_layer INIT random();
	LOOP 1000000 TIMES
		get_input(input_vec, output_vec);
		input_layer : out := input_vec;
		APPLY feed_forward() TO hidden_layer;
		APPLY feed_forward() TO output_layer;
		APPLY back_propagate_out(output_vec) TO output_layer;
		APPLY back_propagate_hid() TO hidden_layer;
	END;
END;

	8. Conclusion

A high level programming language for neural computing is presented.
The descriptions of network topologies are easy to read and
comfortable to work with. The user can define suitable unit and
connection types for various applications. The main concept of Condela are
the two levels: the Condela level with the definition of network
topology and behavior, and the system level with the definition of the
unit components and learning strategies. Even the inexperienced user
can start immediately to work with the predefined unit and connection
types.

	9. References

[1]	Nikolaus Almassy: "A Compiler for Condela-III",
	Master thesis, Technical University Vienna Apr. 1990.
[2]	Monika Koehle, Franz Schoenbauer: "CONDELA - A Language for
	Neural Networks", Proceedings of the nEuro '88 in Paris.
[3]	Martin Schwarzenecker: "Design and Implementation of a
	pre-Compiler for Condela-II", Master thesis, Technical
	University Vienna, 1990. 
[4]	Philip Treleaven, Michael Recce:  "Programming Languages for
	Neurocomputers", in "European Seminar on Neural Computing" by
	IBC, Technical Services LTD., London 1988. 
[5]	David Rumelhart, James McClelland and the PDP Research group
	(eds.): "Parallel distributed Processing", MIT Press, Cambridge 1986
[6]	M.E. Lesk and E. Schmidt: "LEX - A Lexical Analyzer Generator",
	Computer Science Technical Report No. 39, Bell Laboratories,
	Murray Hill, N.J., 1975. 
[7]	S.C. Johnson: "YACC - Yet Another Compiler-Compiler",
	Bell Laboratories, Murray Hill, N.J., 1977. Technical Services
	LTD., London 1988.
[5]	David Rumelhart, James McClelland and the PDP Research group
	(eds.): "Parallel distributed Processing", MIT Press, Cambridge 1986
[6]	M.E. Lesk and E. Schmidt: "LEX - A Lexical Analyzer Generator",
	Computer Science Technical Report No. 39, Bell Laboratories,
	Murray Hill, N.J., 1975. 
[7]	S.C. Johnson: "YACC - Yet Another Compiler-Compiler",
	Bell Laboratories, Murray Hill, N.J., 1977.
