========================================================
(C)1993 Institute for New Generation Computer Technology
(Read COPYRIGHT for detailed information.)
========================================================

Merging Message Streams
=======================
December 17, 1993
Takashi Chikayama (ICOT)

This document describes the stream merger provided as a standard
feature of the KLIC system.


1. Introduction

A stream merger is a process that takes input multiple message streams
represented as lists of messages, and passes all the message from all
the input streams to a single output stream, also represented as a
list.

The output consists of all the messages in the inputs with duplicates
preserved.  When two messages are ordered in one of the input streams,
their order is also preserved in the output.  When messages are from
different input streams, their order in the output is unpredictable.
The order may differ in one execution of the same program from
another.  The behavior of mergers is thus nondeterministic.

For example, when there are two input stream, [1, 2, 3] and [a, b, c],
the output can be something like [1, 2, a, b, 3, c] or [1, a, 2, b, c,
3], but will never be [1, a, 3, b, c, 2].


2. Defining a Binary Merger in KL1

A binary (two-input) stream merger can be defined in KL1 as follows.

    merge([M|In1],In2,Out) :- Out=[M|OutT], merge(In1,In2,OutT).
    merge(In1,[M|In2],Out) :- Out=[M|OutT], merge(In1,In2,OutT).
    merge([],In2,Out) :- Out=In2.
    merge(In1,[],Out) :- Out=In1.

- The first clause forwards one message coming from the first input
  stream given as the first argument to the output stream given as the
  third argument, and recurively calls the predicate merge for
  repetitive execution.

- The second clause does the same for the second input stream.

- The third claue is used when the first input stream has no more
  messages in it.  In this case, the second input stream is directly
  connected to the output.  As there are no more messages to merge
  from the first input stream, the result of the merging should be the
  same as the second input stream.

- The fourth clause provides the corresponding feature for when the
  first input stream has no more messages in it.

When messages come from both the first and the second at the same
time, either the first or the second clause is arbitrarily choen.
This is the source of the nondeterminacy of the merger.

Although binary mergers are easy to define in KL1, it is not so easy
for mergers with arbitrarily many input streams.  It is also desirable
to add new input streams dynamically, which makes it still harder.
Also, mergers are used quite frequently in KL1 programs and thus
should be quite efficient.  Thus, the KLIC system provides a merger as
its standard feature, which is decribed below in this document.


3. Creating a Merger Process

A new merger can be created by the following predicate.

 generic:new(merger,Input,Output)
	A new merger with single input stream is created.
	Its input stream is Input and its output is Output.

The merger process thus created is not actually do any merging
immediately after its creation.  It only forwards the messages from
Input to Output, without changing the order.

The merger process will start merging when more input streams are
added, using the feature described in the next section.


4. Adding and Deleting Input Streams

To add a new input stream to a merger, unify the input with a vector
whose elements are input streams.  For example, if you need a binary
merger, do the following.

	generic:new(merger,Input,Output), Input={In1,In2}

This means the same as the following.

	generic:new(merger,{In1,In2},Output)

After doing the above, the merger will merge messages from two input
streams, In1 and In2, to the output stream Output.

Input streams to a merger can be added not only immediately after its
creation but at any time on demand.  Two more input streams are added,
for example, by the following.

	In2 = {In2A,In2B,In2C}

After this, the merger will have four input streams, In1, In2A, In2B
and In2C.

When one of the input streams is no longer needed, that input stream
can be simply closed, by unifying it with an atom [].

The size of the vector unified with an input stream can be arbitrarily
large or small.  When it has only one element, the number of input
streams will not be changed.  When the vector has no elements,
unifying with it has the same effect as closing the stream.

The output stream will be closed, i.e., the tail of the output list is
unified with [], when all the input streams have been closed.


5. Some Clues

- Messages merged can be a data structure containing unbound
  variables.  Such messages are sometimes called "incomplete
  messages".  Incomplete messages are convenient for constructing a
  server-client process structure.  Giving values to variables in
  messages can be used for communicating backward from the server to
  the client.

- Merging may look deterministic on sequential implementations.  Do
  never rely on it.  It will become really nondeterminitic on parallel
  implementations.


6. Bugs

- When a input stream is discarded without being closed, the merger
  process will wait for messages from the stream forever, making the
  whole program unable to terminate.  It may be hard to know which
  process discarded the input stream.
