% -*-LaTeX-*-
\documentclass[11pt,a4paper]{article}
\usepackage{a4wide}
\usepackage{times}
\usepackage{epsf}

\begin{document}

\tolerance800
\emergencystretch2em
\doublehyphendemerits5000
\hfuzz0pt
\leftskip0pt minus 1pt
\rightskip0pt minus 1 pt

\title{\bf Clockwise for Linux: Principles}

\newcommand{\dynpar}{dynamic partition}
\newcommand{\Dynpar}{Dynamic partition}

\author{Peter Bosch \\
        {\small University of Twente} \\
        {\small Netherlands} \\
	{\small peterb@huygens.org}}

\maketitle
\bibliographystyle{/home/pegasus/peterb/lib/tex/inputs/refplainpacked}

\begin{abstract}
This paper describes the operating principles of Clockwise for Linux.
It describes the data structures that are kept on disk, the data
structures that are maintained in core, it describes the Clockwise
deamons and it describes how to use the Clockwise driver interface
efficiently.
\end{abstract}

\section{Introduction}

Clockwise for Linux is a Linux implementation of the Clockwise
real-time disk server~\cite{Bosch97,Bosch99}.  Its primary task is
provide real-time and best-effort storage facilities to higher level
software on Linux machines.  Applications that typically benefit from
real-time service are, for example, continuous-media applications.
Applications that use the best-effort service from Clockwise for Linux
are for example entire \textsc{ext}2 Linux file systems.

The whole idea of Clockwise for Linux is that it provides a scheduling
and disk management abstraction layer \emph{on top of} existing device
drivers and \emph{underneath} existing file systems.
Figure~\ref{fig:cfl} shows an overview of the system with and without
Clockwise.  On the left hand side of the figure the original Linux
kernel structure is shown: users communicate with a Linux file system
(such as the \textsc{ext}2 file system).  This module communicates
with the block handling inside Linux to retrieve as much data as is
possible from the block cache.  Eventually, a request needs to
serviced on disk and a disk device driver is activated.
\begin{figure}
 \centering
 \epsfxsize=\textwidth
 \epsfbox{cfl.eps}
 \caption{\small{Linux with Clockwise.}\label{fig:cfl}}
\end{figure}

Linux device drivers are started by means of a request queue.  High
level software prepares disk requests and associates a number of
memory buffers with the request.  Next, the high level software
inserts the request into a disk request queue by means of a standard
insertion routine called \textsc{add\_request}.  When a request
finishes, the disk device driver calls a routine called
\textsc{end\_request} to clean up the request and to resume the higher
level software.

Clockwise for Linux adds an extra queuing, addressing and scheduling
layer between the high level block handling and low level disk drivers
as is shown in Figure~\ref{fig:cfl}.  Clockwise for Linux behaves
exactly like a real disk driver to higher level software and it calls
real disk drivers to have them service actual disk requests.  Thus,
Clockwise for Linux does not communicate with any real hardware.

As is also described in~\cite{Bosch99}, Clockwise for Linux implements
\dynpar{}s.  A \dynpar{} is a logical grouping of disk blocks on one
or more disks.  From a user perspective, \dynpar{}s are laid out
consecutively, but in reality blocks can be scattered over many disks.
The storage unit of \dynpar{}s are long ranges of disk sectors
(currently, each \dynpar{} block has a size of 1\,\textsc{mb}).

When a request arrives in Clockwise for Linux, it maps the \dynpar{}
block request to a real disk and disk offset.  Clockwise does this by
maintaining a \dynpar{} table which describes the location and size of
\dynpar{} \emph{dpnode}s.  Each \emph{dpnode} describes the logical
disk numbers and block offsets of each allocated \dynpar{} block.  The
location and size of the \dynpar{} table is stored in each disk's
superblock\footnote{All disks must agree on the location}.  Each
disk's logical disk address is also found in the disk's superblock.

\Dynpar{}s in Clockwise for Linux are identified by the minor device
number that is passed to the Clockwise for Linux device driver.  Since
this an 8-bit value in Linux, Clockwise for Linux can only maintain
256 \dynpar{}s.  However, the last entry is reserved for configuration
purposes, so only 255 real \dynpar{}s can be created at the same time.
Clearly, this amount is limited, and there is an effort underway in
Linux to extend the size of device numbers.  When the minor device
number is extended, more \dynpar{}s can be created simultaneously.

In principle, each \dynpar{} acts much like a real raw disk
partition.  Hence, it can be used to format standard Linux file
systems inside the \dynpar{}s.  To do so, a user first creates an
empty \dynpar{} and publishes the \dynpar{} in the Linux file name
space as a block special device file with the appropriate major and
minor device number.  Next, the user formats a new file system in the
reserved \dynpar{} through the standard \textsc{mkfs(8)} maintainance
utilities.  Finally, the file system can be checked with
\textsc{fsck(8)} and mounted into the Linux name space.  E.g., the
following examples creates a \dynpar{} of a \textsc{gb}, the blocks
are stored on logical disk 0 and 1 and the \dynpar{} can be accessed
from Linux through \/mnt\/usr0.
\begin{verbatim}
unix$ cwreserve -c /dev/clockwise usr0 1024 1 0 3
unix$ cwls -c /dev/clockwise
dpnum            fsize  id   iaddr  isz dpname
-=============================================
[000]       1073741824   0   2048  8192 usr0
unix$ mknod /dev/cwb_usr0 b 60 0
unix$ mknod /dev/cwc_usr0 c 60 0
unix$ mkfs /dev/cwb_usr0
[...]
unix$ mount /dev/cwb_usr0 /mnt/usr0
\end{verbatim}

\Dynpar{}s that are accessed through the Linux file system are
currently always accessed with a \emph{Best Effort} (\textsc{bef})
class of service.  The reason for this is that currently there is no
way to associate QoS settings to a \dynpar{} that is accessed through
the Linux file system\footnote{Also note that this is not a
  fundamental problem.}.

To access a \dynpar{} with a real-time class of service, Clockwise for
Linux needs to be accessed through the raw device interface.  Contrary
to standard Linux device drivers, Clockwise \dynpar{}s can be accessed
in a raw manner.  Internally, Clockwise maps the raw device call to a
block device call, which is serviced by the standard device drivers.
When the raw device is opened, a user can specify a block size and
bandwidth with which the \dynpar{} needs to be scheduled.  The
scheduling principle themselves are described in~\cite{Bosch99}.

The remainder of this paper describes more of the detail of
Clockwise.  In particular, Section~\ref{sec:deamons} describes the
Clockwise deamons, Section~\ref{sec:mm} describes Clockwise memory
management, Section~\ref{sec:video} describes the Clockwise support
for a video file server.

\section{Clockwise deamons}\label{sec:deamons}

As described earlier, Clockwise for Linux implements an extra queuing
and scheduling layer between the Linux file system and the actual
device drivers.  In fact, each disk maintains its own best-effort and
real-time disk queues.

Each real disk is served by a Clockwise deamon.  This deamon is
responsible for driving the real disk through its original disk
driver.  The deamon waits for disk requests to arrive from higher
level software (e.g. the block handling), and when such a request
arrives in one of the aforementioned queues, the deamon picks up the
request and sends it to the real disk driver through its
\textsc{add\_request} function.  Next, it waits for the request's
completion and finalizes the request.

Clockwise for Linux maintains two internal queues: one for best-effort
traffic and one for real-time traffic.  The selection process for
picking up requests follows the $\Delta L$ disk scheduling principles,
which is described in detail in Chapter 5 and 6 of~\cite{Bosch99}.
Roughly, the scheduler selects best-effort requests when there is
slack time in the schedule or when there are no queued real-time
requests (\emph{i.e.}, a best-effort request can be invoked before
queued real-time requests).  Real-time requests are invoked when the
slack time is exhausted or when there are no queued best-effort
requests.

The Clockwise deamons are activated through a user level utility
called \textsc{cwd}.   This utility examines a configuration file and
forks off a process for each disk that is listed in the configuration
file.  The forked off process performs an \textsc{add\_disk} Clockwise
call.  This call only returns when Clockwise terminates.

Once all Clockwise deamons are present, a final Clockwise call starts
Clockwise by performing a \textsc{go} call.

\section{Clockwise memory management}\label{sec:mm}

In contrast to the Nemesis version of Clockwise (see Chapter 5
of~\cite{Bosch99}), the Clockwise for Linux version does not implement
any memory scheduling.  The Nemesis version tries to balance physical
memory usage between a number of competing continuous-media
applications.  Call-backs in applications are used to revoke memory
from Nemesis applications.

There are two reasons why memory balancing has not been implemented in
the Linux version.  The first reason is that Linux does not lend
itself well for a callback mechanism such as is used in the Nemesis
version.  The second reason is more fundamental: one can wonder if
memory management is a task for a disk scheduler.  I know believe it
is not -- a disk scheduler should not be bothered by memory management
issues.  

When applications require real-time service from a disk, it is obvious
they require physical memory to store and retrieve data.  In the
Clockwise for Linux version, applications are required to obtain
physical memory from Linux itself (\emph{i.e.}, by exercising the
\textsc{mlock(2)} system call).

Current Linux implementations do not allow direct \textsc{dma}
transfers to and from physical memory.  Instead, a driver allocates a
\textsc{dma} buffer from Linux and uses the \textsc{cpu} to copy data
into or from such \textsc{dma} buffers.  Clearly this is less than
optimal.  

By fixing the scatter-gather functionality in existing disk device
drivers (most notably the \textsc{ncr} \textsc{scsi} driver) and by
introducing virtual-memory calls to obtain the physical addresses of
application virtual memory, data is directly \textsc{dma}-ed from and
to user memory.  Only when an application has not locked the virtual
memory, and when Clockwise for Linux cannot lock the presented virtual
memory, Clockwise for Linux uses the old \textsc{dma} buffers
structure.

\section{Clockwise video file server support}\label{sec:video}

Clockwise has been designed with the storage and retrieval of
continous media (\emph{i.e.} digital audio and video) in mind.  This
means that it is relatively straightforward to implement a
full-fledged continuous-media server on top of Clockwise.  In fact, a
prototype implementation of such a continuous-media file server
already exist.

Currently, Clockwise is used as a distributed continuous-media storage
server.  Clients on a network render and/or record audio and video,
which they retrieve or forward from a central Clockwise file server.
For each session (a single audio of video stream), a client needs to
start a \emph{servlet} on the file server that takes care of the
recording of data in a \dynpar{} or the playback of data from a
\dynpar{}.

To retrieve data from a remote Clockwise server over the network, one
can use the \emph{pullatm} utility.  This utility listens on a raw
\emph{atm} virtual circuit for client requests and based on the
requests, it sends \emph{chunks} from the \dynpar{} on another
\emph{atm} virtual circuit back to the client for, for example, video
rendering.  The latter is a simple utility that copies the data it
receives on an \textsc{atm} virtual circuit into a \dynpar{}.

\emph{Pullatm} operates through a \emph{playlist}.  A playlist is
a (possibly long) list of entries that describe at which time, what
parts of a \dynpar{} are to be played onto the network.  So, to
playback a video sequence, the playlist typically contains the time
stamps of the individual frames, the offset in the \dynpar{} and the
size of the frame.  \emph{Pullatm} simply traverses the list and when
frames are due, the packets are forwarded to the network.

To make sure that pullatm can send data on time to the client, it
prepares a block preload list.  For this it analyzes the playlist and
releases Clockwise I/O requests in time to make sure that when the
frames are due, the data is available.  In its default configuration,
pullatm preloads at most two buffers.  If, however, there is a lot of
varation on the data stream, a client can instruct pullatm to preload
more than two buffers.

The playlist and data are transmitted over an unreliable
communication's channel (raw \textsc{atm}).  We believe that the only
way to reduce end-to-end latency between server and client and to
reduce the amount of buffering on the client is by using an unreliable
protocol.  Hence, there is often not enough time to try to retransmit
lost data.  This means that the transfer protocols need to be
resilient against data loss on the network.  During a session there
are only two types of data transfers going on: the client sends
playlists to the server and the server sends frames to the client.
Since playlists usually contain a fairly large number of frames that
need to be transmitted, losing one or two playlists every once in a
while is not a problem.  Digitized audio and video that is transmitted
from the server to the client can be lost and may be noticed.  Losing
digital video is usually not a problem -- they eye usually does not
notice\footnote{Except when transmitting data that is compressed by a
  temporal compression algorithm: frames may rely on eachother.}.
Losing audio is more serious -- you can hear it.  One may consider
using something like \textsc{tcp} for audio transmissions, but for now
we have decided to just leave it.

A user who is watching a movie may decide it likes to continue at a
different location in the movie.  To facilitate this, the client's
player (\emph{i.e.}, the application that reads the network and
renders the audio or video locally), can send a new playlist to the
server.  This new playlist replaces the old playlist and pullatm
simply follows the instructions from the new playlist.

Pullatm can be supplied with a number of QoS parameters.  In
particular, pullatm needs to be supplied with a buffer size and
bandwidth.  These parameters are used to request I/O bandwidth from
Clockwise.  Also, pullatm needs to known how many preload or
write-behind buffers it can use.  The default is two.

Currently, pullatm uses raw \textsc{atm} virtual circuits.  This is
far from ideal: one needs to know which \textsc{atm} virtual circuits
belong to which user and transfers.  Ideally, one would use
\textsc{udp} to transfer data.  Unfortunately, the current \textsc{ip}
over \textsc{atm} implementation under Linux is not stable enough.  It
is expected that this will improve quickly.\footnote{Given the rate
 with which message appear on the atm-linux mailing list.}

Ease of use of the system has been improved by using a \emph{broker}.
This broker maintains a list of \emph{(vpi, vci)} pairs for all
clients and servers.  To establish a connection between client and
server, the broker is queried for the already established \textsc{atm}
virtual circuit.

Alternatively, one can use the utility \emph{pull} to initiate
transfers.  This utility reads data from \dynpar{}s and outputs the
data on standard output.  When this utility is combined with, for
example, web-support, remote clients can watch movies across the
network.

Finally, a deamon called the \textsc{vfsd} (\emph{video file server
deamon}) listens to a well known \textsc{tcp} port and is able to
start and control pullatm utilities.  So, to establish a data
transfer, a client first requests the broker for a (vpi, vci) pair,
and with these parameters it contacts the \textsc{vfsd} for service.
During a session a client must maintain contact with the
\textsc{vfsd}: if contact is lost, \textsc{vfsd} terminates pullatm.

\section{Summary}\label{sec:summary}

This short paper has presented some of the Clockwise internals.  For a
comprehensive overview, please refer to my
Ph.D. thesis~\cite{Bosch99}, or visit my web-site at:
http://www.huygens.org/$\sim$peterb/clockwise.html.

\bibliography{refsdb}
\end{document}
