CONTENTS
OVERVIEW
----- SOURCE FILES
----- OBJECT FILES
----- EXECUTABLES
----- PROTECTIONS
MAKEFILES AND GMAKE BASICS
----- gmake library
----- gmake
EXTRA MAKEFILE GOODIES
----- gmake t=debug
----- gmake t=fast
----- gmake t=profile
----- gmake t=matlab
----- Zapping and Cleaning
----- private_sources
----- user_flags, warn_flags, user_libs
----- gmake backup
----- gmake grep
----- gmake TAGS
ADDING/DELETING/MOVING A SOURCE CODE LIBRARY
----- Moving libraries painlessly
DOCUMENTATION
OTHER THINGS THAT MOORE LAB FOLKS SHOULD KNOW
----- mailing list
----- `loads`
The source code tree under /afs/cs/project/learn/group is intended to make it easy to (1) build C code libraries which can be used and shared by our group, and (2) compile executables for multiple platforms.
IMPORTANT: I recommend making a symbolic link called "g" from all of your home directories to this directory. Do this with the command:
ln -s /afs/cs/project/learn/group ~/g
The rest of this document refers to the group directory by the name ~/g.
IMPORTANT: We're switching to CVS. This means you in general will NOT be working under the ~/g directory but rather in your own directory. Be sure to read the CVS pages.
A code library is a group of related source files stored in their own directory anywhere in AFS. Our group directory ~/g keeps a symbolic link pointing to each actual source directory, so we can refer to them like this no matter where they actually reside:
~/g/utils - Andrew's collection of useful utilities
~/g/fit - Justin's collection of function approximators
~/g/dset - Andrew, Brigham and Mary's datset structure
The source files in these subdirectories should be modified by their owner(s) only.
In addition, each source directory has a symlink called ".g" which points back to ~/g. All these links and permissions are set up automatically when you use "mkdirlib" to create a new library (see below).
All object files from a library live in subdirectories of the source directory. The subdirectories have names like this:
damut/SPARC.debug/
damut/SPARC.fast/
damut/SPARC.profile/
damut/SPARC.matlab/
damut/PMAX.matlab/
damut/LINUX.fast/
damut/HP.profile/
damut/ALPHA.fast/
etc.
These subdirectories are created on-the-fly by gmake, when needed, so there typically won't be all 5*4=20 possible subdirectories present.
Unlike the source directories, these directories are writeable by any of us, which allows us to recompile each other's libraries whenever necessary.
Each code library may also generate a single executable program, which is given the same name as the directory. This executable is built from its corresponding library, additional sibling libraries if needed, and a single additional source file called main.c, which is presumed to define the main() function. For example, ~/g/rccar/main.c is used to generate the executable program "rccar", a program that tests the car interface.
There are symbolic links to the most-recently compiled version of the executable program from two places: in the source code directory (e.g. ~/g/rccar), and in a special directory of binaries, ~/g/bin . This latter directory is a good one to add to your PATH in your .login file.
All lab members belong to the AFS group, jab:moore.lab . To check, use the command
pts group jab:moore.lab
This group has full write access (rlidwka) to ~/g and all object
directories, and read access (rla) to all the source directories ~/g/*
.
Only the owner(s) of each source directory has write access to it. This is so you don't change anything by accident in the directory. If you need to write into someone else's source directory in an emergency, then you *do* have the power to give yourself write access to the directory using the command "fs sa ~/g/DirName <userid> write", but you should tell the owner when you do this, and when you're done, take away your special rights with "fs sa ~/g/DirName <userid> none". You'll still be left with your normal group rights.
This whole scheme is held together by the magic of the "make" utility--or, specifically, "gmake" (GNU's machine-independent version of make). You must use gmake rather than make!
In each code library directory, you need only specify a very simple Makefile giving the name of the library, lists of the source and include files, a list of any sibling directories which should be searched for header files, and a command to include a file called .g/Make.common. Here are the complete contents of the file ~/g/rccar/Makefile:
here = rccar
includes = rccar.h lowcar.h serial.h
sources = rccar.c lowcar.c serial.c smem.c keyboard.c
siblings = fit gmbl kdtr damut
include .g/Make.common
With this Makefile in place, the two main gmake commands you need to know are "gmake library" and simply "gmake".
This will compile any sources which need compiling, and then package up all the object files into an archive library file. For example, in the rccar directory, on a Sparc, "gmake library" produces the file '~/g/rccar/SPARC.debug/librccar.a'.
The command "gmake" by itself is used to generate an executable file. First, this will recursively execute the command "gmake library" in each of the sibling directories. Then it will build the current library. Finally, the additional source file 'main.c' is compiled, and the executable program is generated by linking the sibling libraries, the current library, and main.o.
The executable program has the same name as the library. Links to the program can be found in ~/g/bin, and also in the source directory.
A note of caution: the sibling libraries are linked into the executable in exactly the order given here. It seems that sometimes this ordering is significant! Try to put the most basic libraries (e.g. utils) on the right-hand end of the line.
There are a number of options to gmake which you may find useful. Here are some sample commands:
gmake t=debug
(same as default gmake)
builds executable, uses -g flags allowing debugging with gdb.
Object files go to the YOUR_ARCH.debug subdirectory.
gmake t=fast
builds executable, uses -O2 -DAMFAST flags for optimizing
Object files go to the YOUR_ARCH.fast subdirectory.
gmake t=profile
builds executable, uses -pg -O2 -DAMFAST flag for profiling with gprof.
Object files go to the YOUR_ARCH.profile subdirectory.
Here's how to profile your C code if you work in the ~/g land:
1. Recompile like this: gmake t=profile
2. Run the new executable: ./my_prog When it completes, there will be a file called gmon.out in the directory.
3. Profile like this: gprof my_prog > /tmp/my_profile
The output has three sections: a call graph profile, a flat profile, and an index. It's self-documenting and very handy!
gmake t=matlab
builds executable & matlab interface, uses -DMATLAB_COMPILE -O2
Object files go to the YOUR_ARCH.matlab subdirectory.
A MATLAB executable can be made by the following command in a source directory:
gmake matlab
It creates a file with a .mex extension to it and places it in the g/bin
directory.
In order to execute it from MATLAB you should add the following line to your
.login or .cshrc:
setenv MATLABPATH '/afs/cs/project/learn/group/bin'
Using gmbl2 as an example, the routine can be called through MATLAB by:
gmbl2('xsch alpha.mbl l90:000 t9')
which would execute the program exactly as if it had been run from the
unix command line. If you want to pass a MATLAB matrix in directly rather than
having it read the disk file, you could do one of the following:
gmbl2('xsch',[1 2 3; 3 4 5; 3 4 3; 3 9 9],'l90:00 t9')
a = [1 2 3; 3 4 5; 3 4 3; 3 9 9];
gmbl2('xsch',a,'l90:00 t9')
There are remaining problems. The worst problem is that all malloc'ed memory remains throughout the MATLAB session. am_malloc and am_free are not enough to truly give the memory back over several calls from MATLAB. am_exit and am_free_to_os partially fix the problem. The latter gives back all the memory associated with the free lists, but it doesn't know about any am_malloc'd, but not am_free'd memory. This is not a problem for any execution that makes it to (and passes) the am_malloc_report(), but aborted executions on my_error() or hitting 'q' at a wait_for_key() still leaves lots of memory lying around. Finally, we have a mechanism to pass MATLAB matrices directly into the software, but no method to retrieve them (there is a way, just not implemented by me yet)
Sometimes, you may want to delete a library's object files ( *.o and *.a ) to force your program to be recompiled. The Makefile provides four commands to do these deletions automatically: "gmake zap", "gmake zapsibs", "gmake clean", and "gmake cleansibs".
"gmake zap" -- removes *.o files for current lib, current arch
"gmake zapsibs"-- removes *.o for current lib & sibling libs, current arch
"gmake clean" -- removes *.o files for current lib, all arches
"gmake cleansibs"-- removes *.o files for current lib & sibling libs, all arches
It's also possible to target a particular sibling(s) for zapping or cleaning. Use e.g. "gmake gmbl.zap fit.zap".
It is possible to specify additional source files which, like main.c, will be compiled and linked into the directory's executable file, but will *not* be included in the library archive which siblings can use. To do this, add a line like this:
private_sources = foo.c bar.c baz.c
to your Makefile. (Do not include main.c in this list.)
user_flags, warn_flags, user_libs
If you need to specify additional compile options beyond the defaults generated by t=debug, t=fast, t=profile, and t=matlab (as described above), you may call gmake like this:
gmake user_flags=-DDERBYSHIRE
Alternatively, just add the line "user_flags:=-DDERBYSHIRE" to your
Makefile, before the `include .g/Make.common' line.
Another variable, warn_flags, is set by default to have gcc produce helpful warning messages. To disable this, compile using the command:
gmake warn_flags=
Other variables used in Make.common that you may wish to override include user_libs and extra_link_flags. Poke around in Make.common to figure them out. :-)
To take a "snapshot" of the current sources, includes and Makefile in the current library, use the special command
gmake backup
This will create (if necessary) a subdirectory called backup, and then
put into that directory a compressed tar file containing all the
relevant source and documentation files. The tar file will be named
by the current date, e.g. 95-01-14.tar.gz . Thus, if you make
multiple backups on a single day, they will overwrite one another.
If you need to later restore files from a backup archive, you can use a command like this:
zcat backup/95-01-14.tar.gz | tar xvf - ttt.c main.c
Don't forget to periodically delete stale backup files, to save disk space.
Another handy feature: if you want to search for a particular string (say, "foobar") in all your program's source and include files-- including the ones in sibling directories--then you can use the command:
gmake grep string='foobar'
From the GNU documentation: ``A "tag table" is a description of how a multi-file program is broken up into files. It lists the names of the component files and the names and positions of the functions in each file. Grouping the related files makes it possible to search or replace through all the files with one command. Recording the function names and positions makes it possible to use the `Meta-.' command, which finds the definition of a function without asking for information on the file it is in.''
For example, type:
gmake TAGS
Now, within emacs, visit, for example, main.c. Do "M-x tags-search RET dym_sum RET y". If all works fine, you will be transported to the file .g/utils/amdym.h to the line where this function is declared. (I am assuming that "utils" is one of the "siblings" in your Makefile). Multi-file search-and-replaces also work. See the XEmacs "info" page about tags.
The gmake magic considers all "sources" and "private sources" as indicated in your Makefile, as well as all *.h (but not *.c) files in any of your "siblings" directories. You can do a "etags -a" if you want more files thrown in (eg., the *.c files in siblings).
You should "gmake TAGS" every time there is a big change to the set of tags (function names, typedefs, etc.). gmaking TAGS every compilation is clearly overdoing it.
ADDING/DELETING/MOVING A SOURCE CODE LIBRARY
The `mkdirlib' script should be used to generate a new source library. It lets you choose which AFS volume to mount the new directory on. In the ~/g directory, type:
mkdirlib my-library-name
and you'll get an explanation of how to specify the volume using a 2nd command-line parameter.
You can use `rmdirlib library-name` to delete a library. (It's so easy, you hardly need the script.)
I've written a little script, move_dirlib, which automatically moves a dirlib (and all of its subdirectories except for old object files) to a new AFS volume. Use it like this:
cd ~/g
movedirlib my-lib-name learn-##
This renames the old my-lib-name library to my-lib-name.space-waster After you've tried out a gmake in the new my-lib-name directory just to make sure everything still works, delete the space-waster directory to reclaim the disk space.
If your library will be used by others, please document its usage in a 'README' file in that library's directory.
There is also a file called ~/g/INDEX , where we keep quickie keyword descriptions of the useful functions from all the libraries. Search this file if you're looking for a routine to do a particular thing. And feel free to add your own descriptions!
Finally, I like to keep a running log of the code I've written and experiments I've run in a file called "NOTES" in each library directory.
OTHER THINGS THAT MOORE LAB FOLKS SHOULD KNOW
Our group mailing list is stored in the file ~/g/misc/moore-lab . Any of us can edit the list, but please let Justin and Andrew know when you do.
To send email to everyone on the list, address your email to: moore-lab@cs.cmu.edu
The `loads` command will show the current CPU and memory usage of each of the lab machines. The full pathname is ~/g/bin/loads . Machines near the top of the list, and machines where nobody is logged in at the console, are your best bet when starting a big job.