NetBSD and the Intel ATM Driver

  • Getting Started
  • Switch Operations
  • Performance Monitoring
  • Using Credit
  • Debugging
  • Booting an Alternate Kernel
  • Bug List
  • Future Plans
  • Random NetBSD Details
  • VCNectar Machine Configurations
  • VCNectar Switch Port Configurations
  • Other related WWW sites

    • Getting Started --- IJET Basics

    Directory Organization

    On the PCs, the kernel sources should be available in /big/netbsd-sys/1.1/ijet-sys-stable. The IJET driver is found within this tree in /big/netbsd-sys/1.1/ijet-sys-stable/dev/pci/jetland. At some point it may move into the i386 specific sub-tree. Kernel config files are in /big/netbsd-sys/1.1/ijet-sys-stable/arch/i386/conf and the compilation directories are /big/netbsd-sys/1.1/ijet-sys-stable/arch/i386/ compile/KERNEL_NAME.

    Loading a Kernel

    Assume you want to run the kernel in /big/netbsd-sys/1.1/ijet-sys-stable/ arch/i386/compile/IJET.N48.CREDIT/netbsd, the steps are:

    Building a Kernel

    We are running NetBSD -current. To build a new kernel, the steps are as follows:

    Driver Utilities

    The directory /big/netbsd-sys/1.1/ijet-sys-stable/dev/pci/jetland/france is the current location of the driver utilities. This directory is located in the kernel source tree because it is highly dependent on some of the driver include files which currently change on a frequent basis.

    Assuming you're running a standard IJET kernel and you want the default setup, the only command you should have to issue is `hostname`.up; where hostname is the simple hostname of the machine you are on. The following commands are executed by this file:

    % ./debugmask 0
    debugmask takes as an optional single argument a string consisting of the various debuglevels you wish to make active. The debuglevels are [0-5][a-z]. The argument -all may also be used. A null argument turns off all debugging. See the debugging section for more details.
    % ./loaducode
    simply loads the microcode contained in ucode.dat into the adaptor. This is a mandatory step.
    % ./newsetup
    Registers some configuration variables with the driver. The number of RxTx VCs, message buffers, etc can be set. This is also a mandatory step prior to ifconfig'ing the device up. This will probably be renamed back to setup in the near future.
    % /sbin/route delete -net 192.80.210
    Solely as a precaution that the net has not been previously registered. In the past, the switch has been broadcasting itself as a gateway for this network; which prevented us from registering any other devices on that net.
    % /sbin/ifconfig ijet0 some_address -arp
    Finally tell the driver to register the device as up. Currently, we can not configure the device down. This will hopefully be resolved soon.
    Setarp.c contains the various mappings from IP addresses to VCid. This should probably be read in via some file. Currently, our mapping of IP addresses to machines is done by using the subnet 192.70.210 and appending the final octet of the host's ethernet IP address. For example, hephaestus has the normal IP address of 128.2.250.45, so its ATM IP address is 192.70.210.45. Obviously, should we get addresses that conflict, we'll have to move to a different scheme.

    Switch Operations

    We currently have two different switches. A Fore switch and a BNR switch. I'll describe how to set up each of these.

    BNR Switch

    The BNR switch is known as Kato. Telnet'ing to kato will put you at a command prompt. The following commands and command scripts are typical:
    cd "/afs/cs/project/vcnectar/harvard"
    Most of the command scripts are located in this directory.
    < loadup.scr
    This script takes some time to run as it performs diagnostics on each of the port cards it finds.
    vc2 3, 45, 0, 4, 68, 0
    Creates a bi-directional VC from port 3, VC 45, no flow control to port 4, VC 68, no flow control.
    vc1 3, 45, 0 3, 45, 0
    This would just create a loopback VC on port3.
    lsvcs
    Shows the status of all the created paths.

    To turn off credit cells on a port use "nofc portNo" at the prompt and turn credit on use "yesfc portNo". Note, our driver does not currently issue credit, so we should set the switch up to send out credit cells but not to expect them.

    The correct way to make a loopback VC with flow control only on the link to the switch, one would use 'vc1 3, 45, 1, 3, 45, 0'. Similarly you should use the 'vc1' command twice if you want a bidirectional VC between two machines with flow control only on the input link to the switch.

    FORE Switch

    The FORE switch is known as Boopsie. Telnet'ing to boopsie will bring up the login prompt. The login name for administration and configuration is asx and the password is coreyisgod. Login will put you at a command line interface called AMI (ASX - Management Interface). The following points are general to the command line interface:
    Typing '?' at any prompt will display a list of commands at that level.
    Typing 'help command' will display help on that command.
    Type 'up' to go up the menu one level.
    Commands with a '>' at the end indicate a sub-menu. Typing the command name will take you down to that menu.

    Setting up PVCs

    To setup pvcs you have to be at the 'configuration:vcc>' menu. The command syntax to setup pvcs is:

    new iport ivpi ivci oport ovpi ovci

    For example to set up a pvc between ports a1 and a2 with vpi, vci = 0, 50, you would say 'new a1 0 50 a2 0 50'. This only creates a uni-directional pvc. You have to explicitly create another pvc in the other direction using the command 'new a2 0 50 a1 0 50'.

    There are other optional parameters that you can give to the command used to create pvcs. They are used to specify peak bandwidth, cdvt, and tag/drop policy. Typing 'new' at the configuration:vcc> menu without any options brings up the list of options.

    To delete a pvc use the command:

    delete iport ivpi ivci oport ovpi ovci

    Turning signalling ON/OFF

    The following commands can be used to turn signalling on/off:
    Typing delete port vpi at the configuration:spans> menu would turn SPANS off on that port. Vpi is typically 0.
    Typing new port vpi at the configuration:spans> menu would bring up SPANS on that port. Vpi is again typically 0. There are other optional parameters like AAL4/5, cdvt and tag/drop.
    Typing delete port vpi at the configuration:uni30> menu would turn UNI signalling off on that port. Vpi is typically 0.
    Typing new port vpi at the configuration:uni30> menu would bring up UNI on that port. Vpi is again typically 0. There are other optional parameters like ILMI, AAL4/5 etc.

    Performance Monitoring

    The BNR switch supports monitoring the performance of the VCs. There are several things that one could monitor including bandwidth, queue size, credit received from the downstream node, etc. These performance measures are displayed on a real-time X display on the monitoring workstation. The architecture of the monitoring software provided by Harvard is like this: A real-time X display program binds to a well-known UDP socket and recieves measured samples from the switch and displays it on the workstation. The switch has to be configured with the IP address of the workstation and also the VC parameters that have to be monitored.The following explains how to setup the workstation and the switch for monitoring.

    Monitoring workstation setup

    You need to do the following to startup the display program on the monitoring workstation

    Switch setup

    You need to do the following to startup the monitoring software on the switch

    Monitoring options

    The following monitoring commands are available to graph various parameters. Note that the "ivc" is the internal VC number of the switch. This can be obtained by typing "lsvcs" at the command prompt. Also note that the meaning of the parameter "port" is different for different graphing commands --- for some it is the input port and for some it is the output port of the same ivc. The "slowdown" is a number in the range 0 ... 31, which specifies the number of samples to skip over before taking a measurement and "avg" is the number of measurements to average before transmitting the result. Values of 7 for slow down and 10 for avg work best for us.

    Debugging

    Currently there are two basic mechanisms that are employed to facilitate the debugging process. The first is the standard kernel printf() function. When printf() is called within the kernel it prints the message to the console, and additionally writes it to /var/log/messages. However, the internal console buffer is only 4k, so if the message isn't written to the file soon enough, information is overwritten.

    The second method of writing debugging messages makes use of a much larger internal circular buffer. Messages are written to this buffer, and read by a user level process. An example would be:

    The -f means to go ahead and wait for messages to arrive in the buffer; printing them out as they occur. The debugmask is a bitmask specifying which debug messages you're interested in. Within the kernel, these various debug levels are denoted by using the macros debug[0-5]() and debug[a-z](). The debugmask program sets the appropriate kernel logging mask. For example, debugmask 34 would turn debugging on for events 3 and 4 only.

    Booting an Alternate Kernel

    If you've built a new kernel and you find it doesn't boot properly, there are a couple of methods for fixing the problem.

    Hopefully, you've saved a working kernel under another name (e.g., /netbsd.OLD). If so, you should be able to interrupt the boot process by hitting any key just after the boot blocks have been loaded. Unfortunately, since these machines are so fast, the timing loop that waits for input is very small. Hopefully this will be fixed someday. If the boot process halts, you can then just type the name of the alternate kernel.

    If you can't seem to get the boot process to halt, or if you haven't saved a stable kernel elsewhere, you have to place a new kernel in the root partition from a fixit floppy. Currently, I have a copy of the fixit floppy in /usr/distrib/floppies on the host marilyn. The file kernel.fs is a filesystem image...to place it onto a floppy use the following command.

    You should then be able to boot with this floppy. Just hit return at all the prompts, until you reach a shell prompt. The command copy_kernel should install a new kernel on your root filesystem. On all our existing machines, the root filesystem is on disk wd0. In particular, it is in the wd0a partition.

    Known Bugs

    An incomplete list of known bugs:
    1. 48*N Bug -- Packets which are a multiple of 48 bytes are not handled properly. They end up merging with the next packet. An End-Of-Pdu bit is not set until a packet arrives which is not a multiple of 48 bytes. Currently, there is a slow software workaround which pulls individual packets out of the merged buffer. However, there is no guarantee that a non 48 byte packet will ever arrive or that we will not run out of resources before it does.

      If we are sending only between PCs which have this bug, we can add a one word header to prevent 48N packets from ever being sent. This does, however, make us incompatible with other interfaces. To enable this feature, define the identifier N48HACK either in the file if_ijet.c or via an options line in the kernel configuration file.

    2. Spurious CRC errors are occasionally seen on the P90's. While some of these are legitimate errors, many are due to problems with the FIFOs overflowing when transmitting/receiving at full rate. Tends to happen more with larger packet sizes. The alphas seem to be able to send back-to-back packets fast enough to cause the adaptor similar problems.

    Future Plans

    Here is a partial list, in no particular order, either of fixes that will be applied to the driver or new features that will be implemented:
    1. Change ijetifioctl() to properly reset the board.

    2. Change credit processing to be on a per VC basis. Currently, either all VCs use credit or none do.

    3. Allow credit to be updated via an ioctl(). Currently, if we run out of credit, we will wait forever for another credit cell to arrive. But the switch only sends out credit after a certain amount of data has arrived. If we just happen to lose those credit cells, the switch does not resend them. A simple ioctl() will allow us to recover from this deadlock.

    4. Prashant is attempting to get Vince to work with our adapter. This requires a different interface than the simple IP model.

    Random NetBSD Details

    Mounting the MSDOS file system

    The DOS partition of the disk is accessible from NetBSD by mounting the filesystem as root with: 
        # mount_msdos /dev/wd0f /mnt
    Upon finishing, you can unmount the volume with 
        # umount /mnt
    making sure that you are not in the /mnt tree or using any files contained therein. Files can be copied back and forth between different filesystems, but certain details have to be remembered. DOS filenames are formatted as 8.3; 8 character primary name w/ a 3 character extension. I believe all filenames are folded to uppercase when copying into DOS, and folded to lowercase when moving to NetBSD. I've successfully untar'd files into a DOS partition, but have had problems when simply trying to rename directories. For this reason, it is probably a wise idea to occasionally run scandisk from within DOS after modifying its filesystem.

    Installing NetBSD

    Random things that probably aren't of much interest to anyone but me. I'll put this in a separate file at some time.
    1. Boot off the kernel floppy. Switch to the install floppy. Run the install program. Regardless of what NetBSD thinks the cyl/tracks/sectors is, use what pfdisk (under dos) says it is. In fact, use pfdisk to set up the NetBSD partition. If the install program asks if you wish to overwrite the partition table, you've screwed up. ^C and try again.
    2. Boot off the kernel floppy. Use this same floppy to copy a kernel onto the disk.
    3. Reboot from the hard disk. Use ifconfig/route to put ourselves on the net.
    4. Ftp a distribution/packages over from some machine that has them. Do Set_tmp_dir, Extract foo.
    5. Get tarpkgs and packages as well.
    6. Change the following files: