Python Libraries

What are they?

One of my main functions here was to write libraries that support Nan and Narayanan, or my own research scripts. Whenever a piece of code is going to be used twice its worth making a module, and sometimes when it's only being used once. In general, all of our text processing and database, and some algorithmic work, is done in python.

Code Tree

/net/fugu/usr0/tools/pylib/-- This is what goes on the python path. /algo -- Code to perform general computational algorithms (see below) UnionFind -- module to execute UnionFind on set, based on CLR /Graph -- Graph algorithms UndirectedGraph -- Graph object with algorithms taking advantage of edge symmetry. (deprecated) DirectedGraph -- Graph object with graph algorithms based on CLR /BLAST -- Code wrapping common BLAST functions __init__.py -- Contains BLAST constants and command line formatting DB -- DBProfile class definition, code wrapping formatdb Results -- Alignment class for alignment data, Results class for parsing BLAST output /CDART -- Code wrapping CDART database objects (incomplete) PssmHit -- Represents object in "hits" table (incomplete) /DurandLab -- Code specific to our computing environment or DurandLab database __init__.py -- Constants and functions (pushConn, popConn) for connecting to database (currently motif), and determining paths usable throughout the lab ProteinSequence -- Contains ProteinSequence class, representing entries in DurandLab.prot_seq table and provides format translations ProteinSequenceSet -- Contains ProteinSequenceSet class, representing sets of entries defined in DurandLab.prot_seq_sets and DurandLab.prot_seq_set_membership ProteinArchitecture -- Code for discerning domain architecture based on CDART data (experimental) /ensembl -- Functions for retrieving information from ensembl's public MySQL server. /FASTA -- Functions for parsing FASTA files FastaSeq -- Contains FastaSeq object representing one sequence and header FastaFile -- Contains File-like object FastaFile for parsing entire file /SwissProt -- Contains SwissProt object for wrapping SwissProt DB entry and SwissProtFile object for parsing SwissProt flatfiles. /util -- general purpose code utilities arg_stream -- useful for creating files that read parameters from cmd line or stdin distinct -- removes duplicates from a list exec_cmd -- runs commands locally or remotely and returns result list2dict -- turns 2 lists to a dictionary Dummy -- empty class that you can assign attributes to eof -- determines if a file has reached end histogram -- divides list into buckets and counts (deprecated, will be improved) TablePrinter -- Prints columned output

Standard Operating Procedures

Find out what a module does.
Simply fire up the python interpreter, import the module, and run help([module]). A "man" style interface will describe purpose and function of the libraries.
Use RCS to Manage Version Control
At points where code is changing slowly and being used often, it's useful to use RCS for version control even if you are the sole programmer. Many of the current modules contain "RCS" directories where I've done this in the bast. They are not up to date, so if you decide to use RCS make sure to check in the latest versions before checking anything out. You can read about rcs with man rcs. In general, you must keep a seperate RCS subdirectory as a repository for each directory you want to version-control.
Create a new module
There are a lot of design decisions when writing a new python module. I'll try to walk through some of them.
1. Decide where in the above tree the module fits. If possible, put it in an existing category. If you want to support modules, create your module as a directory rather than a single file. Make your directory, and place a file __init__.py in the directory. This will be the module itself, the code executed when the module is imported. Any other files placed in the directory will be submodules which can be imported explicitly.
2. Decide how users are going to import your code. Python supports two import paradigms:
  - import [module] makes the object or function (functions are objects!) available to a program as module.object.
  - from [module] import [resource] puts an object or function directly into the current namespace.
  The former is less confusing to a reader, but can lead to tediously long function calls and object names in the case of submodules. If you've seperated code into a new file that really belongs in the parent module, you might have the parent do a from ... import call to make that resource available in a standard way earlier in the module tree. Then users can use the standard interface and won't accidentally screw up their namespace.
3. The first """ ... """ string in a module or function is used as documentation when a user calls "help()" on your module. Use this A LOT! It'll save time answering questions, or trying to remember what your own code does.
4. Remember, a module is not just code but is actually executed at startup. This can be used to initialize resources that will be used throughout the module. The best example of this is with database connections, as described below.

Use Database interaction in a module
Managing database connections can be tricky if you're writing functions that access the database. MySQL can support around 50-100 connections, but will then refuse added connections even if the old ones are sitting idle. The solution used in the lab is to have modules accessing the database create one connection on startup, and reuse that connection throughout the functions. This is not thread-safe but is otherwise ok. The following code demonstrates how:

import DurandLab db = DurandLab.getConn(dbName = 'DurandLab') db_auth = 0 c = db.cursor() def authenticate(uname, passwd) : global db, c, db_auth db.change_user(uname, passwd) db.select_db('DurandLab') c = db.cursor() db_auth = 1

As you can see, the DurandLab module wraps the database connection to make changing our db config easier. A single db object is initialized for the whole module as a global variable, and a cursor is created. The authenticate() function allows the user to input a username and password to gain write access if they are calling functions which require write access. Any functions that do require such should check the db_auth variable and throw an error if it's not on. Finally, if your functions call another function that requires authentication, make sure your authenticate function authenticates that module as well!!

back home
bobsedge@andrew.cmu.edu