A Content Discovery System (CDS) is a distributed system that enables the discovery of contents. A node in a CDS can publish and provide contents, issue queries looking for contents, store contents or contents' meta-data published by other nodes, and resolve other nodes' queries. There exists a wide spectrum of distributed applications that either themselves are CDS systems or use a CDS as one of their major components. Examples include service discovery services, peer-to-peer (P2P) object sharing systems, sensor networks and publication-subscription (pub/sub) systems.

The primary task of a CDS is to efficiently locate the set of contents that matches a client's query. Existing CDS systems have difficulties in achieving both rich functionality and scalability. At one end, they may be able to scale to the Internet level but offer limited functionality, e.g., they support exact content name lookup [Chord, CAN, Pastry, Tapestry] only, or the search of strictly hierarchical content names [DNS], or they consider static contents only, e.g., search engines [Google]. At the other end, they may offer general searching capability of both static and dynamic contents, but their searching mechanisms are not scalable [Gnutella, KaZaa].

In this project, we design Camel, a distributed and scalable CDS that overcomes the above difficulties and enables powerful content discovery on the Internet. Camel uses a Distributed Hash Tables (DHT) as an overlay network substrate, and possesses the following properties:

• Scalability.
  Camel achieves scalability through the use of Rendezvous Points (RPs), and thus avoids system-wide message flooding at both content registration and query time.

• Load Balancing.
  Camel deploys a novel mechanism that uses Load Balancing Matrices (LBMs) to dynamically balance both registration and query load in a truly distributed fashion to ensure its throughput, even under extremely skewed load, such as flash crowds.

• Rich searchability.
  Camel utilizes a flexible attribute-value based naming scheme for searching, and provides efficient support for complex queries, such as subset matching based queries, range and similarity queries.

Camel is designed as a generic software layer such that high level applications can be built on top of it. We have implemented Camel in a simulator as well as a real Internet implementation. As a proof of concept, we integrated Camel with a content-based music classification engine, and implemented a distributed music information retrieval system.

Please refer to our publications for more technical details.