Shore-MT


Goal

Implementation of a scalabe storage manager for emerging hardware. Shore-MT is a scalable multi-threaded port of the Shore storage manager.

Details

Database storage managers have long been able to efficiently handle multiple concurrent requests. Until recently, however, a computer contained only a few single-core CPUs, and therefore only a few transactions could simultaneously access the storage manager's internal structures. This allowed storage managers to use non-scalable approaches without any penalty. With the arrival of multicore chips, however, this situation is rapidly changing. More and more threads can run in parallel, stressing the internal scalability of the storage manager. Systems optimized for high performance at a limited number of cores are not assured similarly high performance at a higher core count, because unanticipated scalability obstacles arise.

We benchmark four popular open-source storage managers (Shore, BerkeleyDB, MySQL, and PostgreSQL) on a modern multicore machine, and find that they all suffer in terms of scalability. We briefly examine the bottlenecks in the various storage engines. We then implement Shore-MT, a multithreaded and highly scalable version of Shore which we developed by identifying and successively removing internal bottlenecks. When compared to other DBMS, Shore-MT exhibits superior scalability and 2-4 times higher absolute throughput than its peers. We also show that designers should favor scalability to single-thread performance, and highlight important principles for writing scalable storage engines, illustrated with real examples from the development of Shore-MT.

Download

Recently we released Shore-MT. It can be downloaded from here.

Related Documenation

[VLDB09]

Improving OLTP Scalability using Speculative Lock Inheritance

  Ryan Johnson, Ippokratis Pandis, and Anastassia Ailamaki.
In Proceedings of the 35th VLDB, Lyon, France, August 2009.


[EDBT09]

Shore-MT: A Scalable Storage Manager for the Multicore Era

  Ryan Johnson, Ippokratis Pandis, Nikos Hardavellas, Anastassia Ailamaki and Babak Falsafi.
In proceedings of the 12th EDBT, Saint Petersburg, Russia, 2009.
PDF [340k]


[DAMON08]

Critical Sections: Re-emerging Scalability Concerns for Database Storage Engines

  Ryan Johnson, Ippokratis Pandis, and Anastassia Ailamaki.
In proceedings of the 4th DaMoN, Vancouver, Canada, June 2008.
PDF [251k]


[TR08]

Shore-MT: A Quest for Scalability in the Many-Core Era

  Ryan Johnson, Ippokratis Pandis, Nikos Hardavellas, and Anastassia Ailamaki.
Carnegie Mellon University Technical Report CMU-CS-08-114, April 2008.