SDI/ISTC

This talk describes the Splice Machine RDBMS designed to power today's new class of modern applications that require high scalability and high-availability while simultaneously executing OLTP and OLAP workloads. Splice Machine is a full ANSI SQL database that is ACID compliant, supports secondary indexes, constraints, triggers, and stored procedures. It uses a unique, distributed snapshot isolation algorithm that preserves transactional integrity, and avoids the latency of 2PC methods. The talk will also present a variety of distributed join algorithms implemented in the Splice Machine executor and how the optimizer automatically evaluates each query, and sends it to the right data flow engine. OLTP queries such as small read/writes and range queries are executed on HBase, and OLAP queries such as large joins or aggregations are executed on Spark. The system can ensure that OLAP queries do not interfere with OLTP queries because the engines are run in separate processes each with tiered and prioritized resource management. We will also describe a few use cases where Splice Machine has been deployed commercially.

Monte Zweben is CEO of Splice Machine -- maker of the first dual-engine RDBMS on HBase and Spark. Monte's early career was spent with the NASA Ames Research Center as the Deputy Chief of the Artificial Intelligence Branch, where he won the prestigious Space Act Award for his work on the Space Shuttle program. Monte then founded and was CEO of Red Pepper Software, a leading supply chain optimization company, which merged in 1996 with PeopleSoft, where he became VP and General Manager of the Manufacturing Business Unit.

In 1998, Monte was the founder and CEO of Blue Martini Software -- the leader in e-commerce and multi-channel marketing systems. Blue Martini went public on NASDAQ in one of the most successful IPOs of 2000, and is now part of JDA. Following Blue Martini, he was the chairman of SeeSaw Networks, a digital, place-based media company. Monte is also the co-author of Intelligent Scheduling and has published articles in the Harvard Business Review and various computer science journals and conference proceedings. Zweben currently serves as Chairman of Rocket Fuel Inc. (NASDAQ:FUEL) and serves on the Dean's Advisory Board for Carnegie-Mellon's School of Computer Science.

This talk describes the Splice Machine RDBMS designed to power today's new class of modern applications that require high scalability and high-availability while simultaneously executing OLTP and OLAP workloads. Splice Machine is a full ANSI SQL database that is ACID compliant, supports secondary indexes, constraints, triggers, and stored procedures. It uses a unique, distributed snapshot isolation algorithm that preserves transactional integrity, and avoids the latency of 2PC methods.

The talk will also present a variety of distributed join algorithms implemented in the Splice Machine executor and how the optimizer automatically evaluates each query, and sends it to the right data flow engine. OLTP queries such as small read/writes and range queries are executed on HBase, and OLAP queries such as large joins or aggregations are executed on Spark. The system can ensure that OLAP queries do not interfere with OLTP queries because the engines are run in separate processes each with tiered and prioritized resource management. We will also describe a few use cases where Splice Machine has been deployed commercially.

Monte Zweben is CEO of Splice Machine -- maker of the first dual-engine RDBMS on HBase and Spark. Monte's early career was spent with the NASA Ames Research Center as the Deputy Chief of the Artificial Intelligence Branch, where he won the prestigious Space Act Award for his work on the Space Shuttle program. Monte then founded and was CEO of Red Pepper Software, a leading supply chain optimization company, which merged in 1996 with PeopleSoft, where he became VP and General Manager of the Manufacturing Business Unit.

In 1998, Monte was the founder and CEO of Blue Martini Software -- the leader in e-commerce and multi-channel marketing systems. Blue Martini went public on NASDAQ in one of the most successful IPOs of 2000, and is now part of JDA. Following Blue Martini, he was the chairman of SeeSaw Networks, a digital, place-based media company. Monte is also the co-author of Intelligent Scheduling and has published articles in the Harvard Business Review and various computer science journals and conference proceedings. Zweben currently serves as Chairman of Rocket Fuel Inc. (NASDAQ:FUEL) and serves on the Dean's Advisory Board for Carnegie-Mellon's School of Computer Science.

Faculty Host: Andy Pavlo

This talk will provide an overview of LinkedIn's distributed stream processing platform, including Samza/Kafka/Databus. It will first cover the high level scenarios for stream processing in LinkedIn, followed by detailed requirements around scalability, re-processing, accuracy of results, and ease of programmability; then we will focus on the requirements of stateful stream processing applications and explain how Samza’s state management allows us to build applications that meet all the above requirements. The key concepts, architecture and usage in LinkedIn's stream processing pipeline will be explained, including state management in Samza, the use and configuration of Kafka and Databus as input/output and as a change log.

We will also discuss in detail how we leverage the reliable, replayable messaging system (i.e. Kafka) together with durable state management in Samza to build a Lambda-less stream processing platform. The key mechanism to achieve a unified process model between batch and real-time stream is windowing. We will dive into the requirements and our solutions to windowing a real-time stream in this talk as well.

Yi Pan graduated from UCI with a Ph.D. in Computer Science in 2008. Since then, he has worked in distributed platforms for Internet applications for 8 years. He started at Yahoo! working on Yahoo!'s NoSQL database project, leading the development of multiple features, such as real-time notification of database updates, secondary index, and live-migration from legacy systems to NoSQL databases. Later, he joined and led the development of the Cloud Messaging System, which is used heavily as a pub-sub service and transaction log for distributed databases at Yahoo!. Since 2014, he joined LinkedIn and quickly became the lead of the Apache Samza team at LinkedIn, which provides a scalable stream processing service for the whole company. >

Faculty Hosts: Majd Sakr, Garth Gibson

Partially funded by Yahoo! Labs

Today's cloud computing infrastructure requires substantial trust. Cloud users rely on both the provider's staff and its globally-distributed software/hardware platform not to expose any of their private data. We introduce the notion of shielded execution, which protects the confidentiality and integrity of a program and its data from the platform on which it runs (i.e., the cloud operator's OS, VM and firmware). The talk presents two prototype systems that allow applications to run in the cloud without having to trust it.

Haven is the first system to achieve shielded execution of unmodified legacy applications, including SQL Server and Apache, on a commodity OS (Windows) and commodity hardware. Haven addresses the dual challenges of executing unmodified legacy binaries and protecting them from a malicious host. VCCC is the first practical framework that allows users to run distributed MapReduce computations in the cloud while keeping their code and data secret, and ensuring the correctness and completeness of their results. Both systems leverage the hardware protections of Intel SGX to defend against privileged code and physical attacks such as memory probes.

Marcus Peinado is an Architect in the Platform Infrastructure Group at Microsoft Research, Redmond. His interests include Operating Systems, Trusted Computing and System Security. His past and current projects in these areas include Haven, VCCC, Hyper-V, Windows Media security, Controlled Channel attacks and the MAS rootkit detector. Marcus holds a Ph.D. from Boston University.

Faculty Hosts: Majd Sakr, Garth Gibson

User-facing, latency-critical services account for a large fraction of the servers in large-scale datacenters, but typically exhibit low resource utilization during off-peak times. An effective approach for extracting more value from these servers is to co-locate the services with batch workloads. Thus, in this paper, we build systems that harvest spare compute cycles and storage space from datacenters for such batch workloads. The main challenge is minimizing the performance impact on the services, while being aware of and resilient to their utilization and management patterns. To overcome this challenge, we propose techniques for giving priority over the resources to the services, and leveraging historical information about them. Our results characterize the dynamics of how services are utilized and managed in ten large-scale production datacenters. Using real experiments and simulations, we also show that our techniques eliminate data loss and unavailability in many scenarios, while protecting the co-located services and significantly improving batch job execution time.

BIO: 
Marcus Fontoura is currently a Partner Architect at Microsoft, where he works on the end-to-end architecture for Azure compute. In his previous roles at Microsoft, Marcus worked on the production infrastructure for Bing and in several Bing Ads projects. Prior to Microsoft, Marcus was a Research Scientist at Google (2011-2013) where he worked in the Search Infrastructure team. His focus was on the serving systems powering Google.com search. At Google, Marcus worked in many projects including performance and scalability of retrieval engines, novel compression schemes, indexing systems, and networking. He also worked in retrieval techniques for large-scale machine learning systems.

Before joining Google, Marcus was a Research Scientist at Yahoo! Research (2005-2010) working on several advertising projects. Marcus also worked as the architect for a large-scale software platform for indexing and content serving, which is used in several of Yahoo!'s display and textual adverting systems. Yahoo! Superstar and was awarded with two Yahoo! You Rock awards. Prior to Yahoo!, Marcus worked at the IBM Almaden Research Center (2000-2005), where he co-developed a query processor for XPath queries over XML streams. This was one of key components of the implementation of the XML data type in the IBM DB2 Relational Database System. In another project at IBM, he was one of the key researchers developing an Enterprise Search Engine that resulted in a new software product for IBM - the IBM OmniFind Enterprise Search.

Marcus finished his Ph.D. studies in 1999, at the Pontifical Catholic University of Rio de Janeiro, Brazil (PUC-Rio), in a joint program with the Computer Systems Group, University of Waterloo, Canada. The main contributions from his Ph.D. thesis have been condensed in the book The UML Profile for Framework Architectures, published by Addison-Wesley in 2001. After finishing his Ph.D. Marcus was a post-doctoral researcher in the Computer Science Department at Princeton University for one year (1999-2000). Marcus is an ACM Distinguished Member and an IEEE Senior Member. He has more than 25 issued patents and more than 50 published papers. Marcus has been in several program committees over the years, including SIGIR, WWW, WSDM, KDD, and CIKM. Recently Marcus was a co-chair of the WWW 2013 developers track.

Faculty Hosts: Garth Gibson, Majd Sakr

Aramid is a new cluster resource reservation system for datacenters with heterogeneous resources and interconnects, enabling predictable performance for deadline-driven production jobs and services sharing a cluster with best-effort jobs. To do so, Aramid combines a declarative language that explicitly describes job-specific heterogeneity preferences with novel reservation and scheduling algorithms. Specifically, Aramid’s Heterogeneous Reservation Definition Language (HRDL) allows users to request time-varying capacity reservations across heterogeneous resource collections. But the dynamic nature of cluster environments creates a challenge. Estimated job run times are often inaccurate. Cluster machines may reboot and dynamically acquire new capabilities. Dynamic information on data locality and interference may become available.

TetriSched is the new cluster scheduler that works in tandem with a resource reservation system to dynamically adapt to changing conditions. It continuously re-evaluates the cluster schedule, leverages jobs’ time profiles and spatial constraints, and plans ahead to decide which jobs to defer to wait for preferred resources. It leverages significant flexibility afforded by declaratively captured space-time soft constraints and achieves significantly higher SLO attainment and cluster utilization.

Additional information: Heterogeneity and SLO-aware resource reservations.
In submission to Hadoop Summit, June 28-30, 2016, San Jose, CA.

Alexey Tumanov is a 6th year Ph.D. Candidate at Carnegie Mellon advised by Dr. Gregory Ganger. Prior to Carnegie Mellon, Alexey graduated with a research-based M.Sc. in Computer Science from York University and worked on para-virtualization technology at the University of Toronto as a full-time Research Assistant. His most recent research focused on support for static and dynamic heterogeneity, hard and soft placement constraints, time-varying resource capacity guarantees, and combinatorial constraints in heterogeneous datacenters at scale.

Today's Internet has serious security problems. Of particular concern are distributed denial-of-service (DDoS) attacks, which coordinate large numbers of compromised machines to make a service unavailable to other users. DDoS attacks are a constant security threat with over 20,000 DDoS attacks occurring globally every day. They cause tremendous damage to businesses and have catastrophic consequences for national security. In particular, over the past few years, adversaries have started to turn their attention from traditional targets (e.g., end-point servers) to non-traditional ones (e.g., ISP backbone links) to cause much larger attack impact.

In this presentation, I review recent results regarding non-traditional DDoS attacks and potential defense mechanisms. First, I review a non-traditional type of link-flooding attack, called the Crossfire attack, which targets and floods a set of network links in core Internet infrastructure, such as backbone links in large ISP networks. Using Internet-scale measurements and simulations, I show that the attack can cause huge connectivity losses to cities, states, or even countries for hours or even days. Second, I introduce the notion of the routing bottlenecks, or small sets of network links that carry the vast majority of Internet routes, and show that it is a fundamental property of Internet design; i.e., it is a consequence of route-cost minimizations. I also illustrate the pervasiveness of routing bottlenecks around the world, and measure their susceptibility to the Crossfire attack. Finally, I explore the possibility of building a practical defense mechanism that effectively removes the advantages of DDoS adversaries and deters them from launching attacks. The proposed defense mechanism utilizes a software-defined networking (SDN) architecture to protect large ISP networks from non-traditional DDoS attacks.


BIO:

Min Suk Kang is a Ph.D. candidate in Electrical and Computer Engineering (ECE) at Carnegie Mellon University. He is advised by Virgil D. Gligor in CyLab. Before he joined Carnegie Mellon, he worked as a researcher as part of Korean military duty at the Department of Information Technology at KAIST Institute. He received B.S. and M.S. degrees in Electrical Engineering and Computer Science (EECS) at Korea Advanced Institute of Science and Technology (KAIST) in 2006 and 2008, respectively. His research interests include network and distributed system security, wireless network security, and Internet user privacy.

Clusters are increasingly being used in industry to process large graphs, and many distributed graph processing systems have emerged out of both the research community and industry. In this class of distributed systems we explore certain design decisions that appear to be natural and intuitive at first glance, but in fact may turn out to be harmful, in that these decisions hurt performance. Concretely, we first show how intelligent partitioning schemes (for splitting the graph across servers) may not always be worth the cost--in many cases they can be replaced by simpler hash-based schemes that achieve an orders of magnitude improvement in performance. Second we show how checkpointing in such engines is expensive, and that it can be replaced by opportunistic failure recovery mechanisms, which achieve highly accurate answers even under failures. The talk will also present experimental results from our two resulting systems, LFGraph and Zorro, respectively.

Indranil Gupta (Indy) is an Associate Professor of Computer Science at the University of Illinois at Urbana-Champaign. His Distributed Protocols Research Group works on large-scale distributed systems, with a recent focus on cloud and big data systems. Indranil is recipient of the NSF CAREER award in 2005, and Best Paper Awards (IEEE ICAC 2015, BigMine 2012). He has (co-)chaired leading distributed computing conferences (ICDCS 2016 Cloud Computing and Datacenters Track, IEEE P2P 2014, ACM/IFIP/Usenix Middleware 2010, IEEE SASO 2010, StoDiS 2005, ACM PODC 2007 General Chair). Previously, Indranil received his PhD from Cornell University in 2004 and Bachelors degree from IIT Madras in 1998. He worked at Google (2011-12) as a full-time employee/Visiting Scientist in Mountain View, and previously at IBM Research and Microsoft Research.

Faculty Host: Garth Gibson

For the past two decades Internet service providers have focused on increasing the amount of bandwidth available to home users. But for many Internet services, such as electronic commerce and search, reducing latency is the key to improving the user experience and increasing service provider revenues. While in principle the speed of the Internet could nearly match the speed of light, in practice inefficiencies in the physical infrastructure and in network protocols result in latencies that are often one to two orders of magnitude larger than lower bound implied by the speed of light. Hence, we propose a challenge to the networking research community: build a speed-of-light Internet. This talk explores the various causes of delay on the Internet, sketches out two approaches for improving the physical infrastructure, and explores which applications will benefit most from reduced latency.

Bruce Maggs received the S.B., S.M., and Ph.D. degrees in computer science from the Massachusetts Institute of Technology in 1985, 1986, and 1989, respectively. His advisor was Charles Leiserson. After spending one year as a Postdoctoral Associate at MIT, he worked as a Research Scientist at NEC Research Institute in Princeton from 1990 to 1993. In 1994, he moved to Carnegie Mellon, where he stayed until joining Duke University in 2009 as a Professor in the Department of Computer Science. While on a two-year leave-of-absence from Carnegie Mellon, Maggs helped to launch Akamai Technologies, serving as its first Vice President for Research and Development. He retains a part-time role at Akamai as Vice President for Research.

Maggs's research focuses on networking, distributed systems, and security. In 1986, he became the first winner (with Charles Leiserson) of the Daniel L. Slotnick Award for Most Original Paper at the International Conference on Parallel Processing, and in 1994 he received an NSF National Young Investigator Award. He was co-chair of the 1993-1994 DIMACS Special Year on Massively Parallel Computation and has served on the steering committees for the ACM Symposium on Parallel Algorithms and Architectures (SPAA), the ACM Internet Measurement Conference (IMC), and the ACM HotNets Conference. He has served on the program committees of numerous ACM conferences including STOC, SODA, PODC, and SIGCOMM.

Faculty Host: Peter Steenkiste

Storage systems rely on maintenance tasks, such as backup and layout optimization, to ensure data availability and good performance. These tasks access large amounts of data and can significantly impact foreground applications. The premise of this talk is that storage maintenance could be performed more efficiently by prioritizing processing of data that is currently cached in memory. Data can be cached either due to other maintenance tasks requesting it previously, or due to overlapping foreground I/O activity. I will be presenting Duet, a framework that provides notifications about page-level events to maintenance tasks, such as a page being added or modified in memory. Tasks use these events as hints to opportunistically process cached data. Our results show that tasks using Duet can complete maintenance work more efficiently because they perform fewer I/O operations. The I/O reduction depends on the amount of data overlap with other maintenance tasks and foreground applications. Consequently, Duet's efficiency increases with additional tasks because opportunities for synergy appear more often.

George Amvrosiadis is a Ph.D. Candidate and member of the Systems and Networks lab of the Computer Science Department at the University of Toronto. Under the supervision of Prof. Angela Demke Brown and Prof. Ashvin Goel, he is currently investigating ways to enable applications with a strong storage component to collaborate on overlapping work, allowing them to reach their goals faster. His earlier work focused on storage reliability (under the supervision of Prof. Bianca Schroeder), high-performance clustered file systems (at IBM Research), and using data analytics to uncover pain points in modern backup and recovery software (at Symantec Research Labs). He enjoys operating systems research, with a focus on storage, especially when rooted in insights from real-world data analysis.

Faculty Host: Garth Gibson

Pages

Subscribe to SDI/ISTC