log in  |  register  |  feedback?  |  help  |  web accessibility
Logo
A Hitchhiker's Guide to Fast and Efficient Data Reconstruction in Erasure-coded Data Centers
James Litton - University of Maryland
Wednesday, August 13, 2014, 2:00-3:00 pm Calendar
  • You are subscribed to this talk through .
  • You are watching this talk through .
  • You are subscribed to this talk. (unsubscribe, watch)
  • You are watching this talk. (unwatch, subscribe)
  • You are not subscribed to this talk. (watch, subscribe)
Abstract

Erasure codes such as Reed-Solomon (RS) codes are being extensively deployed
in data centers since they offer significantly higher reliability than data
replication methods at much lower storage overheads. These codes however
mandate much higher resources with respect to network bandwidth and disk IO
during reconstruction of data that is missing or otherwise unavailable.
Existing solutions to this problem either demand additional storage space or
severely limit the choice of the system parameters.

In this paper, we present Hitchhiker, a new erasure-coded storage system
that reduces both network traffic and disk IO by around 25% to 45% during
reconstruction of missing or otherwise unavailable data, with no additional
storage, the same fault tolerance, and arbitrary flexibility in the choice
of parameters, as compared to RS-based systems. Hitchhiker "rides" on top of
RS codes, and is based on novel encoding and decoding techniques that will
be presented in this paper. We have implemented Hitchhiker in the Hadoop
Distributed File System (HDFS). When evaluating various metrics on the
data-warehouse cluster in production at Facebook with real-time traffic and
workloads, during reconstruction, we observe a 36% reduction in the
computation time and a 32% reduction in the data read time, in addition to
the 35% reduction in network traffic and disk IO. Hitchhiker can thus reduce
the latency of degraded reads and perform faster recovery from failed or
decommissioned machines.

Bio

James Litton is a second year PhD student working with Dr. Bobby Bhattacharjee. His interests lie primarily in systems and networking.

This talk is organized by Ramakrishna Padmanabhan