What better way to start a blog on BigData then to share one's thoughts about the just concluded First HBase Conference - HBaseCon 2012. The one day conference on May 23rd was a great opportunity to see many of the big names as well as enthusiasts of the HBase world come together and share their experiences and advice.
Unlike many other conferences that I have attended in the past, this was unique because there wasn't a single vendor or vendor booth in the conference - it was "all stuff and no fluff". Ofcourse, there was Cloudera (a vendor!) - but hey, they were the conference organizer, and true to the spirit of the conference, there wasn't a single "Cloudera salesperson" around trying to ask you about your project or trying to sell you their consulting services.
The conference was followed by an HBase Hackathon that was a wonderful experience to get an insight into the minds of the HBase committers and a great learning experience.
There were a number of takeaways from the HBaseCon which are listed below in no particular order. Note that these are purely my takeaways of the sessions that I attended. There were 4 tracks in parallel - so there were more than 4 times the number of sessions that I have listed below!
One of the biggest takeaway from the HBaseCon was Facebook's sharing of their use of HBase by Karthik Ranganathan. Facebook's showcase HBase implementation is their messaging application. We all know the amount of data that is pumped into Facebook by everyone and we would think that they would have a humongous HBase cluster. However (atleast) I was wrong - cause they have done a wonderful job of sharding their data and the shards are spread across a number of HBase clusters - referred to as "cells". The cells have about 200 nodes with some judicious use of intra-datacenter and inter-datacenter replication. It was also a pleasant surprise that their production environments use Hadoop version 0.89-fb - while the current stable Apache HBase version is 0.94. Finally, they are using HBase like an OLTP data store - configuring the HBase block size as 16KB (default is 64KB). This block size has allowed them to get the best balance in terms of compression, block index size control and efficient and effective use of block cache & memstore. My guess is that the rationale behind this block size is that it allows efficient use of the memory structures and blocks from the low, hardware-level L2/L3 and TLB (translation lookaside buffers) cache and associated memory management system/architecture.
A second takeaway from Facebook was their use of HDFS hard-links for backups. Hard-links are Facebook's creation that are being fed into the open-source trunk (see Hadoop Jira 3370). The immutable nature of HFiles makes them an ideal candidate for backup as described in the session "HBase Backup" by Sunil Sitaula (Cloudera) and Madhuwanti Vaidya (Facebook).
And a third takeaway from Facebook was Ryan Thiessen showing their battle scars of being DevOps for large HBase installations as theirs. He was able to drive home the fact about how different a large, distributed system is from a traditional RDBMS installation. When designing a traditional RDBMS system, one has the luxury of packing in all kinds of redundancy - RAID disks, redundant RAID controllers, redundant NICs, HA clusters, etc. However that design cannot be scaled to a large number of cells, each composed of about 200 nodes. And as per their experience - rack and cluster level network switch failures were the most disruptive failures. This was inspite of the Hadoop Namenode and HBase Master nodes being the obvious SPOFs (single points of failures).
Lars Hofhansl from Salesforce provided a very easy to follow walk-through of some of the HBase APIs encouraging one (or atleast me!) to peek into the code and try to understand some of the internals. His session - "Learning HBase Internals" explained how HBase guarantees row-level atomic operations, how blocks are used, how compression is applied, etc.
Ian Varley's (Salesforce) session on HBase Schema Design progressively walked us through how to design entities and entity relationships. He also emphasized the "cost of verbosity" in your schema design - i.e. the larger your rowkeys, column-family and column names, higher is the overhead for each data cell metadata. He pointed out the use of very succinct data structure and schema design in TSDB.
Finally, Mikhail Bautin (Facebook) had a ton of best-practices and checklists for us. Every slide of his had a few tips - and I don't remember a lot of them at this time. So be on the lookout for his presentation to be shared/published on HBaseCon, Cloudera or somewhere else!
I hope that this HBaseCon has set the bar and we will see more knowledge sharing and a conference that is "more stuff and less fluff" and zero to minimal hustling from vendors.
Ah, before I forget, let me share my Hackathon experience. It was organized/hosted by Cloudera at their offices. It was an immense pleasure to see all the big HBase names like Todd Lipcon, Michael Stack, Lars Hofhansl, Andrew Purtell, Lars George and many others come together in encouraging people like me to download and compile HBase source code, browse through the Jira tickets, do code-reviews, etc. In fact David S Wang - the Cloudera HBase development team manager himself sat with me for a while walking me through the process of building Hadoop and HBase from source.
This article is a tribute to the HBase team and their warmth and eagerness to help the community.
