HBase Basics

Apache HBase is a distributed, column-oriented NoSQL database modeled after Google's Bigtable paper, running on top of HDFS to provide random, real-time read/write access to individual rows, something HDFS and Hive alone cannot do efficiently since HDFS is optimized for large sequential scans and append-only writes rather than point lookups or updates. HBase trades away SQL joins, secondary indexes, and multi-row transactions for horizontal scalability and low-latency access to billions of rows and millions of columns, making it suited to use cases like a real-time recommendation lookup table or a time-series sensor archive rather than ad-hoc analytical queries.

HBase organizes data into tables of rows identified by a unique rowkey, sorted lexicographically, and within each row, columns are grouped into column families declared at table-creation time, with individual column qualifiers created dynamically at write time without any schema change; every cell additionally carries a version, a timestamp, so a get can retrieve the latest value or a specific historical version, and because HBase stores columns physically together per column family rather than per row, queries that touch only one family avoid reading unrelated data. Rowkey design is the single most consequential HBase modeling decision, since all fast lookups and range scans depend on it, a poorly chosen key, like a monotonically increasing timestamp, causes writes to pile onto one region while the rest of the cluster sits idle, a problem called hotspotting.

An HBase cluster's master daemon, the HMaster, handles table-level DDL, region assignment, and load balancing, while the actual data lives on RegionServers, each hosting many Regions, contiguous rowkey ranges of a table that automatically split once they exceed a configured size threshold, and each RegionServer also colocates with the HDFS DataNode on the same physical machine for data locality. ZooKeeper coordinates cluster state, tracking which RegionServer is currently serving which region and electing a new HMaster if the active one fails, and every write first goes to a Write-Ahead Log (WAL) on that RegionServer before being acknowledged, so a crashed RegionServer's in-memory writes can be replayed from its WAL during recovery.

Writes land first in an in-memory MemStore per column family and are appended to the WAL for durability; once a MemStore fills past a threshold, it's flushed to disk as an immutable HFile, and because a single logical row's data can end up scattered across many HFiles over time, a background minor compaction periodically merges several small HFiles into a larger one, while a more expensive major compaction merges all HFiles for a region into one, permanently discarding cells marked for deletion and old versions beyond the configured retention count. Reads therefore may need to check the MemStore plus multiple HFiles and merge results on the fly, which is why HBase read latency, while still far faster than a full HDFS scan, is typically higher than its write latency.