Understanding HBase column-family performance options

虽说是篇关于hbase的老文章了，但是现在看看说的还是很不错的，原文地址（翻墙吧）：

http://jimbojw.com/wiki/index.php?title=Understanding_HBase_column-family_performance_options

In the comments to Understanding HBase and BigTable, I recieved some insightul questions. Here I attempt to answer them, in no particular order.

Picking the correct HBase performance options is akin to deciding which engine use, or whether to use CHAR vs VARCHAR vs TEXT in a relational database. These decisions can make a big impact on the amount of data stored and the speed with which it is created, updated, read, and deleted.

This article assumes the reader is familiar with HBase concepts, particularly its column-oriented nature and the relationship between rows, column families, columns and cells. If this seems foreign to you, I recommend revisiting the aforementioned Understanding HBase article before reading further.

For information on the syntax used to create a table using the options mentioned here, see HbaseShell (Hadoop wiki).

Are there any performance implications that are implied with column families?

Definitely. All the columns within a column family will share the same characteristics such as versioning and compression. By default, HBase does not employ any kind of compression on cell data, but two alternative compressions may be specified: BLOCK and RECORD. 

数据决定：出于压缩和解压效率的考虑

Block compression

Say you have a single column which will contain large blobs of text data, and you only want to keep one version for any given row.

In that case, you'd probably want that column to belong to a column family which supports BLOCK compression, since this compression type will span across multiple rows in order to achieve the best compression ratio.

Record compression

On the other hand, say you had a variable number of rows containing text data, of which you'd want to keep multiple versions. Then you might want those columns to belong to a family which uses RECORD compression, since this compression type will be be localized within each row.

Although compression ratios generally would be better with BLOCK compression rather than RECORD compression, access times for RECORD compression would theoretically be faster since only a single row would need to be pulled in order to decompress a given cell.

Should columns be grouped into families based on how they're used in a particular application?

Yes, absolutely. Consider these other column family options: BLOOMFILTER, IN_MEMORY, MAX_LENGTH and MAX_VERSIONS.

Bloom Filters

If a column family supports bloom filters, that means that an extra index is kept which helps cut down on the time necessary to determine if a given column exists in a given row. This has nothing to do with the cell values, just the the row/column identifiers.

In the case where you have a very large number of variably named columns, each cell having a small amount of data, you may want to specify them in a column family utilizing a bloom filter, so that lookup times are reduced. 

Column的删除：需要重建BLOOMFILTER...代价很大

Like any index, bloom filters incur an additional storage cost (memory) and an update cost (time). The sole purpose of a bloom filter is to quickly determine whether a given input has ever been seen before, using a minimum of storage space. Inserting new items and checking for existing items are both fast. The one slow operation is deletion, which requires rebuilding the entire index from scratch, but skipping the deleted item. Other bloom filter variants support a certain deletion tolerance, but given enough deletions the index would still need to be rebuilt.

IN_MEMORY

Another example characteristic is the IN_MEMORY option, which directs HBase to keep cell values loaded in memory more aggressively than it would normally do. The upside is that this should really speed up certain kinds of read/write patterns.

The downside is, of course, that this eats up RAM, and secondarily that it may interfere with making HDFS backups since the data would be written to disk less frequently (again, this is speculation as I haven't seen any benchmarks on the issue).

MAX_LENGTH and MAX_VERSIONS

Deciding which characteristics to apply to a column family is important from a performance perspective, but rarely have an impact on actual functionality. About the only settings which make a functional difference are MAX_VERSIONS and MAX_LENGTH, which specify how many versions of a cell to keep (default is 3), and how many bytes of data can be stored in each cell version (default is the max size of a 32bit signed integer).

Summary

In order to pick the appropriate performance options for your column families, you'll have to consider the forms of data likely to be stored, as well as the manner in which is inserted, updated and retrieved.