HBase是否适合存储Blob数据?

来源:互联网 发布:linux vi命令退出 编辑:程序博客网 时间:2024/06/04 18:19
This is a tricky question - it depends on how large the blobs are, what is the read/write ratio and the overall sizing of the system (disk/memory relative to the read/write requirements). 

Let me try to point out some obvious downsides (which may not matter depending on above):

  1. HBase periodically compacts it's entire index on-disk. It does so to provide efficient random lookups against this index. Because the index and row data (blob in other words) are stored together (what is called an 'index-organized-table' in classic RDBMS lingo) - the compaction of index also requires the movement of row data. The more the size of the row/blob in proportion to the index itself - the more the cost of the compactions. If one is storing petabytes of data composed of large blobs (say multiple megabytes each) - then the compactions will become very expensive (and wasteful).
  2. Committing a (large) blob to disk will be unnecessarily expensive (first the entire blob will be written once to the commit log and then it will be written out again to the actual HFile. (Each of those writes would itself be 3x replicated by hdfs and u might want to write to another data center too!)
  3. If the blobs are immutable (for example image data) - then the cost of (HDFS) replication becomes very high (3x per data center). For immutable data - one can easily store one copy per data center (and fall back to other data centers   if one copy goes missing). If u are storing large amount of data - this could affect cost a lot.
  4. Finally - the index itself becomes more fragmented when large blobs are stored inline (HBase will try to compact it periodically - but if u are writing large amounts of data - there will always be some fragmentation). Which would imply some (hard to quantify modulo caching strategies) degradation in the performance of the index itself.

A simple solution is to use HBase to only store the index itself and store blobs elsewhere. As a naiive example - one could store the blobs as files in an external file system (could be HDFS or not) and store the name(s) of the file/offset in hbase (against the identifier for the blob). See for example Oracle documentation around storing LOBs inline or in external file systems. One of the main issues with this approach is the lack of atomicity in writing the blob data and updating the index (a problem Oracle takes care of internally). Usually not a big deal (since errors are infrequent) - but worth pointing out.