《Hadoop: The Definitive Guide》读书笔记 -- Chapter 3 The Hadoop distributed Filesystem

Chapter 3 The Hadoop distributed Filesystem

Design:

1. Very Large Files

2. Streaming Data Access (Write once, read-many-times)

3. Commodity Hardware

NOT GOOD FIT

1. Low-latency data access（HBase is currently a better choice for low-latency）

2. Lots of small files

3. Multiple writers, arbitrary file modification(No support for multiple writters? Why? Reliabilit?)

HDFS Concepts

1. Blocks

Unlike a filesystem for a single disk, a file in HDFS that is much smaller than a single block not occupy a full block's worth underlying storage.

Why HDFS block large? minize the time of seek

2. Namenodes and Datanodes

Namenode(master): manage filesystem namespace, maintain filesystem tree and its metadata(persistently), blocks location(in memory, reconstruct)

Datanode(slave): blocks

Namenode(master) Resilient Mechanism:

a) Back up filesystem and metadata

b) secondary namenode

3. Block Caching

4. HDFS Federation

Problem: files number limit to single node. 

Solution: cluster of namenodes. ex: NodeA for /user, NodeB for /etc

5. HDFS High Availability

http://blog.cloudera.com/blog/2012/03/high-availability-for-the-hadoop-distributed-file-system-hdfs/

The Command Line interface

Hadoop Filesystems

Hadoop has an abstract notion of filesystem, of which HDFS is just one implementation.

其他的包括：S3、Azure、FTP等。WebHDFS可以使用Http的方式访问文件系统

Data Flow

写时的容错处理