hadoop 学习笔记（1）

在Coursera上学习的一门课程  ：

Hadoop Platform and Application Framework

by University of California, San Diego

https://www.coursera.org/learn/hadoop/home/welcome

里面讲得很好，就是我这边的网下不下来一个cloudera的软件，我也正在学习中，对于HADOOP的了解很有帮助。接下来记一些笔记。

Lesson1:

Common:libraries and utilities

Yarn :enhancesde power of a Hadoop compute cluster ，a resource-management platform,scheduling.

Mapreduce:a programming model for large scale data processing.

HDFS:Hadoop Distributed File System(Hadoop分布式文件系统)：

week2：

Yarn, Tez and Spark:都是framework

HDFS2：storage layer

YARN： essentially the basic execution engine in the next generation of Hadoop

Hbase ，other apps： work though on YARN

week3:

HDFS:

1.Introduction to HDFS:

HDFS Design Concept:

• Scalable distributed filesystem

• Distribute data on local disks on several nodes

• Low cost commodity hardware

HDFS Design Factors :

• Hundreds/Thousands of nodes => • Need to handle node/disk failures

• Portability across heterogeneous hardware/software

• Handle large data sets

• High throughput

Approach to meet HDFS design goals:

• Simplified coherency model – write once read many.

• Data Replication – helps handle hardware failures

• Move computation close to data

• Relax POSIX requirements – increase throughput

2.HDFS Architecture and Configuration:

Summary of HDFS Architecture

• Single NameNode - a master server that manages the file system namespace and regulates access to files by clients.

• Multiple DataNodes – typically one per node in the cluster.Functions:

• Manage storage

• Serving read/write requests from clients

• Block creation, deletion, replication based on instructions from NameNode

Performance Envelope of HDFS :

• Able to determine number of blocks for a given file size

• Key HDFS and system components impacted by block size

• Impact of small files on HDFS and system

Default block size is 64MB

10GB = 10 X 1024.  blocks = 10 X 1024/64 =160 bolcks.

3.Read / Write process in HDFS:

另外附上课里面一个学生区域的统计 ：可以发现印度学生真的真的很多，北美的学习者也很多，我们的学习还要努力啊！