Hadoop守护进程

一、五大守护进程

hadoop1.x

NameNode

DataNode

SecondaryNameNode

TaskTracker

JobTracker

hadoop2.x

SecondaryNameNode

ResourceManager

NodeManager

NameNode

DataNode

二、三种启动与关闭方式

1、sbin/start-all.sh

sbin/stop-all.sh

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ jps

31355 Jps

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ sbin/start-all.sh

This script isDeprecated. Instead use start-dfs.sh and start-yarn.sh

17/05/29 09:14:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting namenodes on [localhost]

localhost: starting namenode, logging to /myFile/04.BigData/hadoop-2.7.0/logs/hadoop-zhaojunyan-namenode-zhaojunyandeMacBook-Pro.local.out

localhost: starting datanode, logging to /myFile/04.BigData/hadoop-2.7.0/logs/hadoop-zhaojunyan-datanode-zhaojunyandeMacBook-Pro.local.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /myFile/04.BigData/hadoop-2.7.0/logs/hadoop-zhaojunyan-secondarynamenode-zhaojunyandeMacBook-Pro.local.out

17/05/29 09:14:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

starting yarn daemons

starting resourcemanager, logging to /myFile/04.BigData/hadoop-2.7.0/logs/yarn-zhaojunyan-resourcemanager-zhaojunyandeMacBook-Pro.local.out

localhost: starting nodemanager, logging to /myFile/04.BigData/hadoop-2.7.0/logs/yarn-zhaojunyan-nodemanager-zhaojunyandeMacBook-Pro.local.out

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ jps

31888 Jps

31650 SecondaryNameNode

31762 ResourceManager

31847 NodeManager

31466 NameNode

31547 DataNode

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ sbin/stop-all.sh

This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh

17/05/29 09:15:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Stopping namenodes on [localhost]

localhost: stopping namenode

localhost: stopping datanode

Stopping secondary namenodes [0.0.0.0]

0.0.0.0: stopping secondarynamenode

17/05/29 09:15:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

stopping yarn daemons

stopping resourcemanager

localhost: stopping nodemanager

no proxyserver to stop

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$

2、sbin/start-dfs.sh

sbin/start-yarn.sh(hadoop2.x) sbin/start-mapred.sh(hadoop1.x)

sbin/stop-yarn.sh(hadoop2.x) sbin/stop-mapred.sh(hadoop1.x)

sbin/stop-dfs.sh

dfs: DataNode SecondaryNameNode NameNode

yarn: ResourceManager NodeManager

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ jps

33595 Jps

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ sbin/start-dfs.sh

17/05/29 09:27:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting namenodes on [localhost]

localhost: starting namenode, logging to /myFile/04.BigData/hadoop-2.7.0/logs/hadoop-zhaojunyan-namenode-zhaojunyandeMacBook-Pro.local.out

localhost: starting datanode, logging to /myFile/04.BigData/hadoop-2.7.0/logs/hadoop-zhaojunyan-datanode-zhaojunyandeMacBook-Pro.local.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /myFile/04.BigData/hadoop-2.7.0/logs/hadoop-zhaojunyan-secondarynamenode-zhaojunyandeMacBook-Pro.local.out

17/05/29 09:27:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ jps

33778 DataNode

33883 SecondaryNameNode

33695 NameNode

33951 Jps

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ sbin/start-yarn.sh

starting yarn daemons

starting resourcemanager, logging to /myFile/04.BigData/hadoop-2.7.0/logs/yarn-zhaojunyan-resourcemanager-zhaojunyandeMacBook-Pro.local.out

localhost: starting nodemanager, logging to /myFile/04.BigData/hadoop-2.7.0/logs/yarn-zhaojunyan-nodemanager-zhaojunyandeMacBook-Pro.local.out

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ jps

33778 DataNode

34009 ResourceManager

33883 SecondaryNameNode

34139 Jps

33695 NameNode

34095 NodeManager

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ sbin/stop-yarn.sh

stopping yarn daemons

stopping resourcemanager

localhost: stopping nodemanager

no proxyserver to stop

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ jps

33778 DataNode

34296 Jps

33883 SecondaryNameNode

33695 NameNode

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ sbin/stop-dfs.sh

17/05/29 09:29:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Stopping namenodes on [localhost]

localhost: stopping namenode

localhost: stopping datanode

Stopping secondary namenodes [0.0.0.0]

0.0.0.0: stopping secondarynamenode

17/05/29 09:29:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ jps

34558 Jps

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$

3、依次启动NameNode、DataNode、secondarynamenode、resourcemanager、nodemanager，关闭时反序依次关闭。命令1.x与2.x在启动时有些区别。另外：

sbin/hadoop-daemon.sh start datanode（如果有多个datanode，需使用hadoop-daemons.sh）

sbin/yarn-daemon.sh start nodemanager（如果有多个datanode，需使用yarn-daemons.sh）

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ jps

34786 Jps

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ sbin/hadoop-daemon.sh start namenode

starting namenode, logging to /myFile/04.BigData/hadoop-2.7.0/logs/hadoop-zhaojunyan-namenode-zhaojunyandeMacBook-Pro.local.out

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ sbin/hadoop-daemon.sh start datanode

starting datanode, logging to /myFile/04.BigData/hadoop-2.7.0/logs/hadoop-zhaojunyan-datanode-zhaojunyandeMacBook-Pro.local.out

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ jps

34818 NameNode

34888 DataNode

34920 Jps

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ sbin/hadoop-daemon.sh start secondarynamenode

starting secondarynamenode, logging to /myFile/04.BigData/hadoop-2.7.0/logs/hadoop-zhaojunyan-secondarynamenode-zhaojunyandeMacBook-Pro.local.out

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ jps

35920 Jps

34818 NameNode

34888 DataNode

35097 SecondaryNameNode

35897 ResourceManager

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ sbin/yarn-daemon.sh start nodemanager

starting nodemanager, logging to /myFile/04.BigData/hadoop-2.7.0/logs/yarn-zhaojunyan-nodemanager-zhaojunyandeMacBook-Pro.local.out

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ jps

34818 NameNode

34888 DataNode

35960 NodeManager

35097 SecondaryNameNode

35897 ResourceManager

35998 Jps

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ sbin/yarn-daemon.sh stop nodemanager

stopping nodemanager

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ sbin/yarn-daemon.sh stop resourcemanager

stopping resourcemanager

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ sbin/hadoop-daemon.sh stop secondarynamenode

stopping secondarynamenode

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ sbin/hadoop-daemon.sh stop datanode

stopping datanode

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ sbin/hadoop-daemon.sh stop namenode

stopping namenode

zhaojunyandeMacBook-Pro:hadoop-2.7.0 zhaojunyan$ jps

36256 Jps

脚步分析：

sbin/start-all.sh脚本中两个关键信息：

1、# Start all hadoop daemons.  Run this on master node.

2、执行命令分拆为两步

sbin/start-dfs.sh

sbin/start-yarn.sh

再看sbin/start-dfs.sh，有三个关键信息

1、# Optinally upgrade or rollback dfs state.
2、# Run this on master node.

3、"$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \
  --config "$HADOOP_CONF_DIR" \
  --hostnames "$NAMENODES" \
  --script "$bin/hdfs" start namenode $nameStartOpt

$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \
    --config "$HADOOP_CONF_DIR" \
    --script "$bin/hdfs" start datanode $dataStartOpt

"$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \
      --config "$HADOOP_CONF_DIR" \
      --hostnames "$SECONDARY_NAMENODES" \
      --script "$bin/hdfs" start secondarynamenode

再看sbin/start-yarn.sh，有三个关键信息

1、# Start all yarn daemons.  Run this on master node.

2、"$bin"/yarn-daemon.sh --config $YARN_CONF_DIR  start resourcemanager
"$bin"/yarn-daemons.sh --config $YARN_CONF_DIR  start nodemanager

最后看下hadoop-daemon.sh与hadoop-daemons.sh，他们的区别可以在shell脚步中可以看出

hadoop-daemon.sh：# Runs a Hadoop command as a daemon.

hadoop-daemons.sh：# Run a Hadoop command on all slave hosts. （在所有从节点上运行，调用的是hadoop-daemon.sh）
usage="Usage: hadoop-daemons.sh [--config confdir] [--hosts hostlistfile] [start|stop] command args..."

exec "$bin/slaves.sh" --config $HADOOP_CONF_DIR cd "$HADOOP_PREFIX" \; "$bin/hadoop-daemon.sh" --config $HADOOP_CONF_DIR "$@"

总结：推荐使用最后一种方法启动，并通过jps命令查看是否启动成功，如果失败，可通过查看日志分析原因。

PS：hadoop1.x听说要严格按照启动顺序，hadoop2.x不按照顺序启动和关闭也可以成功。

从脚步中也可以看出在stop-all.sh中并没有像hadoop1.x中那样是反序的。

# stop hdfs daemons if hdfs is present
if [ -f "${HADOOP_HDFS_HOME}"/sbin/stop-dfs.sh ]; then
  "${HADOOP_HDFS_HOME}"/sbin/stop-dfs.sh --config $HADOOP_CONF_DIR
fi
# stop yarn daemons if yarn is present
if [ -f "${HADOOP_HDFS_HOME}"/sbin/stop-yarn.sh ]; then
  "${HADOOP_HDFS_HOME}"/sbin/stop-yarn.sh --config $HADOOP_CONF_DIR
fi