hadoop2.2.0安装与启动
来源:互联网 发布:linux安装usb网卡驱动 编辑:程序博客网 时间:2024/05/22 12:06
Setup newest Hadoop 2.x (2.2.0) on Ubuntu
In this tutorial I am going to guide you through setting up hadoop 2.2.0 environment on Ubuntu.After user is created, re-login into ubuntu using hduserRe-login into Ubuntu using hdser and check hadoop versionAt this point, hadoop is installed.
Prerequistive
- $ sudo apt-get install openjdk-7-jdk
- $ java -version
- java version "1.7.0_25"
- OpenJDK Runtime Environment (IcedTea 2.3.12) (7u25-2.3.12-4ubuntu3)
- OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
- $ cd /usr/lib/jvm
- $ ln -s java-7-openjdk-amd64 jdk
- $ sudo apt-get install openssh-server
Add Hadoop Group and User
- $ sudo addgroup hadoop
- $ sudo adduser --ingroup hadoop hduser
- $ sudo adduser hduser sudo
Setup SSH Certificate
- $ ssh-keygen -t rsa -P ''
- ...
- Your identification has been saved in /home/hduser/.ssh/id_rsa.
- Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.
- ...
- $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
- $ ssh localhost
Download Hadoop 2.2.0
- $ cd ~
- $ wget http://www.trieuvan.com/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz
- $ sudo tar vxzf hadoop-2.2.0.tar.gz -C /usr/local
- $ cd /usr/local
- $ sudo mv hadoop-2.2.0 hadoop
- $ sudo chown -R hduser:hadoop hadoop
Setup Hadoop Environment Variables
- $cd ~
- $vi .bashrc
- paste following to the end of the file
- #Hadoop variables
- export JAVA_HOME=/usr/lib/jvm/jdk/
- export HADOOP_INSTALL=/usr/local/hadoop
- export PATH=$PATH:$HADOOP_INSTALL/bin
- export PATH=$PATH:$HADOOP_INSTALL/sbin
- export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
- export HADOOP_COMMON_HOME=$HADOOP_INSTALL
- export HADOOP_HDFS_HOME=$HADOOP_INSTALL
- export YARN_HOME=$HADOOP_INSTALL
- ###end of paste
- $ cd /usr/local/hadoop/etc/hadoop
- $ vi hadoop-env.sh
- #modify JAVA_HOME
- export JAVA_HOME=/usr/lib/jvm/jdk/
- $ hadoop version
- Hadoop 2.2.0
- Subversion https://svn.apache.org/repos/asf/hadoop/common -r 1529768
- Compiled by hortonmu on 2013-10-07T06:28Z
- Compiled with protoc 2.5.0
- From source with checksum 79e53ce7994d1628b240f09af91e1af4
- This command was run using /usr/local/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar
Configure Hadoop
- $ cd /usr/local/hadoop/etc/hadoop
- $ vi core-site.xml
- #Paste following between <configuration>
- fs.default.name
- hdfs://localhost:9000
- $ vi yarn-site.xml
- #Paste following between <configuration>
- yarn.nodemanager.aux-services
- mapreduce_shuffle
- yarn.nodemanager.aux-services.mapreduce.shuffle.class
- org.apache.hadoop.mapred.ShuffleHandler
- $ mv mapred-site.xml.template mapred-site.xml
- $ vi mapred-site.xml
- #Paste following between <configuration>
- mapreduce.framework.name
- yarn
- $ cd ~
- $ mkdir -p mydata/hdfs/namenode
- $ mkdir -p mydata/hdfs/datanode
- $ cd /usr/local/hadoop/etc/hadoop
- $ vi hdfs-site.xml
- Paste following between <configuration> tag
- dfs.replication
- 1
- dfs.namenode.name.dir
- file:/home/hduser/mydata/hdfs/namenode
- dfs.datanode.data.dir
- file:/home/hduser/mydata/hdfs/datanode
Format Namenode
- hduser@ubuntu40:~$ hdfs namenode -format
Start Hadoop Service
- $ start-dfs.sh
- ....
- $ start-yarn.sh
- ....
- hduser@ubuntu40:~$ jps
- If everything is sucessful, you should see following services running
- 2583 DataNode
- 2970 ResourceManager
- 3461 Jps
- 3177 NodeManager
- 2361 NameNode
- 2840 SecondaryNameNode
Run Hadoop Example
- hduser@ubuntu: cd /usr/local/hadoop
- hduser@ubuntu:/usr/local/hadoop$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5
- Number of Maps = 2
- Samples per Map = 5
- 13/10/21 18:41:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
- Wrote input for Map #0
- Wrote input for Map #1
- Starting Job
- 13/10/21 18:41:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
- 13/10/21 18:41:04 INFO input.FileInputFormat: Total input paths to process : 2
- 13/10/21 18:41:04 INFO mapreduce.JobSubmitter: number of splits:2
- 13/10/21 18:41:04 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
- ...
- 转帖地址:<a target="_blank" href="http://codesfusion.blogspot.com/2013/10/setup-hadoop-2x-220-on-ubuntu.html?m=1">http://codesfusion.blogspot.com/2013/10/setup-hadoop-2x-220-on-ubuntu.html?m=1</a>
0 0
- hadoop2.2.0安装与启动
- hadoop2.2.0安装与启动
- hadoop2.2.0编译与安装
- Hadoop2初介绍与OSX安装Hadoop2
- Hadoop2.x与Zookeeper安装
- Hadoop2.7.1安装与配置
- hadoop - hadoop2.6 分布式 - 集群环境搭建 - Hadoop 2.6 分布式安装配置与启动
- linux安装hadoop2.7.2nodemanager启动停止
- windows下安装并启动hadoop2.7.2
- windows下安装并启动Hadoop2.7.2
- windows下安装并启动hadoop2.7.2
- windows下安装并启动hadoop2.6.1
- hadoop2.2.0 安装
- hadoop2.2.0集群安装
- Hadoop2.2.0分布式安装
- win7 安装hadoop2.2.0
- hadoop2.2.0安装
- hadoop2.2.0安装说明
- Java中容器类:ArrayList 详解
- 每天进步一点点
- “复旦602” 电子数字计算机
- chattr命令
- VBscript 中库函数的使用练习
- hadoop2.2.0安装与启动
- I2C通信 读写数据过程
- C/C++ 初学简单笔记 —1—
- 数据库同一张表字段横向与纵向相加
- 演练Ext JS 4.2自定义主题
- [遇到的问题]ubuntu 用wget下载ftp时,报端口号错误
- IOS KVO,NSKeyValueObservance: Observer:, Key path:, Options:
- Kruskal 最小生成树
- CodeForces 208A/D 18周周赛