hadoop2.2.0 singlenode 配置
来源:互联网 发布:我的世界星星知多少 编辑:程序博客网 时间:2024/05/29 18:01
原文: http://codesfusion.blogspot.com/2013/10/setup-hadoop-2x-220-on-ubuntu.html?m=1
Setup newest Hadoop 2.x (2.2.0) on Ubuntu
In this tutorial I am going to guide you through setting up hadoop 2.2.0 environment on Ubuntu.
Note: ericduq has created a shell script (make-single-node.sh) for this setup and it is available at git repo athttps://github.com/ericduq/hadoop-scripts.
Prerequistive
$ sudo apt-get install openjdk-7-jdk$ java -versionjava version "1.7.0_25"OpenJDK Runtime Environment (IcedTea 2.3.12) (7u25-2.3.12-4ubuntu3)OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)$ cd /usr/lib/jvm$ ln -s java-7-openjdk-amd64 jdk$ sudo apt-get install openssh-server
Add Hadoop Group and User
$ sudo addgroup hadoop$ sudo adduser --ingroup hadoop hduser$ sudo adduser hduser sudoAfter user is created, re-login into ubuntu using hduser
Setup SSH Certificate
$ ssh-keygen -t rsa -P ''...Your identification has been saved in /home/hduser/.ssh/id_rsa.Your public key has been saved in /home/hduser/.ssh/id_rsa.pub....$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys$ ssh localhost
Download Hadoop 2.2.0
$ cd ~$ wget http://www.trieuvan.com/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz$ sudo tar vxzf hadoop-2.2.0.tar.gz -C /usr/local$ cd /usr/local$ sudo mv hadoop-2.2.0 hadoop$ sudo chown -R hduser:hadoop hadoop
Setup Hadoop Environment Variables
$cd ~$vi .bashrcpaste following to the end of the file#Hadoop variablesexport JAVA_HOME=/usr/lib/jvm/jdk/export HADOOP_INSTALL=/usr/local/hadoopexport PATH=$PATH:$HADOOP_INSTALL/binexport PATH=$PATH:$HADOOP_INSTALL/sbinexport HADOOP_MAPRED_HOME=$HADOOP_INSTALLexport HADOOP_COMMON_HOME=$HADOOP_INSTALLexport HADOOP_HDFS_HOME=$HADOOP_INSTALLexport YARN_HOME=$HADOOP_INSTALL###end of paste$ cd /usr/local/hadoop/etc/hadoop$ vi hadoop-env.sh#modify JAVA_HOMEexport JAVA_HOME=/usr/lib/jvm/jdk/Re-login into Ubuntu using hdser and check hadoop version
$ hadoop versionHadoop 2.2.0Subversion https://svn.apache.org/repos/asf/hadoop/common -r 1529768Compiled by hortonmu on 2013-10-07T06:28ZCompiled with protoc 2.5.0From source with checksum 79e53ce7994d1628b240f09af91e1af4This command was run using /usr/local/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jarAt this point, hadoop is installed.
Configure Hadoop
$ cd /usr/local/hadoop/etc/hadoop$ vi core-site.xml#Paste following between <configuration> fs.default.name hdfs://localhost:9000$ vi yarn-site.xml#Paste following between <configuration> yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.aux-services.mapreduce.shuffle.class org.apache.hadoop.mapred.ShuffleHandler$ mv mapred-site.xml.template mapred-site.xml$ vi mapred-site.xml#Paste following between <configuration> mapreduce.framework.name yarn$ cd ~$ mkdir -p mydata/hdfs/namenode$ mkdir -p mydata/hdfs/datanode$ cd /usr/local/hadoop/etc/hadoop$ vi hdfs-site.xmlPaste following between <configuration> tag dfs.replication 1 dfs.namenode.name.dir file:/home/hduser/mydata/hdfs/namenode dfs.datanode.data.dir file:/home/hduser/mydata/hdfs/datanode
Format Namenode
hduser@ubuntu40:~$ hdfs namenode -format
Start Hadoop Service
$ start-dfs.sh....$ start-yarn.sh....hduser@ubuntu40:~$ jpsIf everything is sucessful, you should see following services running2583 DataNode2970 ResourceManager3461 Jps3177 NodeManager2361 NameNode2840 SecondaryNameNode
Run Hadoop Example
hduser@ubuntu: cd /usr/local/hadoophduser@ubuntu:/usr/local/hadoop$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5Number of Maps = 2Samples per Map = 513/10/21 18:41:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableWrote input for Map #0Wrote input for Map #1Starting Job13/10/21 18:41:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:803213/10/21 18:41:04 INFO input.FileInputFormat: Total input paths to process : 213/10/21 18:41:04 INFO mapreduce.JobSubmitter: number of splits:213/10/21 18:41:04 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name...
Note: ericduq has created a shell script (make-single-node.sh) for this setup and it is available at git repo athttps://github.com/ericduq/hadoop-scripts.
What to read next
Hadoop FileSystem (HDFS) Tutorial 1
Hadoop 2.x Core (HDFS and YARN) Components Explained
Hadoop Wordcount example
0 0
- hadoop2.2.0 singlenode 配置
- hadoop2.2.0 配置
- 安装配置hadoop2.2.0
- hadoop2.2.0 XML配置
- hadoop2.2.0的配置
- Hadoop2.2.0伪分布式配置
- Hadoop2.2.0完全分布式配置
- Hadoop2.2.0中配置SecondaryNameNode
- Hadoop2.2.0集群安装配置
- Eclipse配置hadoop2.2.0环境
- Hadoop2.2.0 集群配置攻略
- centos安装和配置hadoop2.2.0
- Hadoop2.2.0伪分布式安装配置
- Hadoop2.2.0单机伪分布式安装配置
- Hadoop2.2.0+Hbase.96配置过程
- hadoop2.2.0 伪分布环境配置
- hadoop2.2.0集群安装和配置
- Linux下配置hadoop2.2.0集群
- 另一种眼光看事物
- SetTimer使用方法
- 杭电2089题(用打表法)
- HDU3367
- 博客终于开通啦
- hadoop2.2.0 singlenode 配置
- Cocos2d-x设计模式发掘之一:单例模式
- js date 格式
- Oracle Logminer 做数据恢复 说明示例
- 十六周---指针填空
- Ubuntu下编译androidJNI helloJNI
- 红外遥控系统原理及单片机软件解码程序,我的编写经历(C版本)
- Cocos2d-x设计模式发掘之二:二段构建模式
- java的System.property()可以得到的值