Hadoop-Fedora24搭建Hadoop
来源:互联网 发布:北京婚纱摄影 知乎 编辑:程序博客网 时间:2024/06/09 00:48
1. 安装Java环境并配置好环境变量 /etc/profile末尾追加
export JAVA_HOME=/opt/java
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH:$HOME/bin
2. 新建用户组及用户名 (用于隔离环境)
groupadd hadoop
useradd hduser -g hadoop
(userdel test 删除用户)
3. 为hduser配置ssh
user@ubuntu:~$ su - hduser
hduser@ubuntu:~$ ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hduser/.ssh/id_rsa):
Created directory '/home/hduser/.ssh'.
Your identification has been saved in /home/hduser/.ssh/id_rsa.
Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.
The key fingerprint is: 9b:82:ea:58:b4:e0:35:d7:ff:19:66:a6:ef:ae:0e:d2
hduser@ubuntu The key's randomart image is: [...snipp...]
hduser@ubuntu:~$
5. 安装Hadoop
从hadoop官网上下载
$ cd /opt
$ sudo tar xzvf hadoop-1.2.1.tar.gz
$ sudo mv hadoop-1.2.1 hadoop
$ sudo chown -R hduser:hadoop hadoop
6. 修改$HOME/.bashrc
# Set Hadoop-related environment variables
export HADOOP_HOME=/opt/hadoop
# Set JAVA_HOME (we will also configure JAVA_HOME directly for Hadoop later on)
export JAVA_HOME=/opt/java
# Some convenient aliases and functions for running Hadoop-related commands
unalias fs &> /dev/null
alias fs="hadoop fs"
unalias hls &> /dev/null
alias hls="fs -ls"
# If you have LZO compression enabled in your Hadoop cluster and
# compress job outputs with LZOP (not covered in this tutorial):
# Conveniently inspect an LZOP compressed file from the command
# line; run via:
#
# $ lzohead /hdfs/path/to/lzop/compressed/file.lzo
#
# Requires installed 'lzop' command.
#
lzohead () {
hadoop fs -cat $1 | lzop -dc | head -1000 | less
}
# Add Hadoop bin/ directory to PATH
export PATH=$PATH:$HADOOP_HOME/bin
7. HDFS配置
(1)/opt/hadoop/conf/hadoop-env.sh
# The java implementation to use. Required.
export JAVA_HOME=/opt/java
(2)hadoop.tmp.dir
$ sudo mkdir -p /app/hadoop/tmp
$ sudo chown hduser:hadoop /app/hadoop/tmp
# ...and if you want to tighten up security, chmod from 755 to 750...
$ sudo chmod 750 /app/hadoop/tmp
(3)conf/*-site.xml
配置详见
8.Formatting the HDFS filesystem via the NameNode
/opt/hadoop/bin/hadoop namenode -format
9.启动 start-all.sh
10.关闭 stop-all.sh
11.jps(Java Virtual Machine Process Status Tool) 查看当前运行的java进程
12. wordcount示例
export JAVA_HOME=/opt/java
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH:$HOME/bin
2. 新建用户组及用户名 (用于隔离环境)
groupadd hadoop
useradd hduser -g hadoop
(userdel test 删除用户)
3. 为hduser配置ssh
user@ubuntu:~$ su - hduser
hduser@ubuntu:~$ ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hduser/.ssh/id_rsa):
Created directory '/home/hduser/.ssh'.
Your identification has been saved in /home/hduser/.ssh/id_rsa.
Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.
The key fingerprint is: 9b:82:ea:58:b4:e0:35:d7:ff:19:66:a6:ef:ae:0e:d2
hduser@ubuntu The key's randomart image is: [...snipp...]
hduser@ubuntu:~$
Second, you have to enable SSH access to your local machine with this newly created key.
hduser@ubuntu:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
ssh localhost
- chmod 700 ~/.ssh
- chmod 600 ~/.ssh/*
5. 安装Hadoop
从hadoop官网上下载
$ cd /opt
$ sudo tar xzvf hadoop-1.2.1.tar.gz
$ sudo mv hadoop-1.2.1 hadoop
$ sudo chown -R hduser:hadoop hadoop
6. 修改$HOME/.bashrc
# Set Hadoop-related environment variables
export HADOOP_HOME=/opt/hadoop
# Set JAVA_HOME (we will also configure JAVA_HOME directly for Hadoop later on)
export JAVA_HOME=/opt/java
# Some convenient aliases and functions for running Hadoop-related commands
unalias fs &> /dev/null
alias fs="hadoop fs"
unalias hls &> /dev/null
alias hls="fs -ls"
# If you have LZO compression enabled in your Hadoop cluster and
# compress job outputs with LZOP (not covered in this tutorial):
# Conveniently inspect an LZOP compressed file from the command
# line; run via:
#
# $ lzohead /hdfs/path/to/lzop/compressed/file.lzo
#
# Requires installed 'lzop' command.
#
lzohead () {
hadoop fs -cat $1 | lzop -dc | head -1000 | less
}
# Add Hadoop bin/ directory to PATH
export PATH=$PATH:$HADOOP_HOME/bin
7. HDFS配置
(1)/opt/hadoop/conf/hadoop-env.sh
# The java implementation to use. Required.
export JAVA_HOME=/opt/java
(2)hadoop.tmp.dir
$ sudo mkdir -p /app/hadoop/tmp
$ sudo chown hduser:hadoop /app/hadoop/tmp
# ...and if you want to tighten up security, chmod from 755 to 750...
$ sudo chmod 750 /app/hadoop/tmp
(3)conf/*-site.xml
配置详见
8.Formatting the HDFS filesystem via the NameNode
/opt/hadoop/bin/hadoop namenode -format
9.启动 start-all.sh
10.关闭 stop-all.sh
11.jps(Java Virtual Machine Process Status Tool) 查看当前运行的java进程
12. wordcount示例
1
上一篇:hadoop-datanode启动不起来解决方案
下一篇:Fedora-sudoers中添加用户
相关热门文章
- Hadoop集群完全分布式模式环境...
- 大数据之快速搭建hadoop2.6集...
- Zookeeper分布式集群部署...
- Hadoop初探之Hadoop生态圈...
- Hadoop学习全程记录——在Ecli...
- linux dhcp peizhi roc
- 关于Unix文件的软链接
- 求教这个命令什么意思,我是新...
- sed -e "/grep/d" 是什么意思...
- 谁能够帮我解决LINUX 2.6 10...
给主人留下些什么吧!~~
评论热议
0 0
- Hadoop-Fedora24搭建Hadoop
- hadoop集群搭建(hadoop)
- Hadoop-1.Hadoop搭建
- HADOOP: 搭建hadoop集群
- hadoop 搭建
- hadoop搭建
- hadoop搭建
- hadoop搭建
- Hadoop搭建
- hadoop搭建
- 搭建hadoop
- hadoop搭建
- Hadoop搭建
- Hadoop搭建
- hadoop搭建
- hadoop 搭建
- Hadoop搭建
- hadoop 搭建
- Android-无线调试
- Fedora-卸载OpenJDK安装sunjdk
- Fedora-ipv6关闭方法
- input选择框 勾选 不勾选 数组的操作
- 串口通信的基本知识
- Hadoop-Fedora24搭建Hadoop
- 关于String、Int和Object类型之间的转换
- Fedora-sudoers中添加用户
- Hadoop-将python代码运行在hadoop上
- Hadoop-常用shell命令
- 最小生成树(MST)的性质及算法 [转】
- Java代码的35个优化
- $.each
- 51nod 1006 最长公共子序列Lcs【LCS】
原创粉丝点击
热门IT博客
热门问题
老师的惩罚
人脸识别
我在镇武司摸鱼那些年
重生之率土为王
我在大康的咸鱼生活
盘龙之生命进化
天生仙种
凡人之先天五行
春回大明朝
姑娘不必设防,我是瞎子
洋葱切法图解
洋葱木耳炒鸡蛋
洋葱怎么切丝图解
洋葱的英语单词
洋葱什么时候种植
洋葱可以生吃吗
洋葱汁的做法
洋葱炒土豆片
怀孕可以吃洋葱吗
清炒洋葱的做法
洋葱种子图片
洋葱的种植时间和方法
生洋葱的功效与作用
洋葱怎么做好吃又营养
洋葱炒鸡蛋有毒吗
生洋葱的吃法
洋葱种植图片
洋葱炒蛋怎么做
洋葱什么时候收获
洋葱炒肉怎么做
炒洋葱可以搭配什么菜
洋葱炒蛋有毒
洋葱做法大全带图解
洋葱生吃的做法
出口洋葱后期行情
怎样炒洋葱好吃
洋葱拌木耳的做法
洋葱孕妇可以吃吗
洋葱炒肉丝的家常做法
洋葱汁的功效与作用
洋葱英文发音
洋葱木耳怎么做好吃
洋葱去甲醛吗
洋葱放在房间的好处
洋葱怎么切丝
洋葱长在哪里
洋葱英文怎么读
洋葱配什么菜炒好吃
洋葱泡红酒的功效与作用
洋葱可以炒什么
洋葱的功效与作用及食用方法