程序博客网 > 淘宝举报被仿冒品牌

hadoop/hbase 环境搭建

来源：互联网 发布：淘宝举报被仿冒品牌 编辑：程序博客网 时间：2024/06/08 07:26

hadoop安装

（1）创建hadoop用户组和用户 见（1.1，1.5）

（2）配置hadoop用户信任关系

1）生成 非对称密钥 ：ssh-keygen -t rsa //一直回车

  

1）znode01,znode02,znode03 添加信任关系

(1)znode01 无密码访问znode02

(2)在znode02 ：mkdir ~/.ssh/tmp ; touch ~/.ssh/authorized_keys;

(3)在znode01 ：scp  ~/.ssh/id_rsa.pub hdpusr@znode02:~/.ssh/tmp //此时输入znode02密码即可

(4)在znode02 ：cat ~/.ssh/tmp/id_rsa.pub >> ~/.ssh/authorized_keys

(5)znode01 ssh 无密码访问自己 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

......重复以上相似操作配置其他机器

 

（3）安装包hadoop-2.6.0.zip准备

mkdir ~/env;mkdir ~/env/hadoop；将hadoop-2.6.0.zip解压到hadoop

 

（4）配置文件修改

（1）修改hadoop-env.sh

 配置JAVA_HOME:   export JAVA_HOME=/usr/software/jdk1.7.0_79

（2）修改core-site.xml

   <configuration>

        <property>

        <name>fs.defaultFS</name>

        <value>hdfs://hdpcls1</value>

    </property>

        <property>

        <name>io.file.buffer.size</name>

        <value>131072</value>

    </property>

    <property>

        <name>hadoop.tmp.dir</name>

        <value>/home/hdpusr/env/hadoop/tmp</value>

    </property>

        <property>

        <name>ha.zookeeper.quorum</name>

        <value>znode01:29181,znode02:29181,znode03:29181</value>

    </property>

</configuration>

 

（3）修改hdfs-stie.xml

<configuration>

 

<property>

<name>dfs.nameservices</name>

<value>hdpcls1</value>

</property>

 

 

<property>

<name>dfs.ha.namenodes.hdpcls1</name>

<value>nn1,nn2</value>

</property>

 

<property>

<name>dfs.namenode.rpc-address.hdpcls1.nn1</name>

<value>znode01:8920</value>

</property>

 

<property>

<name>dfs.namenode.http-address.hdpcls1.nn1</name>

<value>znode01:59070</value>

</property>

 

<property>

<name>dfs.namenode.rpc-address.hdpcls1.nn2</name>

<value>znode02:8920</value>

</property>

 

<property>

<name>dfs.namenode.http-address.hdpcls1.nn2</name>

<value>znode02:59070</value>

</property>

 

<property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://znode01:8485;znode02:8485;znode03:8485/hdpcls1</value>

</property>

 

<property>

<name>dfs.journalnode.edits.dir</name>

<value>/home/hdpusr/env/hadoop/dfs/jn</value>

</property>

 

<property>

  <name>dfs.journalnode.rpc-address</name>

  <value>0.0.0.0:8485</value>

</property>

 

<property>

  <name>dfs.journalnode.http-address</name>

  <value>0.0.0.0:8480</value>

</property>

 

<property>

  <name>dfs.journalnode.https-address</name>

  <value>0.0.0.0:8481</value>

</property>

 

 

<property>

<name>dfs.ha.automatic-failover.enabled</name>

<value>true</value>

</property>

 

<property>

<name>dfs.client.failover.proxy.provider.hdpcls1</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

 

<property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence</value>

</property>

 

<property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

<value>/home/hdpusr/.ssh/id_rsa</value>

</property>

 

<property>

<name>dfs.ha.fencing.ssh.connect-timeout</name>

<value>30000</value>

</property>

 

<property>

  <name>dfs.datanode.data.dir</name>

  <value>file:/home/hdpusr/env/hadoop/dfs/dn/data01</value>

</property>

</configuration>

（4）修改mapred-site.xml

<configuration>

 

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

</configuration>

 

（5）修改yarn-site.xml

<configuration>

 

<property>

<name>yarn.resourcemanager.ha.enabled</name>

<value>true</value>

</property>

 

<property>

<name>yarn.resourcemanager.cluster-id</name>

<value>rmcls1</value>

</property>

 

<property>

<name>yarn.resourcemanager.ha.rm-ids</name>

<value>rm1,rm2</value>

</property>

 

<property>

<name>yarn.resourcemanager.hostname.rm1</name>

<value>znode01</value>

</property>

<property>

<name>yarn.resourcemanager.hostname.rm2</name>

<value>znode02</value>

</property>

<property>

<name>yarn.resourcemanager.recovery.enabled</name>

<value>true</value>

</property>

<property>

<name>yarn.resourcemanager.store.class</name>

<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>

</property>

 

<property>

<name>yarn.resourcemanager.zk-address</name>

<value>znode01:29181,znode02:29181,znode03:29181</value>

</property>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

（6）修改slaves

vim slaves

 

添加

znode02

znode03

 

（7）创建hadoop目录

mkdir -p ~/env/hadoop/{tmp,dfs/jn,dfs/dn/data01}

（5）环境变量配置

su - hdpusr01

vim ~/.bash_profile

 

export HADOOP_HOME=${HOME}/env/hadoop

export HADOOP_CONF_DIR=$HOME/env/hadoop/etc/hadoop

#export HADOOP_PID_DIR=$HOME/env/hadoop/pid

export HADOOP_LOG_DIR=$HOME/env/hadoop/logs

export HADOOP_LIBEXEC_DIR=$HOME/env/hadoop/libexec

export HADOOP_PREFIX=${HADOOP_HOME}

export HADOOP_COMMON_HOME=${HADOOP_HOME}

export HADOOP_HDFS_HOME=${HADOOP_HOME}

export HADOOP_LIB=${HADOOP_HOME}/lib

export HADOOP_LIBRARY_PATH=${HADOOP_HOME}/lib/native

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HADOOP_HOME}/lib/native

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

export HADOOP_MAPRED_HOME=${HADOOP_HOME}

export HADOOP_YARN_USER=pochdp04

export YARN_CONF_DIR=${HADOOP_CONF_DIR}

export YARN_HOME=${HADOOP_HOME}

export HADOOP_YARN_HOME=${HADOOP_HOME}

export YARN_LOG_DIR=${HADOOP_LOG_DIR}

export YARN_PID_DIR=${HADOOP_CONF_DIR}/../yarn

 

 

source ~/.bash_profile 使环境变量生效

（6）格式化操作，启动停止操作

(1) 保证zk正常运行

(2)znode01,znode02,znode03 启动sbin目录下执行：./hadoop-daemon.sh start journalnode

28365 JournalNode --正常的情况可以看到该进程

        

（3）格式化hdfs

znode01 节点bin目录执行：./hdfs namenode -format

显示：

15/12/18 13:34:35 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1518459121-10.46.52.30-1450416875417

15/12/18 13:34:35 INFO common.Storage: Storage directory /aifs01/users/hdpusr01/hadoop/tmp/dfs/name has been successfully formatted.

15/12/18 13:34:35 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

15/12/18 13:34:35 INFO util.ExitUtil: Exiting with status 0

 

【将/home/hdpusr/env/hadoop/tmp 拷贝之znode02 /home/hdpusr/env/hadoop/tmp 下】

（4）格式化ZK

znode01 节点bin目录执行：./hdfs zkfc -formatZK

 

显示：

15/12/18 13:49:04 INFO ha.ActiveStandbyElector: Session connected.

15/12/18 13:49:04 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/hdpcls1 in ZK.

15/12/18 13:49:04 INFO zookeeper.ZooKeeper: Session: 0x251b09471e80001 closed

15/12/18 13:49:04 INFO zookeeper.ClientCnxn: EventThread shut down

 

成功后zk 出现节点 hadoop-ha

[zk: 127.0.0.1:29181(CONNECTED) 1] ls /

[consumers, config, hadoop-ha, zookeeper, brokers, admin, controller_epoch]

 

（5）启动分布式存储 znode01 sbin上执行： ./start-dfs.sh

启动成功

znode01 进程：

15507 NameNode

14371 JournalNode

23478 DFSZKFailoverController

znode02 进程：

23870 DataNode

23474 JournalNode

23998 DFSZKFailoverController

znode03 进程

16268 JournalNode

16550 DataNode

同时zk注册节点

[zk: 127.0.0.1:29181(CONNECTED) 4] ls /hadoop-ha/hdpcls1

[ActiveBreadCrumb, ActiveStandbyElectorLock]

 

此时可在页面输入http://118.190.79.15:59070 访问

（6）启动分布式资源管理

znode01 sbin上执行：./start-yarn.sh

此时znode01,

出现进程：17746 ResourceManager

znode02

出现进程：24975 NodeManager

znode03

 出现进程17389 NodeManager

 

Zk注册节点 ：rmstore, yarn-leader-election

[zk: 127.0.0.1:29181(CONNECTED) 0] ls /

[hadoop-ha, admin, zookeeper, consumers, config, rmstore, yarn-leader-election, brokers, controller_epoch]

 

 

(7) 全部启动：./start-all.sh，全部停止./stop-all.sh

 

[hdpusr@znode01 sbin]$ ./stop-all.sh

This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh

17/03/25 09:56:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Stopping namenodes on [znode01 znode02]

znode02: no namenode to stop

znode01: stopping namenode

znode03: stopping datanode

znode02: stopping datanode

Stopping journal nodes [znode01 znode02 znode03]

znode03: stopping journalnode

znode02: stopping journalnode

znode01: stopping journalnode

17/03/25 09:57:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Stopping ZK Failover Controllers on NN hosts [znode01 znode02]

znode02: stopping zkfc

znode01: no zkfc to stop

stopping yarn daemons

stopping resourcemanager

znode02: stopping nodemanager

znode03: stopping nodemanager

no proxyserver to stop

[hdpusr@znode01 sbin]$ jps

22992 Jps

[hdpusr@znode01 sbin]$ start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

17/03/25 09:57:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting namenodes on [znode01 znode02]

znode02: starting namenode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-namenode-znode02.out

znode01: starting namenode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-namenode-znode01.out

znode03: starting datanode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-datanode-znode03.out

znode02: starting datanode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-datanode-znode02.out

Starting journal nodes [znode01 znode02 znode03]

znode02: starting journalnode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-journalnode-znode02.out

znode03: starting journalnode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-journalnode-znode03.out

znode01: starting journalnode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-journalnode-znode01.out

17/03/25 09:57:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting ZK Failover Controllers on NN hosts [znode01 znode02]

znode02: starting zkfc, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-zkfc-znode02.out

znode01: starting zkfc, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-zkfc-znode01.out

starting yarn daemons

starting resourcemanager, logging to /home/hdpusr/env/hadoop/logs/yarn-hdpusr-resourcemanager-znode01.out

znode03: starting nodemanager, logging to /home/hdpusr/env/hadoop/logs/yarn-hdpusr-nodemanager-znode03.out

znode02: starting nodemanager, logging to /home/hdpusr/env/hadoop/logs/yarn-hdpusr-nodemanager-znode02.out

 

（8）查看集群状态 bin目录./hdfs dfsadmin -report

（9）http://118.190.79.15:59070/dfshealth.html#tab-overview


//////////////////////////////////////////////////////////////

***********************************************hadoop完全分布式部署**********************************************
0.集群规划
主机名 ip地址 安装的软件 运行的进程
HBS01 10.46.52.30hadoop,hbase namenode,zkfc,resourcemanager
HBS02 10.46.52.31hadoop namenode,zkfc,resourcemanager
HBS03 10.46.52.32hadoop,hbase datanode
HBS04 10.46.52.33hadoop,zookeeper,hbase datanode,nodemanager,journalnode
HBS05 10.46.52.34 hadoop,zookeeper,hbase datanode,nodemanager,journalnode
HBS06 10.46.52.35hadoop,zookeeper,hbase datanode,nodemanager,journalnode




1.创建用户以及配置主机名
mkdir -p /zn/users
useradd -u 351   -g hadoop -G ibss -d /zn/users/hdpusr01 -m hdpusr01
passwd hdpusr01     --stdin <<< Linuhdp_0805
echo -e "\n. ~puwadm/wprofile\n" >> ~hdpusr01/.bash_profile 


useradd -u 352   -g hbase -G ibss -d /zn/users/hbsusr01 -m hbsusr01
passwd hbsusr01     --stdin <<< Linuhbs_0805
echo -e "\n. ~puwadm/wprofile\n" >> ~hbsusr01/.bash_profile 




vi /etc/hosts            
10.46.52.30 HBS01
10.46.52.31 HBS02
10.46.52.32 HBS03
10.46.52.33 HBS04
10.46.52.34 HBS05
10.46.52.35 HBS06


2.配置互信
--每台主机都需要操作
su - hdpusr01
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
chmod 600 .ssh/authorized_keys


--选择一台master操作(HBS01)
ssh HBS02 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 
ssh HBS03 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 
ssh HBS04 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 
ssh HBS05 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 
ssh HBS06 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 


scp ~/.ssh/authorized_keys HBS02:/zn/users/hdpusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS03:/zn/users/hdpusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS04:/zn/users/hdpusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS05:/zn/users/hdpusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS06:/zn/users/hdpusr01/.ssh/authorized_keys


--验证配置是否生效
ssh HBS01 date
ssh HBS02 date
ssh HBS03 date
ssh HBS04 date
ssh HBS05 date
ssh HBS06 date


3.安装jdk
which java




4.安装Hadoop
4.1 安装zookkeper集群(HBS04 - HBS06)
--配置zookeeper集群
[root@HBS04 hdpusr01]# tar xvf /opt/software/zookeeper.ccs01.tgz
[root@HBS04 hdpusr01]# chown -R hdpusr01:hadoop zookeeper/
[root@HBS04 hdpusr01]# rm -rf zookeeper/logs/*
[root@HBS04 hdpusr01]# rm -rf zookeeper/zk-data/version-2/
[root@HBS04 hdpusr01]# rm -rf zookeeper/zk-data/zookeeper_server.pid 
[root@HBS04 hdpusr01]# vi zookeeper/conf/zoo.cfg
dataDir=/zn/users/hdpusr01/zookeeper/zk-data
server.1=HBS04:29888:39889
server.2=HBS05:29888:39889
server.3=HBS06:29888:39889


hdpusr01@HBS04:/zn/users/hdpusr01> vi zookeeper/bin/zkEnv.sh
export ZOOCFGDIR=/zn/users/hdpusr01/zookeeper/conf
export ZOO_LOG_DIR=/zn/users/hdpusr01/zookeeper/logs


[root@HBS04 hdpusr01]# su - hdpusr01
hdpusr01@HBS04:/zn/users/hdpusr01> tar cvf zkper.tar zookeeper/
hdpusr01@HBS04:/zn/users/hdpusr01> scp zkper.tar HBS05:/zn/users/hdpusr01                                                                                                                           
hdpusr01@HBS04:/zn/users/hdpusr01> scp zkper.tar HBS06:/zn/users/hdpusr01


hdpusr01@HBS05:/zn/users/hdpusr01> tar xvf zkper.tar
hdpusr01@HBS05:/zn/users/hdpusr01> echo 2 >zookeeper/zk-data/myid


hdpusr01@HBS06:/zn/users/hdpusr01> tar xvf zkper.tar
hdpusr01@HBS06:/zn/users/hdpusr01> echo 3 >zookeeper/zk-data/myid


--启动zookeeper
hdpusr01@HBS04:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh start
hdpusr01@HBS05:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh start
hdpusr01@HBS06:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh start




4.2 安装hadoop集群（AI-OPT-HBS01 - HBS06）
[root@HBS01 hdpusr01]# tar xvf /opt/software/hadoop-2.6.0.tgz
[root@HBS01 hdpusr01]# chown -R hdpusr01:hadoop hadoop-2.6.0/
hdpusr01@HBS01:/zn/users/hdpusr01> mv hadoop-2.6.0 hadoop


--修改hadoop-env.sh
hdpusr01@HBS01:/zn/users/hdpusr01> vi hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/opt/freeware/jdk1.7.0_79


--修改core-site.xml
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/etc/hadoop> vi core-site.xml
<configuration>
        <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hdpcls1</value>
    </property>
        <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/zn/users/hdpusr01/hadoop/tmp</value>
    </property>
        <property>
        <name>ha.zookeeper.quorum</name>
        <value>HBS04:29181,HBS05:29181,HBS06:29181</value>
    </property>
</configuration>




--修改hdfs-stie.xml
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/etc/hadoop> vi hdfs-site.xml
<configuration>


<property>
<name>dfs.nameservices</name>
<value>hdpcls1</value>
</property>




<property>
<name>dfs.ha.namenodes.hdpcls1</name>
<value>nn1,nn2</value>
</property>



<property>
<name>dfs.namenode.rpc-address.hdpcls1.nn1</name>
<value>HBS01:8920</value>
</property>


<property>
<name>dfs.namenode.http-address.hdpcls1.nn1</name>
<value>HBS01:59070</value>
</property>


<property>
<name>dfs.namenode.rpc-address.hdpcls1.nn2</name>
<value>HBS02:8920</value>
</property>


<property>
<name>dfs.namenode.http-address.hdpcls1.nn2</name>
<value>HBS02:59070</value>
</property>


<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://HBS04:8485;HBS05:8485;HBS06:8485/hdpcls1</value>
</property>


<property>
<name>dfs.journalnode.edits.dir</name>
<value>/zn/users/hdpusr01/hadoop/dfs/jn</value>
</property>


<property>
  <name>dfs.journalnode.rpc-address</name>
  <value>0.0.0.0:8485</value>
</property>


<property>
  <name>dfs.journalnode.http-address</name>
  <value>0.0.0.0:8480</value>
</property>


<property>
  <name>dfs.journalnode.https-address</name>
  <value>0.0.0.0:8481</value>
</property>




<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>


<property>
<name>dfs.client.failover.proxy.provider.hdpcls1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>


<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>


<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/zn/users/hdpusr01/.ssh/id_rsa</value>
</property>


<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>


<property>
  <name>dfs.datanode.data.dir</name>
  <value>file:/zn/users/hdpusr01/hadoop/dfs/dn/data01</value>
</property>
</configuration>
 
--修改mapred-site.xml
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/etc/hadoop> vi mapred-site.xml
<configuration>


<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
 
--修改yarn-site.xml
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/etc/hadoop>  vi yarn-site.xml
<configuration>


<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>


<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>rmcls1</value>
</property>


<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>


<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>HBS01</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>HBS02</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>


<property>
<name>yarn.resourcemanager.zk-address</name>
<value>HBS04:29181,HBS05:29181,HBS06:29181</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>




--修改slaves
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/etc/hadoop> vi slaves 
HBS03
HBS04
HBS05
HBS06


--创建hadoop目录
hdpusr01@HBS01:/zn/users/hdpusr01> mkdir -p hadoop/{tmp,dfs/jn,dfs/dn/data01}




--将配置好的hadoop复制到其他节点
hdpusr01@HBS01:/zn/users/hdpusr01> tar cvf  hdp.tar hadoop
hdpusr01@HBS04:/zn/users/hdpusr01> scp  hdp.tar HBS02:/zn/users/hdpusr01  
hdpusr01@HBS04:/zn/users/hdpusr01> scp  hdp.tar HBS03:/zn/users/hdpusr01  
hdpusr01@HBS04:/zn/users/hdpusr01> scp  hdp.tar HBS04:/zn/users/hdpusr01  
hdpusr01@HBS04:/zn/users/hdpusr01> scp  hdp.tar HBS05:/zn/users/hdpusr01  
hdpusr01@HBS04:/zn/users/hdpusr01> scp  hdp.tar HBS06:/zn/users/hdpusr01  




--解压hadoop包（AI-OPT-HBS02 - HBS05）
tar xvf hdp.tar






--查看zookeeper集群状态（AI-OPT-HBS04 - HBS06）
hdpusr01@HBS04:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /zn/users/hdpusr01/zookeeper/conf/zoo.cfg
Mode: follower


hdpusr01@HBS05:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /zn/users/hdpusr01/zookeeper/conf/zoo.cfg
Mode: leader


hdpusr01@HBS06:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /zn/users/hdpusr01/zookeeper/conf/zoo.cfg
Mode: follower


--启动journalnode（AI-OPT-HBS04 - HBS06）
hdpusr01@HBS04:/zn/users/hdpusr01/hadoop/sbin> ./hadoop-daemon.sh start journalnode
hdpusr01@HBS04:/zn/users/hdpusr01/hadoop/sbin> jps
27035 QuorumPeerMain
28365 JournalNode --正常的情况可以看到该进程




--格式化HDFS（AI-OPT-HBS01）
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/bin> ./hdfs namenode -format
15/12/18 13:34:35 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1518459121-10.46.52.30-1450416875417
15/12/18 13:34:35 INFO common.Storage: Storage directory /zn/users/hdpusr01/hadoop/tmp/dfs/name has been successfully formatted.
15/12/18 13:34:35 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
15/12/18 13:34:35 INFO util.ExitUtil: Exiting with status 0


--拷贝/zn/users/hdpusr01/hadoop/tmp（AI-OPT-HBS01）
hdpusr01@HBS01:/zn/users/hdpusr01> scp -r /zn/users/hdpusr01/hadoop/tmp/* HBS02:/zn/users/hdpusr01/hadoop/tmp




--格式化ZK（AI-OPT-HBS01）
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/bin> ./hdfs zkfc -formatZK
15/12/18 13:49:04 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=HBS04:29181,HBS05:29181,HBS06:29181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@6456c5f4
15/12/18 13:49:04 INFO zookeeper.ClientCnxn: Opening socket connection to server HBS05/10.46.52.34:29181. Will not attempt to authenticate using SASL (unknown error)
15/12/18 13:49:04 INFO zookeeper.ClientCnxn: Socket connection established to HBS05/10.46.52.34:29181, initiating session
15/12/18 13:49:04 INFO zookeeper.ClientCnxn: Session establishment complete on server HBS05/10.46.52.34:29181, sessionid = 0x251b09471e80001, negotiated timeout = 80000
15/12/18 13:49:04 INFO ha.ActiveStandbyElector: Session connected.
15/12/18 13:49:04 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/hdpcls1 in ZK.
15/12/18 13:49:04 INFO zookeeper.ZooKeeper: Session: 0x251b09471e80001 closed
15/12/18 13:49:04 INFO zookeeper.ClientCnxn: EventThread shut down


/* 在格式化ZK后zookeeper中多出的znode */
[zk: 10.46.52.34:29181(CONNECTED) 0] ls /
[zookeeper]
[zk: 10.46.52.34:29181(CONNECTED) 7] ls /
[hadoop-ha, zookeeper]
[zk: 10.46.52.34:29181(CONNECTED) 8] ls /hadoop-ha
[hdpcls1]
[zk: 10.46.52.34:29181(CONNECTED) 9] ls /hadoop-ha/hdpcls1
[]




--启动HDFS（AI-OPT-HBS01）
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/sbin> ./start-dfs.sh
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/sbin> jps
29964 DFSZKFailoverController
29657 NameNode


hdpusr01@HBS02:/zn/users/hdpusr01/hadoop> jps
28086 NameNode
28185 DFSZKFailoverController


hdpusr01@HBS03:/zn/users/hdpusr01/hadoop/etc/hadoop> jps
27826 DataNode    --（AI-OPT-HBS03 - HBS06）




/* 在启动HDFS后zookeeper中多出的znode */
[zk: 10.46.52.34:29181(CONNECTED) 15] ls /hadoop-ha/hdpcls1
[ActiveBreadCrumb, ActiveStandbyElectorLock]
[zk: 10.46.52.34:29181(CONNECTED) 16] ls /hadoop-ha/hdpcls1/ActiveBreadCrumb
[]
[zk: 10.46.52.34:29181(CONNECTED) 17] ls /hadoop-ha/hdpcls1/ActiveStandbyElectorLock
[]
[zk: 10.46.52.34:29181(CONNECTED) 18] get /hadoop-ha/hdpcls1/ActiveBreadCrumb


hdpcls1nn1
          HBS01 ?E(?>
cZxid = 0x10000000a
ctime = Fri Dec 18 13:55:41 CST 2015
mZxid = 0x10000000a
mtime = Fri Dec 18 13:55:41 CST 2015
pZxid = 0x10000000a
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 34
numChildren = 0
[zk: 10.46.52.34:29181(CONNECTED) 19] get /hadoop-ha/hdpcls1/ActiveStandbyElectorLock


hdpcls1nn1
          HBS01 ?E(?>
cZxid = 0x100000008
ctime = Fri Dec 18 13:55:41 CST 2015
mZxid = 0x100000008
mtime = Fri Dec 18 13:55:41 CST 2015
pZxid = 0x100000008
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x351b09482220000
dataLength = 34
numChildren = 0




--启动YARN(HBS01由于resourcemanager进程占用资源较多，机器充足的情况下可以考虑将其与namenode进程分开）
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/sbin> ./start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-resourcemanager-HBS01.out
HBS03: starting nodemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-nodemanager-HBS03.out
HBS04: starting nodemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-nodemanager-HBS04.out
HBS05: starting nodemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-nodemanager-HBS05.out
HBS06: starting nodemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-nodemanager-HBS06.out
注：这个时候主节点上的resourcemanager进程还没启动，从节点上的nodemanager进程已经启动
hdpusr01@HBS03:/zn/users/hdpusr01/hadoop/etc/hadoop> jps
28004 NodeManager --（AI-OPT-HBS03 - HBS06）


--启动resourcemanager
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/sbin> ./yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-resourcemanager-HBS01.out
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/sbin> jps
30921 ResourceManager  --终于看到这个进程了
注：主机AI-OPT-HBS02上的resourcemanager还没启动，需要手工启动
hdpusr01@HBS02:/zn/users/hdpusr01/hadoop/sbin> ./yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-resourcemanager-HBS02.out
hdpusr01@HBS02:/zn/users/hdpusr01/hadoop/sbin> jps
28945 ResourceManager




/* 在启动YARN和resourcemanager后zookeeper中多出的znode */
[zk: 10.46.52.34:29181(CONNECTED) 22] ls /
[rmstore, yarn-leader-election, hadoop-ha, zookeeper]
[zk: 10.46.52.34:29181(CONNECTED) 23] ls /rmstore
[ZKRMStateRoot]
[zk: 10.46.52.34:29181(CONNECTED) 24] ls /rmstore/ZKRMStateRoot
[AMRMTokenSecretManagerRoot, RMAppRoot, RMVersionNode, RMDTSecretManagerRoot]
[zk: 10.46.52.34:29181(CONNECTED) 25] ls /yarn-leader-election
[rmcls1]
[zk: 10.46.52.34:29181(CONNECTED) 26] ls /yarn-leader-election/rmcls1
[ActiveBreadCrumb, ActiveStandbyElectorLock]








--通过命令查看集群状态
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/bin> ./hdfs dfsadmin -report








5.安装HBase
5.1 安装HBase集群（AI-OPT-HBS01 - HBS06）
--选择一台主机操作（AI-OPT-HBS01）
[root@HBS01 hbsusr01]# tar xvf /opt/software/hbase-1.1.2.tgz
[root@HBS01 hbsusr01]# mv hbase-1.1.2 hbase
[root@HBS01 hbsusr01]# chown -R hbsusr01:hbase hbase




--设置集群节点文件
hbsusr01@HBS01:/zn/users/hbsusr01/hbase/conf> vi regionservers
HBS03
HBS04
HBS05
HBS06




--设置hbase配置文件
hbsusr01@HBS01:/zn/users/hbsusr01/hbase/conf> vi hbase-site.xml 
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://HBS01:8920/hbase</value>
</property>


<property>
<name>hbase.master.port</name>
<value>60900</value>
</property>


<property>
<name>hbase.regionserver.port</name>
<value>60920</value>
</property>


<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>


<property>
<name>hbase.zookeeper.quorum</name>
<value>HBS04,HBS05,HBS06</value>
</property>


<property>
<name>hbase.tmp.dir</name>
<value>/zn/users/hbsusr01/hbase/tmp</value>
</property>


<property>
<name>hbase.zookeeper.peerport</name>
<value>29888</value>
</property>


<property>
<name>hbase.zookeeper.leaderport</name>
<value>39888</value>
</property>


<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>29181</value>
</property>


<property>
<name>hbase.rest.port</name>
<value>8980</value>
</property>
</configuration>






--设置hbase环境
hbsusr01@HBS01:/zn/users/hbsusr01/hbase/conf> vi hbase-env.sh 
export HBASE_CONF_DIR=/zn/users/hbsusr01/hbase/conf
export HBASE_PID_DIR=/zn/users/hbsusr01/hbase/pid
export HBASE_LOG_DIR=/zn/users/hbsusr01/hbase/logs


# export HBASE_MANAGES_ZK=true
export HBASE_MANAGES_ZK=false




--创建相关目录
hbsusr01@HBS01:/zn/users/hbsusr01> mkdir -p hbase/{conf,pid,logs,tmp}




--拷贝文件到其他节点并解压
hbsusr01@HBS01:/zn/users/hbsusr01> tar cvf hbase.tar hbase
hbsusr01@HBS01:/zn/users/hbsusr01> scp hbase.tar HBS03:/zn/users/hbsusr01
hbsusr01@HBS01:/zn/users/hbsusr01> scp hbase.tar HBS04:/zn/users/hbsusr01
hbsusr01@HBS01:/zn/users/hbsusr01> scp hbase.tar HBS05:/zn/users/hbsusr01
hbsusr01@HBS01:/zn/users/hbsusr01> scp hbase.tar HBS06:/zn/users/hbsusr01




                         
--配置ssh互信
su - hbsusr01
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
chmod 600 .ssh/authorized_keys


--选择一台master操作(HBS01)
ssh HBS02 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 
ssh HBS03 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 
ssh HBS04 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 
ssh HBS05 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 
ssh HBS06 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 


scp ~/.ssh/authorized_keys HBS02:/zn/users/hbsusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS03:/zn/users/hbsusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS04:/zn/users/hbsusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS05:/zn/users/hbsusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS06:/zn/users/hbsusr01/.ssh/authorized_keys


--验证配置是否生效
ssh HBS01 date
ssh HBS02 date
ssh HBS03 date
ssh HBS04 date
ssh HBS05 date
ssh HBS06 date


--解压hadoop包（AI-OPT-HBS03 - HBS06）
tar xvf hbase.tar




--启动hbase
start-hbase.sh
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hbsusr01, access=WRITE, inode="/":hdpusr01:supergroup:drwxr-xr-x


解决方法：hadoop fs添加目录和权限
hdpusr01@HBS01:/zn/users/hdpusr01> hadoop fs -mkdir /hbase
-mkdir: java.net.UnknownHostException: host-10-1-241-18


临时解决：
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/bin> ./hadoop  fs -mkdir /hbase
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/bin> ./hadoop fs -chown hbsusr01:hbase /hbase


/*
解决方法：配置环境变量
su - hdpusr01
export HADOOP_CONF_DIR=$HOME/hadoop/etc/hadoop
#export HADOOP_PID_DIR=$HOME/hadoop/pid
export HADOOP_LOG_DIR=$HOME/hadoop/logs
export HADOOP_LIBEXEC_DIR=$HOME/hadoop/libexec
export HADOOP_PREFIX=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export HADOOP_LIB=${HADOOP_HOME}/lib
export HADOOP_LIBRARY_PATH=${HADOOP_HOME}/lib/native
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HADOOP_HOME}/lib/native
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_YARN_USER=pochdp04
export YARN_CONF_DIR=${HADOOP_CONF_DIR}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_YARN_HOME=${HADOOP_HOME}
export YARN_LOG_DIR=${HADOOP_LOG_DIR}
export YARN_PID_DIR=${HADOOP_CONF_DIR}/../yarn
*/




su - hbsusr01
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/zn/users/hdpusr01/hadoop/lib/native
export HBASE_HOME=/zn/users/hbsusr01/hbase
export HBASE_CONF_DIR=$HBASE_HOME/conf
export HBASE_LOG_DIR=$HBASE_HOME/logs
export HBASE_PID_DIR=$HBASE_HOME/pid
export PATH=$PATH:$HBASE_HOME/bin


http://blog.itpub.net/20777547/viewspace-1745820/
http://www.aboutyun.com/thread-11909-1-1.html
http://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop-yarn/






***********************************************hadoop伪分布式部署**********************************************
1.创建用户以及配置主机名
mkdir -p /zn/users
useradd -u 451   -g hadoop -G ibss -d /zn/users/hdpusr02 -m hdpusr02
passwd hdpusr02     --stdin <<< Linuhdp_0805
echo -e "\n. ~puwadm/wprofile\n" >> ~hdpusr02/.bash_profile 




2.配置ssh互相
su - hdpusr02
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh AI-UAT-OPT-HBASE03 date


3.解压文件
tar xvf /opt/software/hadoop-2.6.0.tgz 
mv hadoop-2.6.0/ hadoop


4.配置hadoop
vi hadoop-env.sh
export JAVA_HOME=/opt/freeware/jdk1.7.0_79


vi core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/zn/users/hdpusr01/hadoop/tmp</value>
</property>


<property>
<name>fs.defaultFS</name>
<value>hdfs://AI-UAT-OPT-HBASE03:9000</value>
</property>
</configuration>


vi mapred-site.xml
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>


vi yarn-site.xml 
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>


vi hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>


<property>
<name>dfs.namenode.name.dir</name>
<value>file:/zn/users/hdpusr02/hadoop/dfs/name</value>
</property>


<property>
<name>dfs.datanode.data.dir</name>
<value>file:/zn/users/hdpusr02/hadoop/dfs/data</value>
</property>
</configuration>


vi slaves
AI-UAT-OPT-HBASE03


5.创建相应目录
mkdir -p hadoop/dfs/{name,data}


6.启动hadoop
bin/hdfs namenode -format
sbin/start-dfs.sh
sbin/start-yarn.sh




阅读全文
0 0
淘宝举报被仿冒品牌 淘宝举报被仿冒品牌
原创粉丝点击
热门IT博客
skycc软件怎么样skycc软件怎么样
黑马javascript视频黑马javascript视频
多益 网络面试
录入数据 英文
悟空传说网络不大好
西南财大网络教育图片
梦幻西游mac版
sql isequal
sar指标算法
linux是数据库软件么
中世纪2优化9毛子
网络骑士作品特点
mysql全文索引怎么用
佛山 网络英语培训
广铁集团网络招聘
搞笑自拍软件
网络推广要自己建网站
小程序怎么连接数据库
linux vsftpd 配置
方正兰亭超细黑 mac
热门问题 老师的惩罚 人脸识别 我在镇武司摸鱼那些年 重生之率土为王 我在大康的咸鱼生活 盘龙之生命进化 天生仙种 凡人之先天五行 春回大明朝 姑娘不必设防，我是瞎子 短发扎半丸子头的步骤 小丸子头怎么扎 短发半扎丸子头图片 简单丸子头的扎法图解 中短发丸子头 简单的丸子头扎法步骤 丸子头扎法图片 半丸子头扎法短发 可爱丸子头的扎法图解 高丸子头扎法步骤图解 长发半扎丸子头 简单丸子头扎法步骤图解 短发扎丸子头的步骤 丸子头教程 如何绑丸子头 短发丸子头的扎法图解 韩版丸子头 低丸子头 双丸子头图片 长发丸子头的扎法图解 一根皮筋扎丸子头教程 丸子头图解 丸子头简单 简单丸子头 如何梳丸子头 短发半丸子头 如何盘丸子头 韩国丸子头扎法图解 韩式丸子头扎法图解 儿童丸子头的扎法图解 扎丸子头的方法图解 怎么梳丸子头蓬松图解法 简单的丸子头扎法 韩式半扎丸子头 丸子头的梳法 男士丸子头怎么绑 女童丸子头扎法步骤图解 齐刘海丸子头 男士丸子头图片 丸子头短发 盘丸子头简单好看的步骤

程序博客网，程序员的互联网技术博客家园。csdn论坛精品 msdn技术资料都在这里