hadoop_cluser

修改主机名：vi /etc/sysconfig/network

修改主机名和IP的映射关系：vim /etc/hosts

java安装

卸载之前版本的java

1.卸载java：rpm -qa | grep java

  卸载：rpm -e --nodeps  查询出来的包名

1.cd /opt

tar -zxvf jdk-7u75-linux-x64.tar.gz 

2.vi /etc/profile

        export JAVA_HOME=/opt/jdk1.7.0_75

        export PATH=$PATH:$JAVA_HOME/bin

防火墙

        #查看防火墙状态

        service iptables status

        #关闭防火墙

        service iptables stop

        #查看防火墙开机启动状态

        chkconfig iptables --list

        #关闭防火墙开机启动

        chkconfig iptables off

安装zookeeper

1.安装配置zooekeeper集群

1.1解压

tar -zxvf zookeeper-3.4.3.tar.gz 

1.2修改配置

cd /opt/zookeeper-3.4.3/conf/

cp zoo_sample.cfg zoo.cfg

vim /opt/zookeeper-3.4.3/conf/zoo.cfg

修改：

dataDir=/opt/zookeeper-3.4.3/data

dataLogDir=/opt/zookeeper-3.4.3/logs

server.1=s1:2888:3888

server.2=s2:2888:3888

server.3=s3:2888:3888

保存退出

然后创建一个tmp文件夹

mkdir /opt/zookeeper-3.4.3/data

mkdir /opt/zookeeper-3.4.3/logs

再创建一个空文件

touch /opt/zookeeper-3.4.3/data/myid

最后向该文件写入IDaa

echo 1 > /opt/zookeeper-3.4.3/data/myid

1.3将配置好的zookeeper拷贝到其他节点(首先分别在hadoop2、hadoop3

scp -r /opt/zookeeper-3.4.3/ hadoop2:/opt/

scp -r /opt/zookeeper-3.4.3/ hadoop3:/opt/

注意：修改hadoop1、hadoop2对应/opt/zookeeper-3.4.3/tmp/myid内

hadoop2：

echo 2 > /opt/zookeeper-3.4.3/data/myid

hadoop3：

echo 3 > /opt/zookeeper-3.4.3/data/myid

    1.4启动zookeeper

 /opt/zookeeper-3.4.3/bin/zkServer.sh start

 /opt/zookeeper-3.4.3/bin/zkServer.sh status

 报错

[root@s1 bin]# /opt/zookeeper-3.4.3/bin/zkServer.sh status 
JMX enabled by default 
Using config: /opt/zookeeper-3.4.3/bin/../conf/zoo.cfg 
Error contacting service. It is probably not running.

多查看zookeeper.out得知

org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing /opt/zookeeper-3.4.3/bin/../conf/zoo.cfg 

        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:121) 

        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101) 

        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) 

Caused by: java.lang.IllegalArgumentException: serverid null is not a number 

        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:358) 

        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:117) 

        ... 2 more

多一个节点：QuorumPeerMain

解决方法：

1.核对了myid之后，发现确实有问题，修改之，问题 依旧

2.文件夹都在的

3.修改zkServer.sh文件，问题依旧：http://blog.sina.com.cn/s/blog_72827fb101018yn9.html

4.现在查询2181端口：netstat -an |grep 2181已经启动，而且zookeeper.out里面确实没有报错信息了，问题仍没有解决，

tail -f zookeeper.out报错    

2015-02-01 19:22:52,136 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@764] - Notification time out: 60000 
2015-02-01 19:23:52,139 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@368] - Cannot open channel to 2 at election address s2/10.6.0.211:3888 
java.net.ConnectException: 拒绝连接 
at java.net.PlainSocketImpl.socketConnect(Native Method) 
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) 
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) 
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) 
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) 
at java.net.Socket.connect(Socket.java:579) 
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:354) 
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:388) 
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:755) 
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:716) 
2015-02-01 19:23:52,144 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@368] - Cannot open channel to 3 at election address s3/10.6.0.86:3888 

[root@hadoopname snapshot]# echo ruok | nc hadoopName 2181

imok

每一个客户端都要启动一次zookeeper

安装hadoop

1.

 hadoop-env.sh

export JAVA_HOME=/opt/jdk1.7.0_75

export HADOOP_COMMON_LIB_NATIVE_DIR=/opt/hadoop-2.4.1/lib/native

export HADOOP_OPTS="-Djava.library.path=/opt/hadoop-2.4.1/lib"

2.core-site.xml，做如下配置：

<!-- 指定hdfs的nameservice为ns1 --> 

<property><name>fs.defaultFS</name><value>hdfs://ns1</value></property> 

<!-- 指定zookeeper地址 --> 

<property><name>ha.zookeeper.quorum</name><value>s1:2181,s2:2181,s3:2181</value></property> 

<property> 

  <name>io.compression.codecs</name> 

  <value>org.apache.hadoop.io.compress.GzipCodec, 

         org.apache.hadoop.io.compress.DefaultCodec, 

         com.hadoop.compression.lzo.LzoCodec, 

         com.hadoop.compression.lzo.LzopCodec, 

         org.apache.hadoop.io.compress.BZip2Codec</value> 

</property> 

<property><name>io.compression.codec.lzo.class</name><value>com.hadoop.compression.lzo.LzoCodec</value></property> 

<property><name>hadoop.tmp.dir</name><value>/opt/hadoop-2.4.1/tmp</value></property> 

<property><name>hadoop.proxyuser.hadoop.hosts</name><value>*</value></property> 

<property><name>hadoop.proxyuser.hadoop.groups</name><value>*</value></property> 

<property><name>hadoop.native.lib</name><value>true</value></property> 

<property><name>ha.zookeeper.session-timeout.ms</name><value>60000</value></property> 

<property><name>ha.failover-controller.cli-check.rpc-timeout.ms</name><value>60000</value></property> 

<property><name>ipc.client.connect.timeout</name><value>20000</value></property>

3.hdfs-site.xml

alias vi=vim

<configuration> 

<!--指定hdfs的nameservice为ns1，需要和core-site.xml中的保持一致 --> 

<property><name>dfs.nameservices</name><value>ns1</value></property> 

<!-- ns1下面有两个NameNode，分别是nn1，nn2 --> 

<property><name>dfs.ha.namenodes.ns1</name><value>nn1,nn2</value></property> 

<!-- nn1的RPC通信地址 --> 

<property><name>dfs.namenode.rpc-address.ns1.nn1</name><value>s1:9000</value></property> 

<!-- nn1的http通信地址 --> 

<property><name>dfs.namenode.http-address.ns1.nn1</name><value>s1:50070</value></property> 

<!-- nn2的RPC通信地址 --> 

<property><name>dfs.namenode.rpc-address.ns1.nn2</name><value>s2:9000</value></property> 

<!-- nn2的http通信地址 --> 

<property><name>dfs.namenode.http-address.ns1.nn2</name><value>s2:50070</value></property> 

<!-- 指定NameNode的元数据在JournalNode上的存放位置 --> 

<property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://s1:8485;s2:8485;s3:8485/ns1</value></property> 

<!-- 指定JournalNode在本地磁盘存放数据的位置 --> 

<property><name>dfs.journalnode.edits.dir</name><value>/opt/hadoop-2.4.1/journal</value></property> 

<!-- 开启NameNode失败自动切换 --> 

<property><name>dfs.ha.automatic-failover.enabled</name><value>true</value></property> 

<!-- 配置失败自动切换实现方式 --> 

<property><name>dfs.client.failover.proxy.provider.ns1</name><value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value></property> 

<!-- 配置隔离机制方法，多个机制用换行分割，即每个机制暂用一行--> 

<property><name>dfs.ha.fencing.methods</name> 

<value> 

sshfence 

shell(/bin/true) 

</value></property> 

<!-- 使用sshfence隔离机制时需要ssh免登陆 --> 

<property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/root/.ssh/id_rsa</value></property> 

<!-- 配置sshfence隔离机制超时时间 --> 

<property><name>dfs.ha.fencing.ssh.connect-timeout</name><value>30000</value></property> 

<property><name>dfs.replication</name><value>2</value></property> 

</configuration>

4.cp mapred-site.xml.template mapred-site.xml

mapred-site.xml

<configuration>

<!-- 指定mr框架为yarn方式 -->

<property><name>mapreduce.framework.name</name><value>yarn</value></property>

<property><name>mapred.map.output.compress</name><value>true</value></property>

<property><name>mapred.map.output.compress.codec</name><value>com.hadoop.compression.lzo.LzoCodec</value></property>

</configuration>

5.yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties --> 

<!-- 指定resourcemanager地址 -->

<property><name>yarn.resourcemanager.hostname</name><value>s1</value></property>

<!-- 指定nodemanager启动时加载server的方式为shuffle server -->

<property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property>

<property><name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value></property> 

<property><name>yarn.resourcemanager.address</name><value>s1:8032</value></property> 

<property><name>yarn.log-aggregation-enable</name><value>true</value></property> 

<!-- YARN 的 HA 配置 --> 

<property><name>yarn.resourcemanager.ha.enabled</name><value>true</value></property> 

<property><name>yarn.resourcemanager.cluster-id</name><value>yarnHA</value></property>

<property><name>yarn.resourcemanager.ha.rm-ids</name><value>rm1,rm2</value></property>

<property><name>yarn.resourcemanager.hostname.rm1</name><value>s1</value></property>

<property><name>yarn.resourcemanager.hostname.rm2</name><value>s2</value></property>

<property><name>yarn.resourcemanager.zk-address</name><value>s1:2181,s2:2181,s3:2181</value></property>

</configuration> 

6.slaves

修改slaves(slaves是指定子节点的位置，因为要在zhangxin01上启动HDFS、在zhangxin03启动yarn，所以zhangxin01上的slaves文件指定的是datanode的位置，zhangxin03上的slaves文件指定的是nodemanager的位置)

s1

s2

s3

2.5启动zookeeper集群

./zkServer.sh status

2.6启动journalnode（在zhangxin01上启动所有journalnode，注意：是调用的hadoop-daemons.sh这个脚本，注意是复数s的那个脚本）

sbin/hadoop-daemons.sh start journalnode(daemons走的是ssh协议，可以同时启用多个进程)

#运行jps命令检验，s2,s3上多了JournalNode进程

2.7格式化HDFS

#在s11上执行命令:

hadoop namenode -format

#格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成个文件，这里我配置的是/zhangxin/hadoop-2.2.0/tmp，

#然后将/zhangxin/hadoop-2.2.0/tmp拷贝到zhangxin02的/zhangxin/hadoop-2.2.0/下。

scp -r tmp/ zhangxin:/zhangxin/hadoop-2.2.0/

2.8格式化ZK(在zhangxin01上执行即可)

hdfs zkfc -formatZK

zkCli.sh 

2.9启动HDFS(在zhangxin01上执行)

sbin/start-dfs.sh

2.10启动YARN(#####注意#####：是在zhangxin03上执行start-yarn.sh，把namenode和resourcemanager分开是因为性能问题，因为他们都要占用大量资源，所以把他们分开了，他们分开了就要分别在不同的机器上启动)

sbin/start-yarn.sh

到此，hadoop2.4.1配置完毕，可以统计浏览器访问:

http://10.0.8.2:50070

http://192.168.1.202:50070

验证HDFS HA

首先向hdfs上传一个文件

hadoop fs -put /etc/profile /profile

hadoop fs -ls /

然后再kill掉active的NameNode

kill -9 <pid of NN>

sbin/hadoop-daemon.sh start namenode 

验证YARN：

运行一下hadoop提供的demo中的WordCount程序：

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /profile /out

netstat -ntpl |grep 50070

hdfs namenode –bootstrapStandby

测试：hadoop jar /opt/hadoop-2.4.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar wordcount /core-site.xml /out

报错

解决方法：

安装lzo

http://www.linuxidc.com/Linux/2014-03/98601.htm

http://www.linuxidc.com/Linux/2014-03/98602.htm

一、安装lzop:
yum -y install lzop

二、安装lzo
1、wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.06.tar.gz
2、tar -zxvf lzo-2.06.tar.gz
3、mv lzo-2.06 lzo && cd lzo
4、export CFLAGS=-m64
5、./configure -enable-shared

报错：

configure: error: no acceptable C compiler found in $PATH

解决：yum install gcc

6、make && make install【默认安装在了/usr/local/lib下:liblzo2.a  liblzo2.la  liblzo2.so  liblzo2.so.2  liblzo2.so.2.0.0】
7、在/etc/ld.so.conf.d/目录下新建lzo.conf文件,内容：
/usr/local/lib
8、让lzo.conf生效:/sbin/ldconfig -v

三、安装Hadoop-LZO
1、下载源码:https://github.com/twitter/hadoop-lzo
2、解压后是hadoop-lzo-master，进入hadoop-lzo-master目录
3、export CFLAGS=-m64
4、export CXXFLAGS=-m64
5、export C_INCLUDE_PATH=/usr/local/include/lzo
6、export LIBRARY_PATH=/usr/local/lib

maven下载

wget http://maven.apache.org/download.cgi/apache-maven-3.0.5-bin.tar.gz

在dev3上的安装路径：/usr/local/maven

b.解压：tar -zxvf apache-maven-3.0.5-bin.tar.gz 

c.重命名：rm -rf apache-maven-3.0.5-bin.tar.gz 

d.设置环境变量 vi /etc/profile

MAVEN_HOME=/usr/local/maven

export MAVEN_HOME

export PATH=${PATH}:${MAVEN_HOME}/bin

e.输入验证信息，查看是否安装成功：mvn -v

报错：解压

解决：yum install gzip

7、mvn clean package -Dmaven.test.skip=true【可能在hadoop-lzo-master目录下还有一个hadoop-lzo-master目录,就会抛MissingProjectException异常,找不到源码，需要进入hadoop-lzo-master/hadoop-lzo-master目录下编译，或者去掉一层hadoop-lzo-master目录】

8、在当前目录下生成了target，下面有个native/Linux-amd64-64/lib目录，将lib目录下的文件拷贝到hadoop的lib/native目录下:tar -cBf - -C target/native//Linux-amd64-64/lib . | tar -xBvf - -C /usr/local/hadoop/lib/native

cp /opt/hadoop-lzo-master/target/native/Linux-amd64-64/lib/* /opt/hadoop-2.4.1/lib/native/

cp /opt/hadoop-lzo-master/target/native/Linux-amd64-64/* /opt/hadoop-2.4.1/lib/native/

9、将target下的hadoop-lzo-xxx.jar拷贝到hadoop/lib下

cp /opt/hadoop-lzo-master/target/hadoop-lzo-0.4.20-*.jar /opt/hadoop-2.4.1/lib     

cp /opt/hadoop-lzo-master/target/hadoop-lzo-*.jar /opt/hadoop-2.4.1//share/hadoop/mapreduce/lib/

尼玛，还是不行啊

14/03/13 17:25:41 ERROR lzo.LzoCodec: Cannot load native-lzo without native-hadoop 
14/03/13 17:25:43 INFO lzo.LzoIndexer: [INDEX] LZO Indexing file /test2.lzo, size 0.00 GB... 
Exception in thread "main" java.lang.RuntimeException: native-lzo library not available 
at com.hadoop.compression.lzo.LzopCodec.createDecompressor(LzopCodec.java:91) 
at com.hadoop.compression.lzo.LzoIndex.createIndex(LzoIndex.java:222) 
at com.hadoop.compression.lzo.LzoIndexer.indexSingleFile(LzoIndexer.java:117) 
at com.hadoop.compression.lzo.LzoIndexer.indexInternal(LzoIndexer.java:98) 
at com.hadoop.compression.lzo.LzoIndexer.index(LzoIndexer.java:52) 
at com.hadoop.compression.lzo.LzoIndexer.main(LzoIndexer.java:137) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
at java.lang.reflect.Method.invoke(Method.java:597) 
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

十几年前，小学生吃辣条，大学生喝咖啡，十几年后，小学生吃哈根达斯，大学生吃辣条，这不是重点，重点是吃辣条的还是那群人！

Hive只在一个节点上安装即可

 1

1.长传tar包

 

2.解压S

    tar -zxvf hive-0.9.0.tar.gz -C /itcast

3.配置mysql metastore（切换到root用户）

    配置HIVE_HOME环境变量

    擦除：rpm -qa | grep mysql

    解除依赖：rpm -e mysql-libs-5.1.66-2.el6_3.i686 --nodeps

5.安装mysql：yum install -y mysql-server mysql mysql-deve

  查看mysql的版本： rpm -qi mysql-server

  启动mysql：service mysqld start

  设置mysql开机启动：chkconfig mysqld on

    root账号设置密码为 root：mysqladmin -u root password 'root'
    查看端口：netstat -ntpl |grep 3306

    进入mysql：mysql -uroot -proot

    配置文件所在地：vi /etc/my.cnf

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'root' WITH GRANT OPTION;  --修改比较复杂的用户名和密码

FLUSH PRIVILEGES;

查看user表的信息：select * from user;

创建一个新用户：create user 'hive'@'10.6.0.127' identified by 'hive'; 

进入mysql的命令：mysql -h10.6.0.127 -phive -uhive

mysql -h172.31.15.174 -phive -uhive

5.安装hive和mysq完成后，将mysql的连接jar包拷贝到$HIVE_HOME/lib目录下

    如果出现没有权限的问题，在mysql授权(在安装mysql的机器上执行)

 

4.配置hive

    cp hive-default.xml.template hive-site.xml

    修改hive-site.xml（删除所有内容，只留一个<configuration></configuration>）

    添加如下内容：/

<configuration>

    <property>

      <name>javax.jdo.option.ConnectionURL</name>

      <value>jdbc:mysql://10.6.0.127:3306/hive?createDatabaseIfNotExist=true</value>

      <description>JDBC connect string for a JDBC metastore</description>

    </property>

    <property> 

      <name>javax.jdo.option.ConnectionDriverName</name>

      <value>com.mysql.jdbc.Driver</value>

      <description>Driver class name for a JDBC metastore</description>

    </property>

    <property> 

      <name>javax.jdo.option.ConnectionUserName</name>

      <value>root</value>

      <description>username to use against metastore database</description>

    </property>

    <property> 

      <name>javax.jdo.option.ConnectionPassword</name>

      <value>123456</value>

      <description>password to use against metastore database</description>

    </property>

</configuration>

 

测试hive

http://blog.csdn.net/chenyi8888/article/details/7518986

hive> SELECT * FROM pokes; 
OK 
Failed with exception java.io.IOException:java.io.IOException: Cannot create an instance of InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork! 
Time taken: 3.047 seconds 
hive> 

hive

vi /opt/hadoop-2.4.1/etc/hadoop/hadoop-env.sh 添加如下内容

export  HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_HOME/lib/hadoop-lzo.jar

vi /opt/hadoop-2.4.1/etc/hadoop/mapred-site.xml

<property><name>mapreduce.framework.name</name><value>yarn</value></property> 

<property><name>mapred.map.output.compress</name><value>true</value></property> 

<property><name>mapred.map.output.compress.codec</name><value>com.hadoop.compression.lzo.LzoCodec</value></property>

测试hadoop-lzo

[root@s1 native]# hadoop fs -text /d_channel.csv.lzo 
15/02/12 18:02:51 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
15/02/12 18:02:51 WARN lzo.LzoCompressor: java.lang.NoSuchFieldError: lzoCompressLevelFunc
15/02/12 18:02:51 ERROR lzo.LzoCodec: Failed to load/initialize native-lzo library
-text: Fatal internal error
java.lang.RuntimeException: native-lzo library not available
        at com.hadoop.compression.lzo.LzopCodec.createDecompressor(LzopCodec.java:104)
        at com.hadoop.compression.lzo.LzopCodec.createInputStream(LzopCodec.java:89)
        at org.apache.hadoop.fs.shell.Display$Text.getInputStream(Display.java:150)
        at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:98)
        at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:306)
        at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:278)
        at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:260)
        at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:244)
        at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
        at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
        at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)

启动(必须所有的先ssh通之后可以启动起来，不然不同)

 /opt/zookeeper-3.4.3/bin/zkServer.sh start （s1,s2,s3）

 /opt/zookeeper-3.4.3/bin/zkServer.sh status

/opt/hadoop-2.4.1/sbin/hadoop-daemons.sh start journalnode (s1)

/opt/hadoop-2.4.1/sbin/start-dfs.sh  (s1)

/opt/hadoop-2.4.1/sbin/start-yarn.sh  (s1)

--bwlimit=1500kb

停止

 /opt/zookeeper-3.4.3/bin/zkServer.sh stop（s1,s2,s3）

 /opt/zookeeper-3.4.3/bin/zkServer.sh status

/opt/hadoop-2.4.1/sbin/stop-all.sh

sbin/hadoop-daemons.sh stop journalnode (s1)

/opt/hadoop-2.4.1/sbin/stop-all.sh   (s1)

sbin/stop-yarn.sh  (s1)

查看端口：netstat -ntpl 

访问hdfs的状态信息：10.6.0.127:50070

查看yarn-job的执行状态：http://10.6.0.127:8088/cluster

异常

15/03/19 20:11:04 WARN retry.RetryInvocationHandler: Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo over s2/10.0.8.3:9000. Not retrying because failovers (15) exceeded maximum allowed (15) 

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby 

        at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87) 

        at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1639) 

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1193) 

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3514) 

        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:779)

hadoop ha CDH5.01 , 两个NN都是standby

因为zookeeper没先开

hive权限问题解决了，然后就是这个异常了

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 767 bytes  

解决方法：http://blog.csdn.net/wind520/article/details/39890967

mysql -h10.6.0.127 -uroot -p123456 进入hive

SET character_set_client = utf8 ;  

SET character_set_connection = utf8 ;  

SET character_set_database = utf8 ;  

SET character_set_results = utf8 ;  

SET character_set_server = utf8 ;  

SET collation_connection = utf8 ;  

SET collation_database = utf8 ;  

 SET collation_server = utf8 ;  

SET NAMES 'utf8';  

SHOW VARIABLES LIKE 'character%';  

退出mysql

hive --hiveconf hive.root.logger=DEBUG,console 

错误依旧

spark 部署

1.tar -zxvf spark-1.1.0-bin-hadoop2.4.tgz

2.mv spark-1.1.0-bin-hadoop2.4 spark

3.环境变量：export SPARK_HOME=/opt/spark （s1,s2,s3） 

4.cd /opt/spark/conf

   cp spark-env.sh.template spark-env.sh   

   vi spark-env.sh   

export SPARK_MASTER_IP=s1 

export SPARK_MASTER_PORT=7077 

export SPARK_WORKER_CORES=1 

export SPARK_WORKER_MEMORY=1g 

export SPARK_WORKER_INSTANCES=1 

export SPARK_WORKER_DIR=/opt/spark

5.启动spark

1)/opt/spark/bin/spark-shell --master spark://s1:7077

2)./spark-shell master=spark://s1:7077 

3)/opt/spark/bin/spark-shell --master spark://s1:7077,s2:7077,s3:7077

解决方式：

1./etc/hosts  。del 127.0.0.1 xxx 

2.关闭防火墙

root#service iptables stop
root#service ip6tables stop
root#chkconfig iptables off
root#chkconfig ip6tables off

3.无密码登陆 ssh    ssh s1 date  

4.master小写

cd /usr/local/spark-1.1.0-bin-hadoop2.42 bin/run-example SparkPi