Hadoop and Hbase and Spark

Hadoop单机安装！

System：Centos6.5

Software：apache-maven-3.3.3-bin.tar.gz | hadoop-2.7.1.tar.gz | jdk-8u65-linux-i586.tar.gz | scala-2.11.7.tgz | spark-1.5.1-bin-without-hadoop.tgz

详细介绍：http://www.aboutyun.com/thread-12798-1-1.html

linux：

yum -y update 

yum -y install glibc*

tar xzvf apache-maven-3.3.3-bin.tar.gz

tar xzvf hadoop-2.7.1.tar.gz

tar xzvf scala-2.11.7.tgz

tar xzvf spark-1.5.1-bin-without-hadoop.tgz

tar xzvf jdk-8u65-linux-i586.tar.gz

(单机免密登录)

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

cat .ssh/id_dsa.pub >> .ssh/authorized_keys

ssh localhost

export HADOOP\_PREFIX=/usr/local/hadoop

安装RSYNC

yum install -y rsync

复制解压缩后的文件夹到/usr/local/ 目录下 变更文件夹权限及用户 chown spark:spark -R * | chmod 775 -R *

创建用户和组（稍后更新）

写.bash_profile

export SCALA_HOME=/usr/local/scala

export JAVA_HOME=/usr/local/jdk

export MAVEN_HOME=/usr/local/maven

export HADOOP_HOME=/usr/local/hadoop

export PATH=/usr/local/maven/bin:$PATH

export PATH=$JAVA_HOME/bin:$SCALA_HOME/bin:$MAVEN/bin:$HOME/bin:$HADOOP/bin:$HADOOP/sbin:$PATH

export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/jre/lib/dt.jar:$JAVA_HOME/jre/lib/tools.jar

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_HOME/lib:$SCALA/lib:$MAVEN/lib:/usr/local/lib:/usr/lin

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

修改

hadoop etc/hadoop/hadoop-env.sh

修改Java_Home export JAVA_HOME=/usr/jdk (一定是详细目录)

export HADOOP_COMMON_HOME=~/hadoop-2.7.0(没发现命令)

nano /etc/environment

/usr/local/hadoop/bin:/usr/local/hadoop/sbin

在hadoop 目录下创建input

mkdir input

cp etc/hadoop/*.xml input

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+' (注意JAR包版本号)

cat output/*

修改文件etc/hadoop/core-site.xml

添加如下内容：

含义：接收Client连接的RPC端口，用于获取文件系统metadata信息。

<configuration>

  <property>

        <name>fs.defaultFS</name>

        <value>hdfs://localhost:9000</value>

    </property>

</configuration>

修改etc/hadoop/hdfs-site.xml:

含义：备份只有一份

<configuration>

    <property>

        <name>dfs.replication</name>

        <value>1</value>

    </property>

</configuration>

<property>

             <name>dfs.namenode.name.dir</name>

             <value>file:/home/spark/dfs/name</value>

       </property>

      <property>

              <name>dfs.datanode.data.dir</name>

              <value>file:/home/spark/dfs/data</value>

       </property>

  

注意修改权限。

格式化namenode

hdfs namenode -format

或使用

bin/hdfs namenode -format

执行

start-dfs.sh

登录

localip:50070