Mac OS X EI Capitan下安装hadoop

大数据&云计算正在成为当下以及未来科技发展的趋势

           随着科技日新月异的发展，大数据，物联网，云计算慢慢开始进入人们的生活，也慢慢开始影响人们的生活，作为一名IT从业者，我觉得有必要对这些技术开始做一些功课，毕竟，我们需要更加开阔我们的视野。

       当下比较热门的大数据工具是hadoop,很多的视频教程都是以Linux系统为载体讲述的，后来。hadoop添加了Windows版本，而我是一个苹果开发的从业人员，手头只有Mac OS X系统，接下来我就把在OS X EI Capitan上安装hadoop的方法记录下来。

安装hadoop的方法大致可以分成以下几个步骤：

1.安装java JDK

2.下载hadoop软件压缩包并解压到常用目录

3.设置Mac OS X ssh设置

4.设置环境变量

步骤详解：

我的安装环境：

           目前Mac的JDK包是.pkg形式打包的，因此只需要像安装其他应用软件一样安装即可，你可以通过在控制台出入java -version查询当前电脑的java版本。据说是hadoop要安装在JDK 1.6版本以上。

           也可以通过控制台查看java安装的默认路径

           JDK的安装路径是：/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/

Mac OS X  ssh配置：

           更改设置如下：进入system preference –> sharing –> 勾选remote login，并设置allow access for all users。输入 "ssh localhost"，再输入密码并确认之后，可以看到ssh成功

           Mac是自带ssh的，在控制台验证如下：

           ssh配置如下：

macdeMac-mini-2:~ mac$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/Users/mac/.ssh/id_rsa): 【回车】

/Users/mac/.ssh/id_rsa already exists.

Overwrite (y/n)? y

Enter passphrase (empty for no passphrase): 【回车】

Enter same passphrase again: 【回车】

Your identification has been saved in /Users/mac/.ssh/id_rsa.

Your public key has been saved in /Users/mac/.ssh/id_rsa.pub.

           把密钥文件传到ssh账号下：

macdeMac-mini-2:.ssh mac$ cd ~/.ssh

macdeMac-mini-2:.ssh mac$ cp id_rsa.pub authorized_keys

macdeMac-mini-2:.ssh mac$ scp -p ~/.ssh/authorized_keys vb_macos:.ssh/

           当前的ssh每次都需要输入密码，因此需要设置ssh免登陆的方法

macdeMac-mini-2:~ mac$ ps aux | grep ssh

mac               618   0.0  0.1  2518544   5192   ??  S     9:50AM   0:00.05 /usr/bin/ssh-agent -l

mac              1773   0.0  0.0  2445080    808 s001  S+    2:58PM   0:00.00 grep ssh

macdeMac-mini-2:~ mac$ ssh localhost

Last login: Wed Feb  3 14:58:25 2016

macdeMac-mini-2:~ mac$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

Generating public/private dsa key pair.

/Users/mac/.ssh/id_dsa already exists.

Overwrite (y/n)? y                                              

Your identification has been saved in /Users/mac/.ssh/id_dsa.

Your public key has been saved in /Users/mac/.ssh/id_dsa.pub.

The key fingerprint is:

SHA256:K4A8m/0Z00CdRpmC+GJwZxoRksQ01XT8O+ec2v6xI9A [e-mail]

The key's randomart image is:

+---[DSA 1024]----+

|==+=oo...o       |

|oo= +.o+o.       |

| o *  ..=        |

| .+... . .       |

| .+.. . S o      |

|   = . o = E     |

|  o . + o * ..   |

|     . =  .= .o  |

|      o  .oooo.  |

+----[SHA256]-----+

macdeMac-mini-2:~ mac$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

ssh-keygen表示生成秘钥；-t表示秘钥类型；-P用于提供密语；-f指定生成的秘钥文件。这个命令在”~/.ssh/“文件夹下创建两个文件id_dsa和id_dsa.pub，是ssh的一对儿私钥和公钥，公钥追加到授权的key中去。

hadoop 1.2.1下载并且安装

hadoop 1.2.1下载地址    下载hadoop-1.2.1.tar.gz

           解压到Documents目录下，设置环境变量

   终端输入vi ~/.bash_profile

export PAHT=$PATH:/usr/local/apache-tomcat-7.0.67/bin

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home

export JRE_HOME=$JAVA_HOME/jre

export HADOOP_HOME=/Users/mac/Documents/hadoop-1.2.1

export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH

export HADOOP_HOME_WARN_SUPPRESS=1

export  PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home

export JRE_HOME=$JAVA_HOME/jre

     以上2各是java环境变量

export HADOOP_HOME=/Users/mac/Documents/hadoop-1.2.1

这个是hadoop环境变量

export HADOOP_HOME_WARN_SUPPRESS=1

     防止出现Warning:$HADOOP_HOME is deprecated的警告错误

     保存，在终端输入：source ~/.bash_profile

在Hadoop->conf目录下，找到hadoop-env.sh，打开编辑进行如下设置：

配置conf文件夹下的core-site.xml——指定了NameNode的主机名与端口

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>hadoop.tmp.dir</name>

<value>hdfs://localhost:9000</value>

<description>A base for other temporary directories.</description>

</property>

<property>

<name>fs.default.name</name>

<value>hdfs://localhost:8020</value>

</property>

</configuration>

配置hdfs-site.xml——指定了HDFS的默认参数副本数，因为仅运行在一个节点上，所以这里的副本数为1

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

</configuration>

配置mapred-site.xml——指定了JobTracker的主机名与端口

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>mapred.job.tracker</name>

<value>hdfs://localhost:9001</value>

</property>

<property>

<name>mapred.tasktracker.map.tasks.maximum</name>

<value>2</value>

</property>

<property>

<name>mapred.tasktracker.reduce.tasks.maximum</name>

<value>2</value>

</property>

</configuration>

终端运行 hadoop

macdeMac-mini-2:~ mac$ hadoop 

Usage: hadoop [--config confdir] COMMAND

where COMMAND is one of:

  namenode -format     format the DFS filesystem

  secondarynamenode    run the DFS secondary namenode

  namenode             run the DFS namenode

  datanode             run a DFS datanode

  dfsadmin             run a DFS admin client

  mradmin              run a Map-Reduce admin client

  fsck                 run a DFS filesystem checking utility

  fs                   run a generic filesystem user client

  balancer             run a cluster balancing utility

  oiv                  apply the offline fsimage viewer to an fsimage

  fetchdt              fetch a delegation token from the NameNode

  jobtracker           run the MapReduce job Tracker node

  pipes                run a Pipes job

  tasktracker          run a MapReduce task Tracker node

  historyserver        run job history servers as a standalone daemon

  job                  manipulate MapReduce jobs

  queue                get information regarding JobQueues

  version              print the version

  jar <jar>            run a jar file

  distcp <srcurl> <desturl> copy file or directories recursively

  distcp2 <srcurl> <desturl> DistCp version 2

  archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive

  classpath            prints the class path needed to get the

                       Hadoop jar and the required libraries

  daemonlog            get/set the log level for each daemon

 or

  CLASSNAME            run the class named CLASSNAME

Most commands print help when invoked w/o parameters.

出现以上内容说明找到了hadoop的执行程序

对namenode执行格式化操作，出现如下结果：

macdeMac-mini-2:~ mac$ hadoop namenode -format

16/02/03 15:19:43 INFO namenode.NameNode: STARTUP_MSG: 

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = macdeMac-mini-2.local/192.168.2.176

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 1.2.1

STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013

STARTUP_MSG:   java = 1.8.0_40

************************************************************/

16/02/03 15:19:43 INFO util.GSet: Computing capacity for map BlocksMap

16/02/03 15:19:43 INFO util.GSet: VM type       = 64-bit

16/02/03 15:19:43 INFO util.GSet: 2.0% max memory = 1864368128

16/02/03 15:19:43 INFO util.GSet: capacity      = 2^22 = 4194304 entries

16/02/03 15:19:43 INFO util.GSet: recommended=4194304, actual=4194304

16/02/03 15:19:44 INFO namenode.FSNamesystem: fsOwner=mac

16/02/03 15:19:44 INFO namenode.FSNamesystem: supergroup=supergroup

16/02/03 15:19:44 INFO namenode.FSNamesystem: isPermissionEnabled=true

16/02/03 15:19:44 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100

16/02/03 15:19:44 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)

16/02/03 15:19:44 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0

16/02/03 15:19:44 INFO namenode.NameNode: Caching file names occuring more than 10 times 

16/02/03 15:19:44 INFO common.Storage: Image file hdfs:/localhost:9000/dfs/name/current/fsimage of size 109 bytes saved in 0 seconds.

16/02/03 15:19:44 INFO namenode.FSEditLog: closing edit log: position=4, editlog=hdfs:/localhost:9000/dfs/name/current/edits

16/02/03 15:19:44 INFO namenode.FSEditLog: close success: truncate to 4, editlog=hdfs:/localhost:9000/dfs/name/current/edits

16/02/03 15:19:44 INFO common.Storage: Storage directory hdfs:/localhost:9000/dfs/name has been successfully formatted.

16/02/03 15:19:44 INFO namenode.NameNode: SHUTDOWN_MSG: 

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at macdeMac-mini-2.local/192.168.2.176

************************************************************/

说明HDFS安装成功

启动hadoop,输入jus，查看当前进程

macdeMac-mini-2:~ mac$ jps

1361 JobTracker

1953 Jps

1306 SecondaryNameNode

macdeMac-mini-2:~ mac$ 

OK handop已经启动