Hadoop家族环境搭建各种版本选择

System Requirements

Supported Platforms

•     GNU/Linuxis supported as a development and production platform for both server andclient.

•     SunSolaris is supported as a development and production platform for both serverand client.

•     FreeBSDis supported as a development and production platform for both server andclient.

•     Win32is supported as a development platformonly for both server and client.

•     Win64is supported as a development platformonly for both server and client.

•     MacOSXis supported as a development platformonly for both server and client.

 

Hive™ Requirements

•     Java1.7
Note:  Hive versions 1.2 onward require Java 1.7 or newer.Hive versions 0.14 to 1.1 work with Java 1.6 as well. Users are stronglyadvised to start moving to Java 1.8 (see HIVE-8607). 

•     Hadoop2.x (preferred), 1.x (not supported by Hive 2.0.0 onward).
Hive versions up to 0.13 also supported Hadoop 0.20.x, 0.23.x.

•     Hiveis commonly used in production Linux and Windows environment. Mac is a commonlyused development environment. The instructions in this document are applicableto Linux and Mac. Using it on Windows would require slightly differentsteps.  

HBase™  Requirements

This section lists required servicesand some required system configuration.

 

Table2. Java

HBase Version

JDK 6

JDK 7

JDK 8

1.2

Not Supported

yes

yes

1.1

Not Supported

yes

Running with JDK 8 will work but is not well tested.

1.0

Not Supported

yes

Running with JDK 8 will work but is not well tested.

0.98

yes

yes

Running with JDK 8 works but is not well tested. Building with JDK 8 would require removal of the deprecated remove() method of the PoolMap class and is under consideration. SeeHBASE-7608 for more information about JDK 8 support.

0.94

yes

yes

N/A

 

 

Hadoop  Requirements

Hadoop version support matrix

•     "S"= supported

•     "X"= not supported

•     "NT"= Not tested

 

 

HBase-0.94.x

HBase-0.98.x (Support for Hadoop 1.1+ is deprecated.)

HBase-1.0.x (Hadoop 1.x is NOT supported)

HBase-1.1.x

HBase-1.2.x

 

Hadoop-1.0.x

X

X

X

X

X

Hadoop-1.1.x

S

NT

X

X

X

Hadoop-0.23.x

S

X

X

X

X

Hadoop-2.0.x-alpha

NT

X

X

X

X

Hadoop-2.1.0-beta

NT

X

X

X

X

Hadoop-2.2.0

NT

S

NT

NT

X

Hadoop-2.3.x

NT

S

NT

NT

X

Hadoop-2.4.x

NT

S

S

S

S

Hadoop-2.5.x

NT

S

S

S

S

Hadoop-2.6.0

X

X

X

X

X

Hadoop-2.6.1+

NT

NT

NT

NT

S

Hadoop-2.7.0

X

X

X

X

X

Hadoop-2.7.1+

NT

NT

NT

NT

S

 

Pig™  Requirements

Unix and Windows users need thefollowing:

•     Hadoop 0.23.X, 1.X or 2.X- http://hadoop.apache.org/common/releases.html(You can run Pig with different versions of Hadoop by setting HADOOP_HOME topoint to the directory where you have installed Hadoop. If you do not setHADOOP_HOME, by default Pig will run with the embedded version, currentlyHadoop 1.0.4.)

•     Java 1.7- http://java.sun.com/javase/downloads/index.jsp(set JAVA_HOME to the root of your Java installation)

Optional

•     Python 2.7- https://www.python.org(when using Streaming Python UDFs)

•     Ant 1.8- http://ant.apache.org/(for builds)

Nutch 2.3.1  Requirements

The recommended Gora backends for thisNutch release are

•     ApacheAvro 1.7.6

•     ApacheHadoop 1.2.1 and 2.5.2

•     ApacheHBase 0.98.8-hadoop2 (although also tested with 1.X)

•     ApacheCassandra 2.0.2

•     ApacheSolr 4.10.3

•     MongoDB2.6.X

•     ApacheAccumlo 1.5.1

•     ApacheSpark 1.4.1

Nutch 2.3  Requirements

The supported Apache Gorav0.5 backends are;

•     Apache Hadoop1.0.1 & 2.4.0

•     Apache Cassandra2.0.2

•     Apache HBase0.94.14

•     Apache Accumulo1.5.1

•     MongoDB2.12.2

•     Apache Solr4.8.1

•     Apache Avro1.7.6

- See more at:http://nutch.apache.org/#sthash.qer0qldX.dpuf

 

ZooKeeper™ Requirements

ZooKeeper runs in Java, release 1.6 orgreater (JDK 6 or greater, FreeBSD support requires openjdk7). It runs as anensemble of ZooKeeper servers. ThreeZooKeeper servers is the minimum recommended size for an ensemble, and we alsorecommend that they run on separate machines. At Yahoo!, ZooKeeper is usuallydeployed on dedicated RHEL boxes, with dual-core processors, 2GB of RAM, and80GB IDE hard drives.

Chukwa™ Requirements

Chukwa should work on any POSIXplatform, but GNU/Linux is the only production platform that has been testedextensively. Chukwa has also been used successfully on Mac OS X, which severalmembers of the Chukwa team use for development.

The only absolute software requirementsare Java 1.6 or better and Hadoop 0.20.205.0+. HICC, the Chukwa visualizationinterface, requires HBase 0.90.4.

The Chukwa cluster management scriptsrely on ssh; these scripts, however, are not required if you have somealternate mechanism for starting and stopping daemons.

 

Spark™  Requirements

Mahout™  Requirements

Avro™  Requirements

Tez™ Requirements