hadoop hdfs环境搭建
来源:互联网 发布:php采购系统源代码 编辑:程序博客网 时间:2024/05/16 11:51
参考:
http://wiki.apache.org/hadoop/GettingStartedWithHadoop
http://blog.csdn.net/warmspace2008/article/details/8540159
hadoop环境搭建参考官方wiki,首先下载hadoop官方包,我选择的是hadoop-1.2.1。conf目录新增配置文件:
- <?xml version="1.0"?>
- <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
- <configuration>
- <property>
- <name>hadoop.tmp.dir</name>
- <value>/duitang/tmp/hadoop-${user.name}</value>
- </property>
- <property>
- <name>fs.default.name</name>
- <value>hdfs://localhost:54310</value>
- </property>
- <property>
- <name>mapred.job.tracker</name>
- <value>hdfs://localhost:54311</value>
- </property>
- <property>
- <name>dfs.replication</name>
- <value>8</value>
- </property>
- <property>
- <name>mapred.child.java.opts</name>
- <value>-Xmx512m</value>
- </property>
- </configuration>
通过start-dfs.sh启动,
starting namenode, logging to /duitang/dist/sys/hadoop-1.2.1/libexec/../logs/hadoop-yunpeng-namenode-yunpeng-duitang.out
localhost: ssh: connect to host localhost port 22: Connection refused
localhost: ssh: connect to host localhost port 22: Connection refused
结果日志报错:
2013-12-23 16:18:05,197 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /duitang/tmp/hadoop-yunpeng/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible.
分析错误,应该是没有没有创建目录,创建之后再次启动,还是报错:
2013-12-23 16:20:42,797 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException: NameNode is not formatted.
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:331)
原来NameNode需要format,执行hadoop namenode -format
13/12/23 16:22:27 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
13/12/23 16:22:27 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = yunpeng-duitang/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.6.0_29
************************************************************/
Re-format filesystem in /duitang/tmp/hadoop-yunpeng/dfs/name ? (Y or N) Y
13/12/23 16:22:29 INFO util.GSet: Computing capacity for map BlocksMap
13/12/23 16:22:29 INFO util.GSet: VM type = 64-bit
13/12/23 16:22:29 INFO util.GSet: 2.0% max memory = 932118528
13/12/23 16:22:29 INFO util.GSet: capacity = 2^21 = 2097152 entries
13/12/23 16:22:29 INFO util.GSet: recommended=2097152, actual=2097152
13/12/23 16:22:29 INFO namenode.FSNamesystem: fsOwner=yunpeng
13/12/23 16:22:29 INFO namenode.FSNamesystem: supergroup=supergroup
13/12/23 16:22:29 INFO namenode.FSNamesystem: isPermissionEnabled=true
13/12/23 16:22:29 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/12/23 16:22:29 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/12/23 16:22:29 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
13/12/23 16:22:29 INFO namenode.NameNode: Caching file names occuring more than 10 times
13/12/23 16:22:29 INFO common.Storage: Image file /duitang/tmp/hadoop-yunpeng/dfs/name/current/fsimage of size 113 bytes saved in 0 seconds.
13/12/23 16:22:30 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/duitang/tmp/hadoop-yunpeng/dfs/name/current/edits
13/12/23 16:22:30 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/duitang/tmp/hadoop-yunpeng/dfs/name/current/edits
13/12/23 16:22:30 INFO common.Storage: Storage directory /duitang/tmp/hadoop-yunpeng/dfs/name has been successfully formatted.
13/12/23 16:22:30 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at yunpeng-duitang/127.0.0.1
************************************************************/
启动成功,通过http://localhost:50070/ 可以查看。
mapred.job.tracker
hadoop.job.ugi
mapred.job.queue.name
mapred.used.genericoptionsparser true
mapred.map.max.attempts 5
mapred.reduce.max.attempts 5
mapred.child.java.opts -Xmx1024m
mapred.output.compress true
mapred.output.compression.codec org.apache.hadoop.io.compress.DefaultCodec
mapred.compress.map.output true
mapred.map.output.compression.codec org.apache.hadoop.io.compress.LzoCodec
io.compression.codecs org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.LzoCodec
fs.trash.interval 1440
fs.hdfs.impl org.apache.hadoop.hdfs.Yunti3FileSystem
fs.viewfs.impl org.apache.hadoop.fs.viewfs.ViewFileSystem
fs.yunti.prefix.lists hdfs://xxxx:9000,hdfs://xxxx:9000
fs.viewfs.zk.addr
客户端测试:
Exception in thread "main" java.io.IOException: Call to localhost/127.0.0.1:9000 failed on local exception: java.io.EOFException
报此错说明客户端hadoop版本和服务端hadoop版本不一致。修改客户端引用hadoop的版本
- hadoop hdfs环境搭建
- Hadoop HDFS环境搭建
- Hadoop HDFS环境搭建
- Hadoop HDFS分布式环境搭建
- Hadoop中hdfs环境搭建
- Hadoop HDFS分布式环境搭建
- Hadoop HDFS分布式环境搭建
- Hadoop学习笔记-Hadoop HDFS环境搭建
- hadoop hdfs搭建 mapreduce环境搭建 wordcount程序简单注释
- hadoop 分布式环境搭建与hdfs简单实用
- Flume采集日志进入HDFS以及Hadoop环境搭建
- Hadoop HDFS通过QJM实现高可用HA环境搭建
- mac下Hadoop、HDFS、Spark环境的安装和搭建
- linux搭建hdfs环境
- hadoop搭建之HDFS,MapReduce
- hadoop-hdfs 伪分布式搭建
- Hadoop环境搭建之二配置启动HDFS及本地模式运行MapReduce案例(使用HDFS上数据)
- hadoop随笔一之环境搭建及hdfs增删改查
- spring源码导入Eclipse火星版出现了问题
- 2016百度之星初赛Astar Round2B - 区间的价值
- Android postTranslate和preTranslate的理解
- 在前台利用jquery对dom元素进行排序
- GitHub for Windows 安装失败,An error occurred attempting to install github 的解决办法
- hadoop hdfs环境搭建
- 2016-5-23
- MCS(调制与编码策略)对应是Qm和TBS index,那么是由什么公司计算出来的?还是由TBS决定的?(MCS为0-31的表)
- webmagic采集CSDN的Java_WebDevelop页面
- CentOS 系统下编译安装 gcc4.8.2过程及编译问题详解
- 第十三周项目六 立体类族共有的抽象类
- 接入支付宝即时到账接口-java
- http状态码及含义
- 二进制传输 文本传输