cdh5 hadoop redhat 本地仓库配置
来源:互联网 发布:淘宝发布类目受到限制 编辑:程序博客网 时间:2024/06/06 09:40
cdh5 hadoop redhat 本地仓库配置
cdh5 在网站上的站点位置:
http://archive-primary.cloudera.com/cdh5/redhat/6/x86_64/cdh/
在RHEL6上配置指向这个repo非常简单,只要把:
http://archive-primary.cloudera.com/cdh5/redhat/6/x86_64/cdh/cloudera-cdh5.repo
下载存储到本地即可:
/etc/yum.repos.d/cloudera-cdh5.repo
但是如果是离线情况下,网络连接不可用,就需要把整个资源镜像到本地,然后在cloudera-cdh5.repo中配置。我写了一个脚本,用于下载整个站点。虽然用wget一个命令可以搞定,为了练练shell脚本,我还是写了一个。基本思路就是分析网页,找到资源链接,存储到本地目录。脚本中:PATH_MUST_BE_EXSITED必须指向已经存在的本地目录。不废话,上代码:
#!/bin/bash## @file# cdh5_rhel6-downloads.sh## @date# 2014-12-18## @author# cheungmine## @version# 0.0.1pre## downloads all from CDH_URL_PREFIX:# http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/################################################################################## specify where you want to save downloaded packages here:#PATH_MUST_BE_EXSITED="../libs/cdh"# get real path from relative pathfunction real_path() { \cd "$1" /bin/pwd}# server dist resources:#CDH_URL_PREFIX="http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh"CDH_GPGKEY=$CDH_URL_PREFIX"/RPM-GPG-KEY-cloudera"CDH_REPO=$CDH_URL_PREFIX"/cloudera-cdh5.repo"CDH5_REPODATA=$CDH_URL_PREFIX"/5/repodata/"CDH5_RPMS_NOARCH=$CDH_URL_PREFIX"/5/RPMS/noarch/"CDH5_RPMS_X86_64=$CDH_URL_PREFIX"/5/RPMS/x86_64/"# source packages not used:CDH5_SRPMS=$CDH_URL_PREFIX"/5/SRPMS/"# get local absolute path for storing the downloaded:CDH5_LOCALPATH=$(real_path $PATH_MUST_BE_EXSITED)echo "**** downloaded packages will be stored in folder: "$CDH5_LOCALPATH# first we get index pages:#repodata_html=$CDH5_LOCALPATH"/.repodata.index.html"x86_64_html=$CDH5_LOCALPATH"/.x86_64.index.html"noarch_html=$CDH5_LOCALPATH"/.noarch.index.html"wget -c $CDH5_REPODATA -P $CDH5_LOCALPATH -O $repodata_htmlwget -c $CDH5_RPMS_NOARCH -P $CDH5_LOCALPATH -O $noarch_htmlwget -c $CDH5_RPMS_X86_64 -P $CDH5_LOCALPATH -O $x86_64_htmlwget -c $CDH_GPGKEY -P $CDH5_LOCALPATHwget -c $CDH_REPO -P $CDH5_LOCALPATH# download repodata# CDH5_REPODATArepodata_dir=$CDH5_LOCALPATH"/5/repodata"mkdir -p $repodata_direcho -e "process file: '$repodata_html'"while read linedo # start with: <td><a href=" a=`echo $line | sed -n '/<td><a href="/p'` if [ -n "$a" ]; then b=`echo $a | sed -n '/Parent Directory/p'` # do including: Parent Directory if [ -z "$b" ]; then # end with: </a></td> b=`echo $a | sed -n '/<\/a><\/td>/p'` if [ -n "$b" ]; then a=`echo $a | sed -e 's/.*<td><a href="//;s/">.*//'` url=$CDH5_REPODATA$a echo -e "download: $url" wget -c $url -P $repodata_dir -O $repodata_dir/$a fi fi fidone < $repodata_html# download noarch# CDH5_RPMS_NOARCHnoarch_dir=$CDH5_LOCALPATH"/5/RPMS/noarch"mkdir -p $noarch_direcho -e "process file: '$noarch_html'"while read linedo # start with: <td><a href=" a=`echo $line | sed -n '/<td><a href="/p'` if [ -n "$a" ]; then b=`echo $a | sed -n '/Parent Directory/p'` # do including: Parent Directory if [ -z "$b" ]; then # end with: </a></td> b=`echo $a | sed -n '/<\/a><\/td>/p'` if [ -n "$b" ]; then a=`echo $a | sed -e 's/.*<td><a href="//;s/">.*//'` url=$CDH5_RPMS_NOARCH$a echo -e "download: $url" wget -c $url -P $noarch_dir -O $noarch_dir/$a fi fi fidone < $noarch_html# download x86_64# CDH5_RPMS_X86_64x86_64_dir=$CDH5_LOCALPATH"/5/RPMS/x86_64"mkdir -p $x86_64_direcho -e "process file: '$x86_64_html'"while read linedo # start with: <td><a href=" a=`echo $line | sed -n '/<td><a href="/p'` if [ -n "$a" ]; then b=`echo $a | sed -n '/Parent Directory/p'` # do including: Parent Directory if [ -z "$b" ]; then # end with: </a></td> b=`echo $a | sed -n '/<\/a><\/td>/p'` if [ -n "$b" ]; then a=`echo $a | sed -e 's/.*<td><a href="//;s/">.*//'` url=$CDH5_RPMS_X86_64$a echo -e "download: $url" wget -c $url -P $x86_64_dir -O $x86_64_dir/$a fi fi fidone < $x86_64_html# TODO: do we need to check all packages?# remove index pages:rm -f $repodata_html $x86_64_html $noarch_htmlecho "download all packages successfully."
上面脚本可以重复多次运行,不会重复下载。PATH_MUST_BE_EXSITED里面保存了cdh5全部内容。最后把PATH_MUST_BE_EXSITED的全部内容上传到本地ftp服务器上,确保可以访问:
ftp://your-server-ip/pub/libs/cdh/
然后在需要访问的RHEL6机器上,增加一个repo文件,我的是:
# /etc/yum.repos.d/cdh5.repo
[cloudera-cdh5]# Packages for Cloudera's Distribution for Hadoop, Version 5, on RedHator CentOS 6 x86_64name = Cloudera's Distribution for Hadoop, Version 5baseurl = ftp://your-server-ip/pub/libs/cdh/5/gpgkey = ftp://your-server-ip/pub/libs/cdh/RPM-GPG-KEY-cloudera gpgcheck = 1enabled = 1
好了,cdh的本地仓库建好了。只要能访问到这个ftp的服务器中存在这个 /etc/yum.repos.d/cdh5.repo 文件,就可以安装cdh hadoop的软件包。比如安装个zookeeper服务端:
#Installing the ZooKeeper Base Package $ yum install zookeeper# Installing the ZooKeeper Server Package $ yum install zookeeper-server# start zookeeper-server $ service zookeeper-server init --myid=1Using myid of 1
zookeeper安装在/usr/lib/zookeeper,进入bin目录启动和关闭:
$ zkServer.sh start$ zkServer.sh stop
OK!
- cdh5 hadoop redhat 本地仓库配置
- redhat 配置本地源
- 入门级Hadoop集群搭建详细教程(六):yum本地仓库与远程仓库配置
- Maven本地仓库配置
- 配置本地yum仓库
- Maven本地仓库配置
- Maven本地仓库配置
- Maven配置本地仓库
- maven本地仓库配置
- maven本地仓库配置
- 配置本地yum仓库
- maven本地仓库配置
- maven配置本地仓库
- CDH5 hadoop-hive-habse单机版配置
- Maven仓库介绍与本地仓库配置
- Maven仓库介绍与本地仓库配置
- maven 本地仓库 阿里云仓库配置
- Maven仓库介绍与本地仓库配置
- 定义QlikView中的系统字段
- UIScrollView 实践经验
- Vim的常用命令技巧
- Decode Ways
- 社説 20141219 パキスタン襲撃 テロ封じへ国際協調を強めよ
- cdh5 hadoop redhat 本地仓库配置
- eclipse 打jar包
- Rich feature hierarchies for accurate object detection and semantic segmentation(译)
- 社説 20141219 高浜原発「合格」 再稼働に政府支援が不可欠だ
- SQL Server常用技巧
- Rexsee传感器处理
- 立此存照(20)[C++]C++ Primer(4th)消息处理示例
- IOS开发笔记之文字转语音AVSpeechSynthesizer用法
- sql server 日期函数 [LocalTimeToUTC]、[UtcToLocalTime]、ConverToMidnight函数