FTP to HDFS代码示例
来源:互联网 发布:大数据时代与护理 编辑:程序博客网 时间:2024/05/21 08:59
FTP to HDFS
An FTP client that stores content directly into HDFS is good. It allows data from FTP serves to be directly stored into HDFS instead of first copying the data locally and then uploading it into HDFS. The benefits are apparent from an administrative perspective as large datasets can be pulled from FTP servers with minimal human intervention.
This will greatly simplify data being pulled from FTP Servers to HDFS. This also makes it faster as we reduce one hop into local file system.
This will greatly simplify data being pulled from FTP Servers to HDFS. This also makes it faster as we reduce one hop into local file system.
At present we are faced with the issue of our data lying in different remote FTP server locations.
This utility essentially provides following benefits
1. The steps of 'pull data from FTP server', 'store locally', 'tranfer to HDFS' and 'delete local copy' are converted into 1 step - 'Pull data and store into HDFS' .
2. No need to worry about lack of local storage as data goes directly into HDFS.
3. Can be used to run a batch of commands that include pulling data from different FTP servers.
All of this greatly simplifies administrative tasks.
Thanks to Ankur for fixing issue HADOOP-3246.
Following program does the job :
import java.io.IOException;import java.io.OutputStream;import java.net.URI;import java.net.URISyntaxException;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FSDataInputStream;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.hadoop.fs.ftp.FTPFileSystem;import org.apache.hadoop.io.IOUtils;public class FTPtoHDFS{ public static void main(String[] args) throws IOException, URISyntaxException { String src="test1.txt"; Configuration conf = new Configuration(); FTPFileSystem ftpfs = new FTPFileSystem(); ftpfs.setConf(conf); ftpfs.initialize(new URI("ftp://username:password@host"), conf); FSDataInputStream fsdin = ftpfs.open(new Path(src), 1000); FileSystem fileSystem=FileSystem.get(conf); OutputStream outputStream=fileSystem.create(new Path(args[0])); IOUtils.copyBytes(fsdin, outputStream, conf, true); }}
- FTP to HDFS代码示例
- JDK7 ftp上传下载示例代码
- Java FTP上传下载单个文件示例代码
- 基于HDFS的FTP
- 基于HDFS的FTP
- 基于HDFS的FTP
- Accessing HDFS over FTP
- 基于HDFS的FTP
- HDFS小工具 -- HDFS over ftp
- hadoop: hdfs API示例
- Hadoop2.7.1---hdfs-over-ftp
- ftp提取文件到hdfs
- FTP 代码
- ftp代码
- FTP上传文件示例
- FTP上传文件示例
- FTP使用示例
- PySpark to load HDFS
- 深入Java单例模式
- android 获取外置SD卡的路径(非挫版)
- 郁金香驱动笔记
- 【on the way】教你五年内买车买房秘诀
- SELECT INTO 和 INSERT INTO SELECT 两种表复制语句
- FTP to HDFS代码示例
- Mono for Android 初学遇到的问题
- MPEG1和MPEG2码流结构分析
- 压力测试工具ab
- android有用代码片段1
- 三元操作符 if-else
- NYOJ 飞翔
- C++标准问题总结
- 【快如闪电 掌握迅雷5.9常用快捷键§小米】