hadoop hdfs 图片服务器应用部署

来源：互联网发布：乔任梁抑郁症知乎编辑：程序博客网时间：2024/04/27 12:16

一、部署前提

1.Linux环境，本人为ubuntu。

2.配置部署了hadoop，单击集群皆可。

3.安装了apache2和php,并能正常工作。

二、应用介绍

通过调用hadoop HDFS的java API,在PHP上传功能上传到服务器的图片能够自动加载到hadoop文件系统，同时可以下载和删除等。

三、具体实现

1.文件上传功能：

index.html,文件上传页面，提供了文件上传功能和查看文件列表功能

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>

<title>Hadoop</title>
<link href="../3rd/bootstrap/css/bootstrap.css" rel="stylesheet" type="text/css" />
<script type="text/javascript" src="../js/jquery-1.7.2.min.js"></script>
<script type="text/javascript" src="../js/jquery.form.js"></script>
<script type="text/javascript" src="../js/jquery.cookie.js"></script>
<script type="text/javascript">
$(document).ready(function() {

});
</script>
</head>

<body>
<div class="container">
<div class="page-header">
<h1>File Upload&Download Demo</h1>
</div>

<form class="well form-inline" action="upload_file.php" method="post"
enctype="multipart/form-data">
<label for="file">Filename:</label>
<input class="input-small" type="file" name="file" id="file" />
<input class="btn" type="submit" name="submit" value="Submit" />
</form>

<ul class="thumbnails">
<?php
exec("java -jar ./listfiles.jar hdfs://127.0.0.1:9000/user/root/upload/", $files);
foreach ($files as $f) {
$dst = "./tmp/" . basename($f);
$cmd = "java -jar ./getfile.jar ".$f." ".$dst;
if(!file_exists($dst))
exec($cmd);
?>
<li class="span3">

<div class="thumbnail">
<a href=<?php echo "$dst"?> class="thumbnail"> <img src=<?php echo "$dst"?> alt=""> </a>
<div class="caption">
<a class="btn btn-primary" href="del.php?name=<?php echo basename($f)?>">Delete</a>
</div>
</div>
</li>
<?php
}
//print_r($files);
?>

</ul>

</div>

</body>
</html>

upload_file.php,服务器处理代码，先把文件上传到临时文件然后提交到hdfs文件系统，其中会执行一段Java程序，就是hdfs文件系统提供的Java api实现的。

<?php
if (($_FILES["file"]["type"] == "image/jpeg")
&& ($_FILES["file"]["size"] < 4*1024*1024))
{
if ($_FILES["file"]["error"] > 0)
{
echo "Return Code: " . $_FILES["file"]["error"] . "<br />";
}
else
{
echo "Upload: " . $_FILES["file"]["name"] . "<br />";
echo "Type: " . $_FILES["file"]["type"] . "<br />";
echo "Size: " . ($_FILES["file"]["size"] / 1024) . " Kb<br />";
echo "Temp file: " . $_FILES["file"]["tmp_name"] . "<br />";

if (file_exists("upload/" . $_FILES["file"]["name"]))
{
echo $_FILES["file"]["name"] . " already exists. ";
}
else
{
move_uploaded_file($_FILES["file"]["tmp_name"],
"upload/" . $_FILES["file"]["name"]);
echo "Stored in: " . "upload/" . $_FILES["file"]["name"];
$cmd = "java -jar ./uploadfile.jar ./upload/" . $_FILES["file"]["name"] . " hdfs://hadoop.main:9000/upload/" . $_FILES["file"]["name"] ;
system($cmd);

header("location:index.php");
//echo $cmd;
}
}
}
else
{
print_r($_FILES["file"]);
echo "Invalid file";
}
?>

uploadFile.java,把操作系统的文件上传到hdfs

import java.io.*;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.util.Progressable;

public class UploadFile {
public static void main(String[] args) throws Exception {
try{
String localSrc = args[0];
String dst = args[1];
InputStream in = new BufferedInputStream(new FileInputStream(localSrc));

Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(dst), conf);

OutputStream out = fs.create(new Path(dst));

IOUtils.copyBytes(in, out, 4096, true);
System.out.print("success");
}catch(Exception e){
System.out.print("fail" + e.toString());
}
}
}

listfiles.java查看hdfs上的文件列表

public class ListFiles {
public static void main(String[] args) throws Exception {
String dir = args[0];
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(dir), conf);

Path[] paths = new Path[1];
paths[0] = new Path(dir);

FileStatus[] status = fs.listStatus(paths);
Path[] listedPaths = FileUtil.stat2Paths(status);
for (Path p : listedPaths) {
System.out.println(p);
}
}
}

getfile.java,获取某个图片

public class GetFile {
public static void main(String[] args) throws Exception {

String src = args[0];
String dst = args[1];

Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(src), conf);

OutputStream out = new FileOutputStream(dst);

InputStream in = null;
try {
in = fs.open(new Path(src));
IOUtils.copyBytes(in, out, 40960, false);
} finally {
IOUtils.closeStream(in);
}
}
}

四、测试

文件上传

上传完可以看到

hdfs上可以浏览到