windows hadoop

测试用windows 下无守护进程的 hadoop和虚拟机linux下的hadoop配合使用

1. 环境变量
   HADOOP_INCLUDE : D:\soft\hadoop-2.7.3
   PATH : %HADOOP_INCLUDE%\bin;%HADOOP_INCLUDE%\sbin;

2. 错误解决

下载 hadoop-common-2.2.0-bin解压，拷贝到%HADOOP_INCLUDE%\bin，注意不替换已有文件：
注意hadoop-common和hadoop版本号要一致

C:\Users\admin>hadoop fs -lsPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-816/12/15 17:55:06 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary pathjava.io.IOException: Could not locate executable D:\soft\hadoop-2.7.3\bin\winutils.exe in the Hadoop binaries.        at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:379)        at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:394)        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:387)        at org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:440)        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:486)        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)        at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)16/12/15 17:55:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableFound 62 items-ls: Fatal internal errorjava.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: (null) entry in command string: null ls -F C:\Users\admin\.AndroidStudio2.1        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:769)        at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)        at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)        at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1097)        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:659)        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getOwner(RawLocalFileSystem.java:642)        at org.apache.hadoop.fs.shell.Ls.adjustColumnWidths(Ls.java:131)        at org.apache.hadoop.fs.shell.Ls.processPaths(Ls.java:102)        at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)        at org.apache.hadoop.fs.shell.Ls.processPathArgument(Ls.java:90)        at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)        at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)        at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)        at org.apache.hadoop.fs.shell.Command.run(Command.java:165)        at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)        at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:697)        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getOwner(RawLocalFileSystem.java:642)        at org.apache.hadoop.fs.shell.Ls.adjustColumnWidths(Ls.java:131)        at org.apache.hadoop.fs.shell.Ls.processPaths(Ls.java:102)        at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)        at org.apache.hadoop.fs.shell.Ls.processPathArgument(Ls.java:90)        at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)        at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)        at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)        at org.apache.hadoop.fs.shell.Command.run(Command.java:165)        at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)        at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)C:\Users\admin>hadoop fs -ls

几个配置文件

// mapred-site.xml

<configuration>    <property>        <name>mapreduce.framework.name</name>        <value>yarn</value>    </property>    <property>       <name>mapred.job.tracker</name>       <value>hdfs://localhost:9001</value>    </property></configuration>

//yarn-site.xml

<configuration>    <property>        <name>yarn.resourcemanager.hostname</name>        <value>localhost</value>    </property>    <property>        <name>yarn.nodemanager.aux-services</name>        <value>mapreduce_shuffle</value>    </property></configuration>

//core-site.xml
hadoop.tmp.dir配置windows下的tmp位置

<configuration>    <property>        <name>fs.defaultFS</name>        <value>hdfs://localhost:9000</value>    </property>    <property>          <name>hadoop.tmp.dir</name>          <value>/f:/tmp</value>          <description>A base for other temporary directories.</description>      </property></configuration>

//hdfs-site.xml

<configuration>    <property>        <name>dfs.replication</name>        <value>1</value>    </property></configuration>

格式化节点,启动hadoop

注意start-all.cmd回启动四个进程（windows command），stop-all.cmd会停止启动的进程，如果启动的进程中有出现错误的，则调用stop-all.cmd时不会出现错误的windows command

hadoop namenode -formatstart-all.cmdstop-all.cmd