MapReduce的疑难杂症

1.Java.net.SocketTimeoutException: 60000millis timeout while waiting for channel to be ready for read. ch

若果排除代码纰漏，这就是电脑配置引起的问题，因为切片分的太多所以导致运行超时。应该用ConbineTextInputFormat规定每个任务读取文件的最低大小，从而减少切分数

在驱动类中添加

job.setInputFormatClass(CombineTextInputFormat.class);CombineTextInputFormat.setMaxInputSplitSize(job, 4194304);CombineTextInputFormat.setMinInputSplitSize(job, 2000000);

2.

Error: java.lang.RuntimeException: java.io.EOFException        at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:165)        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1268)        at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:35)        at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:87)        at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1600)        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)        at java.security.AccessController.doPrivileged(Native Method)        at javax.security.auth.Subject.doAs(Subject.java:415)        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)Caused by: java.io.EOFException        at java.io.DataInputStream.readInt(DataInputStream.java:392)        at cn.bigdata.mapreduce.climate.ClimateSecondarySort$DateTemperaturePair.readFields(ClimateSecondarySort.java:54)        at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:158)        ... 14 moreContainer killed by the ApplicationMaster.Container killed on request. Exit code is 143Container exited with a non-zero exit code 143

此错误是我在写二次排序程序的时候出现的，问题是在于我的WritableComparable实现类中，write方法中的wirte与readFields方法中的read不一致造成的，如：

@Override
public void readFields(DataInput in) throws IOException {
yearMonth = in.readLine();
temperature = in.readInt();
}


@Override
public void write(DataOutput out) throws IOException {
out.writeChars(yearMonth);
out.writeInt(temperature);
}

yearMonth类型是字符串，这样写的话就会报上面的错误，必须将yearMonth类型改为其它并且保证两个方法中读写类型一致才可以

3.在eclipse远程调试mr程序时出的错：

Java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)Vat org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Native Method)at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode(NativeIO.java:524)at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:476)at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:530)at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:507)at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:305)at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:144)at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:415)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)at com.cloud.bigdata.ad.background.TaskExecutor.run(TaskExecutor.java:135)at com.cloud.bigdata.ad.test.background.executors.isp.HostPVByISPHostTaskExecutorTest.test(HostPVByISPHostTaskExecutorTest.java:28)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at junit.framework.TestCase.runTest(TestCase.java:168)at junit.framework.TestCase.runBare(TestCase.java:134)at org.springframework.test.ConditionalTestCase.runBare(ConditionalTestCase.java:79)at junit.framework.TestResult$1.protect(TestResult.java:110)at junit.framework.TestResult.runProtected(TestResult.java:128)at junit.framework.TestResult.run(TestResult.java:113)at junit.framework.TestCase.run(TestCase.java:124)at junit.framework.TestSuite.runTest(TestSuite.java:232)at junit.framework.TestSuite.run(TestSuite.java:227)at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

其最主要的问题四hadoop对应版本的Windows编译jar包没有到位，去下载后还要将hadoop.dll文件放入C:/windows/System32下

hadoop-2.7.3 windows版编译jar包下载地址http://download.csdn.net/download/dongtest/9701830