nutch index路径找不到问题记录

nutch 在抓取网页的时代报如下错误：

2015-06-30 16:25:53,420 DEBUG security.UserGroupInformation - PrivilegedActionException as:hadoop (auth:SIMPLE) cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_data/crawl_fetchInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_data/crawl_parseInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_data/parse_dataInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_data/parse_textInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/content/crawl_fetchInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/content/crawl_parseInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/content/parse_dataInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/content/parse_textInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_generate/crawl_fetchInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_generate/crawl_parseInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_generate/parse_dataInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_generate/parse_textInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_parse/crawl_fetchInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_parse/crawl_parseInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_parse/parse_dataInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_parse/parse_textInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_text/crawl_fetchInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_text/crawl_parseInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_text/parse_dataInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_text/parse_textInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_fetch/crawl_fetchInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_fetch/crawl_parseInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_fetch/parse_dataInput path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_fetch/parse_text

路径找到不到 。该问题在升级nutch 到1.9的时候出现。
通过定位，要找的该路径不对，file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_fetch/parse_text ，每个路径后面多了一个层冗余路径。正确的路径应该是：
/app/apache-nutch-1.9/data/segments/20150630152657/crawl_fetch/ 到20150630152657时间下的一层就可以了。那肯定是路径寻找出问题了。

nutch index的时候找segment路径，是IndexingJob的如下代码实现

 for (int i = 1; i < args.length; i++) {            if (args[i].equals("-linkdb")) {                linkDb = new Path(args[++i]);                //这里开始解析传入的参数路径            } else if (args[i].equals("-dir")) {                Path dir = new Path(args[++i]);                FileSystem fs = dir.getFileSystem(getConf());                //这里列出了所有的子目录，                FileStatus[] fstats = fs.listStatus(dir,                        HadoopFSUtil.getPassDirectoriesFilter(fs));                Path[] files = HadoopFSUtil.getPaths(fstats);                for (Path p : files) {                    segments.add(p);                }            } else if (args[i].equals("-noCommit")) {                noCommit = true;            } else if (args[i].equals("-deleteGone")) {                deleteGone = true;            } else if (args[i].equals("-filter")) {                filter = true;            } else if (args[i].equals("-normalize")) {                normalize = true;            } else if (args[i].equals("-params")) {                params = args[++i];            } else {                segments.add(new Path(args[i]));            }        }

实际上对调用程序传入的 segment path参数又找了一次子目录，而crawl脚本程序传入的路径已经到20150630152657时间这一层了，导致路径多了一层。
crawl脚本如下：

  echo "Indexing $SEGMENT on SOLR index -> $SOLRURL"  "$bin/nutch" index -D solr.server.url=$SOLRURL "$CRAWL_PATH"/crawldb -linkdb "$CRAWL_PATH"/linkdb "$CRAWL_PATH"/segments/$SEGMENT  ---$SEGMENT即是生成的时间路径

   //IndexerMapReduce 初始化job类负责加上最后的 crawl_fetch、crawl_parse 、parse_data、parse_text 路径  public static void initMRJob(Path crawlDb, Path linkDb,                           Collection<Path> segments,                           JobConf job) {    LOG.info("IndexerMapReduce: crawldb: " + crawlDb);    if (linkDb!=null)      LOG.info("IndexerMapReduce: linkdb: " + linkDb);    for (final Path segment : segments) {      LOG.info("IndexerMapReduces: adding segment: " + segment);      //      FileInputFormat.addInputPath(job, new Path(segment, CrawlDatum.FETCH_DIR_NAME));      FileInputFormat.addInputPath(job, new Path(segment, CrawlDatum.PARSE_DIR_NAME));      FileInputFormat.addInputPath(job, new Path(segment, ParseData.DIR_NAME));      FileInputFormat.addInputPath(job, new Path(segment, ParseText.DIR_NAME));    }    FileInputFormat.addInputPath(job, new Path(crawlDb, CrawlDb.CURRENT_NAME));    if (linkDb!=null)      FileInputFormat.addInputPath(job, new Path(linkDb, LinkDb.CURRENT_NAME));    job.setInputFormat(SequenceFileInputFormat.class);    job.setMapperClass(IndexerMapReduce.class);    job.setReducerClass(IndexerMapReduce.class);    job.setOutputFormat(IndexerOutputFormat.class);    job.setOutputKeyClass(Text.class);    job.setMapOutputValueClass(NutchWritable.class);    job.setOutputValueClass(NutchWritable.class);  }

修改方法：

修改crawl脚本 去掉最后的时间路径 或者修改代码不在找子路径