sqoop 抽取源码流程分析(一) 主流程分析以及各种插件

1. 工具类的继承关系

目前sqoop 提供的各种工具，比如 ImportTool, ExportTool, CodeGenTool 等工具类都是集成于 BaseSqoopTool 这个基础类
下面看看这个BaseSqoopTool 基础类的继承关系

public abstract class BaseSqoopTool    extends com.cloudera.sqoop.tool.SqoopTool {......public abstract class SqoopTool    extends org.apache.sqoop.tool.SqoopTool {......public abstract class SqoopTool {

2. sqoop 之 main 函数

2.1 sqoop 继承关系

public class Sqoop extends Configured implements Tool {......

其中的Tool 和 Configured 是hadoop 中的基础类，毕竟sqoop 之用hadoop 来做的具体实现。

2.2 主函数

 public static void main(String [] args) {    if (args.length == 0) {      System.err.println("Try 'sqoop help' for usage.");      System.exit(1);    }    int ret = runTool(args);    System.exit(ret);  }

2.3 runTool 的实现

runTool 更具传入的sqoop 参数，选择需要使用的工具类名称来构建相应的工具类

......   SqoopTool tool = SqoopTool.getTool(toolName);......   Sqoop sqoop = new Sqoop(tool, pluginConf);    return runSqoop(sqoop, Arrays.copyOfRange(expandedArgs, 1, expandedArgs.length));

那么具体的运行就是在各个工具类的具体实现里面了。
具体的实现可以参考对importTool的具体分析了

2.4 隐藏的关系

 public static int runSqoop(Sqoop sqoop, String [] args) {    try {      String [] toolArgs = sqoop.stashChildPrgmArgs(args);      return ToolRunner.run(sqoop.getConf(), sqoop, toolArgs);......

虽然是在hadoop 里面实现的这个run 方法，不过可以通过下面的具体实现来看出，其实调用的还是sqoop的run方法

 public static int run(Configuration conf, Tool tool, String[] args)     throws Exception{    if(conf == null) {      conf = new Configuration();    }    GenericOptionsParser parser = new GenericOptionsParser(conf, args);    //set the configuration back, so that Tool can configure itself    tool.setConf(conf);    //get the args w/o generic hadoop args    String[] toolArgs = parser.getRemainingArgs();    return tool.run(toolArgs);

2.5 真正的入口点

 public int run(String [] args) {    if (options.getConf() == null) {      // Configuration wasn't initialized until after the ToolRunner      // got us to this point. ToolRunner gave Sqoop itself a Conf      // though.      options.setConf(getConf());    }    try {      options = tool.parseArguments(args, null, options, false);      tool.appendArgs(this.childPrgmArgs);      tool.validateOptions(options);    } catch (Exception e) {      // Couldn't parse arguments.      // Log the stack trace for this exception      LOG.debug(e.getMessage(), e);      // Print exception message.      System.err.println(e.getMessage());      return 1; // Exit on exception here.    }    return tool.run(options);  }

2.5.1 参数解析

public SqoopOptions parseArguments(String [] args,      Configuration conf, SqoopOptions in, boolean useGenericOptions)      throws ParseException, SqoopOptions.InvalidOptionsException {...... // Parse tool-specific arguments.    ToolOptions toolOptions = new ToolOptions();    configureOptions(toolOptions);    CommandLineParser parser = new SqoopParser();    CommandLine cmdLine = parser.parse(toolOptions.merge(), toolArgs, true);    applyOptions(cmdLine, out);    this.extraArguments = cmdLine.getArgs();

可以看出真正的参数解析是在SqoopParser这个类中

那么继续看SqoopParser 的parse方法

 public void processArgs(Option opt, ListIterator iter)      throws ParseException {    // Loop until an option is found.    while (iter.hasNext()) {      String str = (String) iter.next();      if (getOptions().hasOption(str) && str.startsWith("-")) {        // found an Option, not an argument.        iter.previous();        break;      }      // Otherwise, this is a value.      try {        // Note that we only strip matched quotes here.        addValForProcessing.invoke(opt, stripMatchedQuotes(str));      } catch (IllegalAccessException iae) {        throw new RuntimeException(iae);      } catch (java.lang.reflect.InvocationTargetException ite) {        // Any runtime exception thrown within addValForProcessing()        // will be wrapped in an InvocationTargetException.        iter.previous();        break;      } catch (RuntimeException re) {        iter.previous();        break;      }    }    if (opt.getValues() == null && !opt.hasOptionalArg()) {      throw new MissingArgumentException(opt);    }  }