Beam学习笔记（3）：Flink Streaming Pipeline Translator

来源：互联网发布：大数据o2o概念股龙头编辑：程序博客网时间：2024/05/29 18:32

FlinkStreamingPipelineTranslator

1 所有的可用Translator：

// here you can find all the available translators.  static {    TRANSLATORS.put(Read.Bounded.class, new BoundedReadSourceTranslator());    TRANSLATORS.put(Read.Unbounded.class, new UnboundedReadSourceTranslator());    TRANSLATORS.put(TextIO.Write.Bound.class, new TextIOWriteBoundStreamingTranslator());    TRANSLATORS.put(ParDo.MultiOutput.class, new ParDoStreamingTranslator());    TRANSLATORS.put(        SplittableParDo.ProcessElements.class, new SplittableProcessElementsStreamingTranslator());    TRANSLATORS.put(        SplittableParDo.GBKIntoKeyedWorkItems.class, new GBKIntoKeyedWorkItemsTranslator());    TRANSLATORS.put(Window.Assign.class, new WindowAssignTranslator());    TRANSLATORS.put(Flatten.PCollections.class, new FlattenPCollectionTranslator());    TRANSLATORS.put(        FlinkStreamingViewOverrides.CreateFlinkPCollectionView.class,        new CreateViewStreamingTranslator());    TRANSLATORS.put(Reshuffle.class, new ReshuffleTranslatorStreaming());    TRANSLATORS.put(GroupByKey.class, new GroupByKeyTranslator());    TRANSLATORS.put(Combine.PerKey.class, new CombinePerKeyTranslator());  }

2 applyStreamingTransform：

基本逻辑：将PTransform关联到streamingContext

// create the applied PTransform on the streamingContext    streamingContext.setCurrentTransform(node.toAppliedPTransform());    typedTranslator.translateNode(typedTransform, streamingContext);

12种类型的translator多态实现了StreamTransformTranslator.translateNode(T transform, FlinkStreamingTranslationContext context);方法。

0 0