Flume agent端event重复发送（数据暴增不一致）的问题

       一直用flume做数据收集，用了好一阵子都没出现数据误差的问题，今天在导入数据的时候却突然出现了数据暴增的问题，查看原始数据文件，只有几十万条数据，可却收集到了几百万的event，而且还在持续的增加。很奇怪，首先以为是插件更新的问题，后来换成最原始的console作为Collector的sink，问题依旧存在。然后清理配置数据重新启动，问题还是没有解决。

      查看原始数据并与原有的数据对比，发现有超大内容的的event，是业务端的错误。所以估计应该是大event的问题，然后查看源码，发现异样

public class SyslogUdpSource extends EventSource.Base {  static final Logger LOG = LoggerFactory.getLogger(SyslogUdpSource.class);  final public static int SYSLOG_UDP_PORT = 514;  int port = SYSLOG_UDP_PORT; // default udp syslog port  int maxsize = 1 << 16; // 64k is max allowable in RFC 5426  long rejects = 0;  DatagramSocket sock;  public SyslogUdpSource() {  }  public SyslogUdpSource(int port) {    this.port = port;  }  @Override  public void close() throws IOException {    LOG.info("closing SyslogUdpSource on port " + port);    if (sock == null) {      LOG.warn("double close of SyslogUdpSocket on udp:" + port          + " , (this is ok but odd)");      return;    }    sock.close();  }  @Override  public Event next() throws IOException {    byte[] buf = new byte[maxsize];    DatagramPacket pkt = new DatagramPacket(buf, maxsize);    Event e = null;    do { // loop until we get a valid packet, drop bad ones.      sock.receive(pkt);      ByteBuffer bb = ByteBuffer.wrap(buf, 0, pkt.getLength());      ByteBufferInputStream bbis = new ByteBufferInputStream(bb);      DataInputStream in = new DataInputStream(bbis);      try {        e = SyslogWireExtractor.extractEvent(in);      } catch (EventExtractException ex) {        rejects++;        LOG.warn(rejects + " rejected packets. packet: " + pkt, ex);        LOG.debug("raw bytes " + Arrays.toString(pkt.getData()));        // TODO (jon) maybe have a hook here to do something with rejects      }      // need a sane way to fall out of his loop.    } while (e == null);    updateEventProcessingStats(e);    return e;  }  @Override  public void open() throws IOException {    sock = new DatagramSocket(port);  }  public static SourceBuilder builder() {    return new SourceBuilder() {      @Override      public EventSource build(Context ctx, String... argv) {        int port = SYSLOG_UDP_PORT; // default udp port, need root permissions        // for this.        if (argv.length > 1) {          throw new IllegalArgumentException("usage: syslogUdp([port no]) ");        }        if (argv.length == 1) {          port = Integer.parseInt(argv[0]);        }        return new SyslogUdpSource(port);      }    };  }}

64k is max allowable in RFC 5426（有关RFC 5426  event大小的限制）

 

然后网上查看各种资料，原来当event大于64K时，flume agent端会重复发送该event，从而导致数据暴增的问题（更详细的原因有待有空的时候考证）。所以应该在业务端控制好event的大小。