Flume Interceptors

原文：http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.0/ds_Flume/FlumeUserGuide.html

Flume的拦截器可以修改消息内容，Flume提供了集中现成的拦截器，也可以自定义拦截器。拦截器可以组合在一起进行链式拦截。

现成的有：

1）Timestamp Interceptor

This interceptor inserts into the event headers, the time in millis at which it processes the event. This interceptor inserts a header with keytimestamp whose value is the relevant timestamp. This interceptor can preserve an existing timestamp if it is already present in the configuration.

Property NameDefaultDescriptiontype–The component type name, has to be timestamp or the FQCNpreserveExistingfalseIf the timestamp already exists, should it be preserved - true or false

Example for agent named a1:

a1.sources = r1a1.channels = c1a1.sources.r1.channels =  c1a1.sources.r1.type = seqa1.sources.r1.interceptors = i1a1.sources.r1.interceptors.i1.type = timestamp

2）Host Interceptor

This interceptor inserts the hostname or IP address of the host that this agent is running on. It inserts a header with keyhost or a configured key whose value is the hostname or IP address of the host, based on configuration.

Property NameDefaultDescriptiontype–The component type name, has to be hostpreserveExistingfalseIf the host header already exists, should it be preserved - true or falseuseIPtrueUse the IP Address if true, else use hostname.hostHeaderhostThe header key to be used.

Example for agent named a1:

a1.sources = r1a1.channels = c1a1.sources.r1.interceptors = i1a1.sources.r1.interceptors.i1.type = hosta1.sources.r1.interceptors.i1.hostHeader = hostname

3）Static Interceptor

Static interceptor allows user to append a static header with static value to all events.

The current implementation does not allow specifying multiple headers at one time. Instead user might chain multiple static interceptors each defining one static header.

Property NameDefaultDescriptiontype–The component type name, has to be staticpreserveExistingtrueIf configured header already exists, should it be preserved - true or falsekeykeyName of header that should be createdvaluevalueStatic value that should be created

Example for agent named a1:

a1.sources = r1a1.channels = c1a1.sources.r1.channels =  c1a1.sources.r1.type = seqa1.sources.r1.interceptors = i1a1.sources.r1.interceptors.i1.type = statica1.sources.r1.interceptors.i1.key = datacentera1.sources.r1.interceptors.i1.value = NEW_YORK

4）UUID Interceptor

This interceptor sets a universally unique identifier on all events that are intercepted. An example UUID isb5755073-77a9-43c1-8fad-b7a586fc1b97, which represents a 128-bit value.

Consider using UUID Interceptor to automatically assign a UUID to an event if no application level unique key for the event is available. It can be important to assign UUIDs to events as soon as they enter the Flume network; that is, in the first Flume Source of the flow. This enables subsequent deduplication of events in the face of replication and redelivery in a Flume network that is designed for high availability and high performance. If an application level key is available, this is preferable over an auto-generated UUID because it enables subsequent updates and deletes of event in data stores using said well known application level key.

Property NameDefaultDescriptiontype–The component type name has to be org.apache.flume.sink.solr.morphline.UUIDInterceptor$BuilderheaderNameidThe name of the Flume header to modifypreserveExistingtrueIf the UUID header already exists, should it be preserved - true or falseprefix“”The prefix string constant to prepend to each generated UUID

5）Regex Filtering Interceptor

This interceptor filters events selectively by interpreting the event body as text and matching the text against a configured regular expression.The supplied regular expression can be used to include events or exclude events.

Property NameDefaultDescriptiontype–The component type name has to be regex_filterregex”.*”Regular expression for matching against eventsexcludeEventsfalseIf true, regex determines events to exclude, otherwise regex determinesevents to include.

6）Regex Extractor Interceptor

This interceptor extracts regex match groups using a specified regular expression and appends the match groups as headers on the event.It also supports pluggable serializers for formatting the match groups before adding them as event headers.

Property NameDefaultDescriptiontype–The component type name has to be regex_extractorregex–Regular expression for matching against eventsserializers–Space-separated list of serializers for mapping matches to header names and serializing theirvalues. (See example below)Flume provides built-in support for the following serializers:org.apache.flume.interceptor.RegexExtractorInterceptorPassThroughSerializerorg.apache.flume.interceptor.RegexExtractorInterceptorMillisSerializerserializers.<s1>.typedefaultMust be default (org.apache.flume.interceptor.RegexExtractorInterceptorPassThroughSerializer),org.apache.flume.interceptor.RegexExtractorInterceptorMillisSerializer,or the FQCN of a custom class that implements org.apache.flume.interceptor.RegexExtractorInterceptorSerializerserializers.<s1>.name– serializers.*–Serializer-specific properties

The serializers are used to map the matches to a header name and a formatted header value; by default, you only need to specifythe header name and the defaultorg.apache.flume.interceptor.RegexExtractorInterceptorPassThroughSerializer will be used.This serializer simply maps the matches to the specified header name and passes the value through as it was extracted by the regex.You can plug custom serializer implementations into the extractor using the fully qualified class name (FQCN) to format the matchesin anyway you like.

Example 1:

If the Flume event body contained 1:2:3.4foobar5 and the following configuration was used

a1.sources.r1.interceptors.i1.regex = (\\d):(\\d):(\\d)a1.sources.r1.interceptors.i1.serializers = s1 s2 s3a1.sources.r1.interceptors.i1.serializers.s1.name = onea1.sources.r1.interceptors.i1.serializers.s2.name = twoa1.sources.r1.interceptors.i1.serializers.s3.name = three

The extracted event will contain the same body but the following headers will have been addedone=>1, two=>2,three=>3

Example 2:

If the Flume event body contained 2012-10-1818:47:57,614 some log line and the following configuration was used

a1.sources.r1.interceptors.i1.regex = ^(?:\\n)?(\\d\\d\\d\\d-\\d\\d-\\d\\d\\s\\d\\d:\\d\\d)a1.sources.r1.interceptors.i1.serializers = s1a1.sources.r1.interceptors.i1.serializers.s1.type = org.apache.flume.interceptor.RegexExtractorInterceptorMillisSerializera1.sources.r1.interceptors.i1.serializers.s1.name = timestampa1.sources.r1.interceptors.i1.serializers.s1.pattern = yyyy-MM-dd HH:mm

the extracted event will contain the same body but the following headers will have been addedtimestamp=>1350611220000