Yarn核心——Protobuf

Yarn将Protobuf用在RPC通信中，基于Protobuf的方式进行通信。Yarn RPC中的所有参数都采用Protocol Buffer进行序列化和反序列化，体积小，速度快。

(1)Proto文件 student.proto

package tutorial;option java_package="com.jackniu.hadoop_yarn.com.proto";option java_outer_classname="StudentProtos";message Student{    required int32 ID=1;    required string name=2;    required string sex=3;        message StudentPhone{        required  string number=1;        optional int32 type=2;        }    repeated StudentPhone phones=4;}

(2) 编译

bin/protoc  --java_out=javacode  code/student.proto

(3) 编译完成部分代码

package com.jackniu.hadoop_yarn.com.proto;public final class StudentProtos {  private StudentProtos() {}  public static void registerAllExtensions(      com.google.protobuf.ExtensionRegistry registry) {  }  public interface StudentOrBuilder      extends com.google.protobuf.MessageOrBuilder {    // required int32 ID = 1;    /**     * <code>required int32 ID = 1;</code>     */    boolean hasID();    /**     * <code>required int32 ID = 1;</code>     */    int getID();

(4) 使用

package com.jackniu.hadoop_yarn.com.proto;/** * Created by JackNiu on 2017/9/1. */public class ProtocolBufferExample {    public static void main(String[] args) {        StudentProtos.Student student = StudentProtos.Student.newBuilder().setName("jack").setID(1).setSex("2")                .addPhones(StudentProtos.Student.StudentPhone.newBuilder().setNumber("12345").setType(1))                .addPhones(StudentProtos.Student.StudentPhone.newBuilder().setNumber("67890").setType(2))                .build();        System.out.println(student);        // 在发送端发送     }}ID: 1name: "jack"sex: "2"phones {  number: "12345"  type: 1}phones {  number: "67890"  type: 2}

(5) 发送和读取

package com.jackniu.hadoop_yarn.com.proto;import org.apache.hadoop.mapred.FileOutputFormat;import java.io.FileInputStream;import java.io.FileOutputStream;/** * Created by JackNiu on 2017/9/1. */public class ProtocolBufferExample {    public static void main(String[] args)  throws  Exception{        StudentProtos.Student student = StudentProtos.Student.newBuilder().setName("jack").setID(1).setSex("2")                .addPhones(StudentProtos.Student.StudentPhone.newBuilder().setNumber("12345").setType(1))                .addPhones(StudentProtos.Student.StudentPhone.newBuilder().setNumber("67890").setType(2))                .build();        System.out.println(student);        //在发送端发送        FileOutputStream output = new FileOutputStream("example.txt");        student.writeTo(output);        output.close();        FileInputStream inputStream = new FileInputStream("example.txt");        StudentProtos.Student receive_student = StudentProtos.Student.parseFrom(inputStream);        System.out.println(receive_student);     }}输出: example.txt
     jack
      2" 12345    
"   67890控制台输出ID: 1name: "jack"sex: "2"phones {  number: "12345"  type: 1}phones {  number: "67890"  type: 2}ID: 1name: "jack"sex: "2"phones {  number: "12345"  type: 1}phones {  number: "67890"  type: 2}

Yarn中通信都是通过protobuf定义的，默认实现protobuf。