Floodlight模块分析：forwarding模块

转载请注明出处: http://blog.csdn.net/mageover/article/details/51436775

本文是通过分析一次ping过程来学习forwarding模块，从最简单的拓扑结构入手，研究报文在整个网络中的流向，并深入到forwarding模块的源码。选择forwarding模块进行分析好处在于：这个模块是floodlight应用模块，实现了一个完整的功能，代码简洁却不简单，涉及到了许多其它模块所提供的接口，对理解floodlight的结构很有益处。

阅读本文的读者需要了解OpenFlow1.3协议，大致了解Floodlight的模块编写和OpenFlowJ-Loxigen

OpenFlow 1.3.0: http://www.sdnap.com/wp-content/uploads/openflow/openflow-spec-v1.3.0.pdf

how to write a module: https://floodlight.atlassian.net/wiki/display/floodlightcontroller/How+to+Write+a+Module

how to use OpenFlowJ-Loxigen: https://floodlight.atlassian.net/wiki/display/floodlightcontroller/How+to+use+OpenFlowJ-Loxigen

实验环境：两台vbox虚拟机，一台运行mininet模拟网络，一台运行floodlight控制器

一 ping过程分析

以h1 ping h2为例，整个报文的流程如下：

①　h1------>s1

h1 ping h2，h1不知道h2的mac地址，只知道h2的ip地址

此时，h1会构建一个源mac为h1_mac，目的mac为广播地址的arp请求包，h1发送arp请求包到s1

②　s1------>c0

s1收到arp请求包，发现没有对应流表项，产生OPF_PACKET_IN消息，并递交给c0

③　c0------>s1

c0收到OPF_PACKET_IN，发现是一个广播包，c0产生一个PACKET_OUT消息，要求广播该包，并递交给s1

④　s1------>h2

s1根据PACKET_OUT的指令进行广播，将arp请求包广播至h2

⑤　h2------>s1

h2收到h1发来的arp请求包，发现就是请求的自己的mac地址，于是构建一个arp响应包，发至s1，其源mac为h2_mac，目的mac为h1_mac

⑥　s1------>c0

s1收到arp响应包，发现没有对应流表项，产生OPF_PACKET_IN消息，并递交给c0

⑦　c0------>s1

c0收到OPF_PACKET_IN，发现不是广播包或多播包，进行路由计算，产生OPFlowMod消息和PACKET_OUT消息，更改s1中的流表项

⑧　s1------>h1

s1根据PACKET_OUT的指令发送arp响应包至h1

⑨　h1------>s1

h1已知晓h2的mac地址，发送icmp报文，源mac为h1_mac，目的mac为h2_mac

⑩　s1------>c0

s1收到icmp报文，发现没有对应流表项，产生OPF_PACKET_IN消息，并递交给c0

11　c0------>s1

c0收到OPF_PACKET_IN，发现不是广播或多播，进行路由计算，产生OPFlowMod消息和PACKET_OUT消息，更改s1中的流表项

12　s1------>h2

s1根据PACKET_OUT的指令发送icmp报文发送至h2

13　h2------>s1

h2产生icmp响应报文，源mac为h2_mac，目的mac为h1_mac

14　s1------>c0

s1收到icmp响应报文，发现没有对应流表项，产生OPF_PACKET_IN消息，并递交给c0

15　c0------>s1

c0收到OPF_PACKET_IN，发现不是广播或多播，进行路由计算，产生OPFlowMod消息和PACKET_OUT消息，更改s1中的流表项

16　s1------>h1

s1根据PACKET_OUT的指令发送icmp响应报文发送至h1

整个ping过程结束。

二 forwarding源码探究

整个ping的过程主要分为两部分，一部分是对arp的处理，一部分是对icmp的处理。很多步骤在控制器进行处理时其实是相似的，本文主要对ping过程中的2，3，7步进行详细的分析，其余的步骤中，控制器的处理都是类似这几步的。

②　s1------>c0

s1收到arp请求包，发现没有对应流表项，产生OPF_PACKET_IN消息，并递交给c0

对第二步，要注意不同版本的OpenFlow协议其实是做了不同的处理的，在OpenFlow 1.3之前，OpenFlow交换机在收到没有对应流表项的报文时，默认会产生OFP_PACKET_IN ，如果此时抓到该PACKETIN包会发现其reason字段显示的值是NO_MATCH。

而如果是在OpenFlow1.3协议下，交换机启动后默认会加入一条流表项叫做miss table，该流表项会match任何包，优先级被设置为最低，这一条流表项会告诉交换机对匹配到这条表项的包产生一个PACKETIN消息。如果在OpenFlow1.3协议下抓到OFP_PACKET_IN

包，会发现其reason字段显示的是ACTION，这是匹配到了miss table的结果。如下图：

这是在c0端抓到的h1发送出来的arp请求包，可以看到，由于实验使用的是OpenFlow1.3协议，reason字段显示的是ACTION而不是NO_MATCH，也可以看到arp请求包的目的mac是广播地址ff:ff:ff:ff:ff:ff

③　c0------>s1

c0收到OPF_PACKET_IN，发现是一个广播包，c0产生一个PACKET_OUT消息，要求广播该包，并递交给s1

controller c0收到OPF_PACKET_IN消息之后，就交由forwarding模块进行处理：

从源码中可以看到，Forwarding这个类继承了ForwardingBase，整个消息流程如下

OPF_PACKET_IN——>recieve()——>processPacketInMessage()——>doFlood()——>write()

最核心的方法是processPacketInMessage()，代码如下：

public Command processPacketInMessage(IOFSwitch sw, OFPacketIn pi, IRoutingDecision decision, FloodlightContext cntx) {Ethernet eth = IFloodlightProviderService.bcStore.get(cntx, IFloodlightProviderService.CONTEXT_PI_PAYLOAD);// We found a routing decision (i.e. Firewall is enabled... it's the only thing that makes RoutingDecisions)if (decision != null) {...} else { // No routing decision was found. Forward to destination or flood if bcast or mcast.if (log.isTraceEnabled()) {...}if (eth.isBroadcast() || eth.isMulticast()) {doFlood(sw, pi, cntx);} else {doForwardFlow(sw, pi, cntx, false);}}return Command.CONTINUE;}

判断是广播包，转向doFlood():

protected void doFlood(IOFSwitch sw, OFPacketIn pi, FloodlightContext cntx) {//获取到包进入交换机的入端口OFPort inPort = (pi.getVersion().compareTo(OFVersion.OF_12) < 0 ? pi.getInPort() : pi.getMatch().get(MatchField.IN_PORT));// Set Action to flood//构建PacketOut消息的BuilderOFPacketOut.Builder pob = sw.getOFFactory().buildPacketOut();List<OFAction> actions = new ArrayList<OFAction>();//获取可以广播的端口Set<OFPort> broadcastPorts = this.topologyService.getSwitchBroadcastPorts(sw.getId()); if (broadcastPorts == null) {log.debug("BroadcastPorts returned null. Assuming single switch w/no links.");/* Must be a single-switch w/no links */broadcastPorts = Collections.singleton(OFPort.FLOOD);}//添加action，从所有端口（除了入端口）广播出去for (OFPort p : broadcastPorts) {if (p.equals(inPort)) continue;actions.add(sw.getOFFactory().actions().output(p, Integer.MAX_VALUE));}pob.setActions(actions);// log.info("actions {}",actions);// set buffer-id, in-port and packet-data based on packet-inpob.setBufferId(OFBufferId.NO_BUFFER);pob.setInPort(inPort);pob.setData(pi.getData());try {if (log.isTraceEnabled()) {log.trace("Writing flood PacketOut switch={} packet-in={} packet-out={}",new Object[] {sw, pi, pob.build()});}//将Output消息发送给交换机messageDamper.write(sw, pob.build());} catch (IOException e) {log.error("Failure writing PacketOut switch={} packet-in={} packet-out={}",new Object[] {sw, pi, pob.build()}, e);}return;}

Floodlight将OpenFlow协议中的所有概念都抽象成了不同的类，比如OFAction，OFInstruction，Table等等，而最重要的两个类是OFFactory和Builder类，“工厂”类负责建立所有的builder，而builder类可以build出对应的类，比如doFlood()中使用了OFPacketOut.Builder来构建PacketOut消息。在doFlood()的最后转入到了write()方法，将PacketOut消息下发给交换机。

⑦　c0------>s1

c0收到OPF_PACKET_IN，发现不是广播包或多播包，进行路由计算，产生OPFlowMod消息和PACKET_OUT消息，更改s1中的流表项

在第七步，c0收到了自h2呈递上来的arp响应包，这个时候，源mac是h2_mac，目的mac是h1_mac，不再是广播包或多播包，处理流程有了变化：

OPF_PACKET_IN——>recieve()——>processPacketInMessage()——>doForwardFlow()——>pushRoute()——>write()

                                                                                                                                        |                                   |

                                                                                                                                        |                                    ——>pushPacket()——>write()

                                                                                                                                         ——>createMatchFromPacket()

在processPacketInMessage()方法中，控制器判断出不是广播或多播包，转向doForwardFlow()方法：

protected void doForwardFlow(IOFSwitch sw, OFPacketIn pi, FloodlightContext cntx, boolean requestFlowRemovedNotifn) {//获取包进入交换机时的入端口OFPort inPort = (pi.getVersion().compareTo(OFVersion.OF_12) < 0 ? pi.getInPort() : pi.getMatch().get(MatchField.IN_PORT));//获取目的设备IDevice dstDevice = IDeviceService.fcStore.get(cntx, IDeviceService.CONTEXT_DST_DEVICE);DatapathId source = sw.getId();if (dstDevice != null) {//获取源设备IDevice srcDevice = IDeviceService.fcStore.get(cntx, IDeviceService.CONTEXT_SRC_DEVICE);.../* Validate that the source and destination are not on the same switch port */boolean on_same_if = false;//判断源设备和目的设备是不是在同一个交换机的同一端口，如果是就不做任何处理for (SwitchPort dstDap : dstDevice.getAttachmentPoints()) {if (sw.getId().equals(dstDap.getSwitchDPID()) && inPort.equals(dstDap.getPort())) {on_same_if = true;}break;}if (on_same_if) {log.info("Both source and destination are on the same switch/port {}/{}. Action = NOP", sw.toString(), inPort);return;}//获取目的设备attachmentPoints（笔者判断应该是目的交换机有链路的结点集）SwitchPort[] dstDaps = dstDevice.getAttachmentPoints();SwitchPort dstDap = null;/*  * Search for the true attachment point. The true AP is * not an endpoint of a link. It is a switch port w/o an * associated link. Note this does not necessarily hold * true for devices that 'live' between OpenFlow islands. *  * TODO Account for the case where a device is actually * attached between islands (possibly on a non-OF switch * in between two OpenFlow switches). */for (SwitchPort ap : dstDaps) {if (topologyService.isEdge(ap.getSwitchDPID(), ap.getPort())) {dstDap = ap;break;}}/*  * This should only happen (perhaps) when the controller is * actively learning a new topology and hasn't discovered * all links yet, or a switch was in standalone mode and the * packet in question was captured in flight on the dst point * of a link. */if (dstDap == null) {log.warn("Could not locate edge attachment point for device {}. Flooding packet");doFlood(sw, pi, cntx);return; }/* It's possible that we learned packed destination while it was in flight */if (!topologyService.isEdge(source, inPort)) {log.debug("Packet destination is known, but packet was not received on an edge port (rx on {}/{}). Flooding packet", source, inPort);doFlood(sw, pi, cntx);return; }//进行路由Route route = routingEngineService.getRoute(source, inPort,dstDap.getSwitchDPID(),dstDap.getPort(), U64.of(0)); //cookie = 0, i.e., default route//根据收到的包的特征build matchMatch m = createMatchFromPacket(sw, inPort, cntx);U64 cookie = AppCookie.makeCookie(FORWARDING_APP_ID, 0);if (route != null) {log.debug("pushRoute inPort={} route={} " +"destination={}:{}",new Object[] { inPort, route,dstDap.getSwitchDPID(),dstDap.getPort()});log.debug("Creating flow rules on the route, match rule: {}", m);//按路由下发PacketOut和FlowMod消息pushRoute(route, m, pi, sw.getId(), cookie, cntx, requestFlowRemovedNotifn,OFFlowModCommand.ADD);} else {/* Route traverses no links --> src/dst devices on same switch */...}} else {log.debug("Destination unknown. Flooding packet");doFlood(sw, pi, cntx);}}

doForwardFlow()最后转向pushRoute()：

public boolean pushRoute(Route route, Match match, OFPacketIn pi,DatapathId pinSwitch, U64 cookie, FloodlightContext cntx,boolean requestFlowRemovedNotification, OFFlowModCommand flowModCommand) {boolean packetOutSent = false;//路由所包含的端口序列List<NodePortTuple> switchPortList = route.getPath();//遍历这些端口序列（其实是在遍历路由所要经过的交换机），给每个结点交换机下发正确的流表for (int indx = switchPortList.size() - 1; indx > 0; indx -= 2) {// indx and indx-1 will always have the same switch DPID.DatapathId switchDPID = switchPortList.get(indx).getNodeId();IOFSwitch sw = switchService.getSwitch(switchDPID);if (sw == null) {...}// need to build flow mod based on what type it is. Cannot set command later//FlowMod消息的builderOFFlowMod.Builder fmb;switch (flowModCommand) {case ADD:fmb = sw.getOFFactory().buildFlowAdd();break;...}//用于build action的builderOFActionOutput.Builder aob = sw.getOFFactory().actions().buildOutput();List<OFAction> actions = new ArrayList<OFAction>();//build match的builder Match.Builder mb = MatchUtils.convertToVersion(match, sw.getOFFactory().getVersion()); // set input and output ports on the switchOFPort outPort = switchPortList.get(indx).getPortId();OFPort inPort = switchPortList.get(indx - 1).getPortId();mb.setExact(MatchField.IN_PORT, inPort);aob.setPort(outPort);aob.setMaxLen(Integer.MAX_VALUE);actions.add(aob.build());...fmb.setMatch(mb.build()).setIdleTimeout(FLOWMOD_DEFAULT_IDLE_TIMEOUT).setHardTimeout(FLOWMOD_DEFAULT_HARD_TIMEOUT).setBufferId(OFBufferId.NO_BUFFER).setCookie(cookie).setOutPort(outPort).setPriority(FLOWMOD_DEFAULT_PRIORITY);//给FlowMod消息设定对应的actionsFlowModUtils.setActions(fmb, actions, sw);try {...//将FlowMod消息下放给交换机messageDamper.write(sw, fmb.build());}/* Push the packet out the first hop switch *///如果是路由链路中的第一个收到包的交换机，还要构建PacketOut消息，把包发出去if (sw.getId().equals(pinSwitch) &&!fmb.getCommand().equals(OFFlowModCommand.DELETE) &&!fmb.getCommand().equals(OFFlowModCommand.DELETE_STRICT)) {/* Use the buffered packet at the switch, if there's one stored */pushPacket(sw, pi, outPort, true, cntx);packetOutSent = true;}} catch (IOException e) {log.error("Failure writing flow mod", e);}}return packetOutSent;}

接着是pushPacket()：

protected void pushPacket(IOFSwitch sw, OFPacketIn pi, OFPort outport, boolean useBufferedPacket, FloodlightContext cntx) {...//准备用于构建PacketOut消息的builderOFPacketOut.Builder pob = sw.getOFFactory().buildPacketOut();List<OFAction> actions = new ArrayList<OFAction>();//action设置为outputactions.add(sw.getOFFactory().actions().output(outport, Integer.MAX_VALUE));pob.setActions(actions); ...if (pob.getBufferId().equals(OFBufferId.NO_BUFFER)) {byte[] packetData = pi.getData();pob.setData(packetData);}pob.setInPort((pi.getVersion().compareTo(OFVersion.OF_12) < 0 ? pi.getInPort() : pi.getMatch().get(MatchField.IN_PORT)));try {//下发OutputPacket消息messageDamper.write(sw, pob.build());} catch (IOException e) {log.error("Failure writing packet out", e);}}

三 结语

本文通过分析ping的方法简要的分析了报文在OpenFlow网络中的具体转发过程以及其代码实现，其中涉及Forwarding模块的大部分核心代码，还有一些特殊情况的处理没有提及。希望能通过这样一篇文章引导读者了解Floodlight的一些转发机制，对Floodlight的开发有所助益。