OpenSolaris project Brussels wite paper

The Driver Configuration Framework component of the Brussels project became available in SXCEbuild 83 and is included in the OpenSolaris^TM release. The Framework component providesusers the following benefits when configuring drivers that use the GLDv3 framework:

• A single user interface, dladm, is used to configure network driver properties.

• A uniform syntax is used regardless of the properties: dladm subcommand properties data-link.

• The dladm interface is applied to both public and private properties of the driver.

• Using the tool on a specific driver does not disrupt other network connections.

This article elaborates on the administrative features provided by the Brussels project.

Introduction and Problem Description

Prior to the Brussels implementation, configuring data–link parameters on a machine running theSolaris^TM Operating System (Solaris OS) was needlessly difficult. Commands and interfaces were crypticand inconsistent across the drivers, and the process often required service interruptionto all connected networks, regardless of whether the connecting link was actually being impactedby the change.

An example of the deficiencies in these administrative methods is the refining ofinterface maximum transmission unit (MTU) that is typically done to enable support forthe Jumbo Frame MTU. The complications are evident as described in an inquiry in the Sun Managers Mailing List.

Overview of NIC Driver Properties

NIC driver properties that configurable by using the dladm command fall into oneof two categories:

• Public properties that can be applied to any driverof the given media type such as link speed, autonegotiation forEthernet, or the MTU size that can be applied to all data-link drivers.

• Private properties that are particular to a certainsubset of drivers for a given media type. These properties can bespecific to that subset because they are closely related either to thehardware that is associated with the driver or to the details of thedriver implementation itself, such as debugging-related tunables.

Properties of a NIC driver are typically set with default values. However, certainnetworking scenarios might require you to change specific property settings of a NIC.These property settings can be either public or private properties. For example, aNIC might be communicating with an old switch that does not properly performautonegotiation. Or, a switch might have been configured to support Jumbo frames. Or,driver specific properties that regulate packet transmission or packet receiving might need tobe modified for the given driver. With the implementation of the Brussels project,these settings can now be reset by a single administrative tool, dladm.

dladm Subcommands to Administer NIC Properties

For NIC drivers that have been converted to the GLDv3 framework, properties areconfigured by using the dladm command. This command enables you to configure theproperties dynamically without causing any network disruption on other NICs of similar types.The values that you set are stored in a dladm repository and persisteven after you reboot the system or unplumb the interface. Therefore, use dladmas the preferred command to configure NICs, instead of the ndd command.

To administer NIC drivers, you use the following dladm subcommands:

• dladm show-linkprop displays the properties that are associated with the data link.

• dladm set-linkprop sets values for specified data-link properties.

• dladm reset-linkprop restores property settings to the default values.

• dladm show-ether displays Ethernet parameter settings of a data link.

For more information about these commands, see the dladm(1M) man page.

---

Note - Customizing NIC properties by using the dladm command is supported only in networkdrivers that have been converted to the GLDv3 framework, such as the following:

• bge – supported in SXCE build 83

• nge – supported in SXCE build 86

• e1000g – supported in SXCE build 88

• nxge – supported in SXCE build 88

Work continues to make other drivers become supported in the Brussels implementation. Toconfirm whether your specific driver supports this feature, refer to the driver's manpage.

---

The following section provides procedures to set certain NIC driver properties. The selectedproperties are public and common to all NIC drivers. A separate section describesdriver specific properties and also provides procedures to configure selected private properties ofthe e1000g driver..

Procedures and Examples

The following section provides procedures with examples that show how to configure publicdrivers by using the dladm command.

How to Enable Support for Jumbo Frames

Enabling support for Jumbo frames in a network setup is a commontask for most network scenarios. Support for Jumbo frames requires increasing the size ofa data link's maximum transmission unit (MTU). The following procedure includes the useof customized names to identify data links, a feature that is introduced bythe Clearview project. This Clearview overview document introduces the concept of customized link names.

1. On the system that has the link whose MTU you want tomodify, assume the System Administrator role.

   The System Administrator role includes the Network Management profile. To create the roleand assign the role to a user, see Chapter 9, Using Role-Based Access Control (Tasks) in System Administration Guide: Security Services.

2. Identify the specific Ethernet device whose MTU size you need to reset, displaythe links in the system.

   # dladm show-phys

   You might need to perform this step if your network configuration uses customizednames for data links. With customized names, data links are no longer necessarilyidentified by their hardware-based names. For example, the Ethernet device is bge0. However,the data link over the device is renamed net0. Therefore, you would needto configure the MTU size of net0. Refer to the documentation in the Clearview project page in OpenSolarisfor examples of configuration tasks on data links that use customized names.

3. (Optional) Display the data link's current MTU size and other properties.
   • To display a specific property of a data link, use the following syntax:

     dladm show-linkprop -p property data-link

     This command displays the settings of the property that you specify.

   • To display several selected properties of the data link, use the following syntax:

     # dladm show-link data-link

     This command displays data-link information, including MTU size.

   ---

   Note - See How to Display Data Link in the Project Clearview documentation page for additional examplesof the use of the dladm show-link syntax to display data-link information.

   ---

4. Unplumb the interface that is configured over the data link.

   # ifconfig interface unplumb

5. Change the value of the link's MTU size to 9000, the valuefor Jumbo frames.

   # dladm set-linkprop -p mtu=9000 data-link

6. Plumb the IP interface over the link.

   # ifconfig interface plumb IP-address up

   For additional options that you can use with the ifconfig command, see theifconfig(1M) man page.

7. (Optional) Verify that the interface uses the new MTU size by using oneof the command syntaxes in Step 3.

   # dladm show-linkprop -p mtu data-link

8. (Optional) Display the link's current Ethernet settings.

   # dladm show-ether data-link

Example 1 Enabling Support for Jumbo Frames

The following example of enabling support for Jumbo frames builds on the followingscenario:

• The system has two bge NICS: bge0 and bge1.

• The device bge0 is used as a primary interface, while the device bge1 is used for test purposes.

• You want to enable support for Jumbo frames on bge1, while you retain the default MTU size of the primary interface.

• The network configuration uses customized names for data links. The link name of bge0 is net0. The link name of bge1 is web1.

# dladm show-phys
LINK       MEDIA        STATE     SPEED     DUPLEX     DEVICE
net0       ether        up        100Mb     full       bge0
itops1     ether        up        100Mb     full       qfe3
web1       ether        up        100Mb     full       bge1

# dladm show-linkprop -p mtu web1
LINK     PROPERTY     VALUE     DEFAULT     POSSIBLE
web1     mtu          1500      1500        --

# ifconfig web1 unplumb
# dladm set-linkprop -p mtu=9000 web1
# ifconfig web1 plumb 10.10.1.2/24 up

# dladm show-link web1
LINK     CLASS     MTU      STATE     OVER
web1     phys      9000     up        --

Notice that the MTU value is now 9000. In this example, thedladm command enabled you to change web1's MTU size directly. The previous method wouldhave required you to unplumb net0 as well, which would have unnecessarily disruptedthe primary interface's operations.

How to Change Link Speed Parameters

Most network setups consist of a combination of systems with varying speed capabilities.For example, the advertised speed between an older system and a newer systemmight need to be changed to a lower setting to allow communication. Bydefault, all the speed and duplex capabilities of a NIC card areadvertised. This procedure shows how to turn off the gigabit capabilities and advertise onlythe megabit capabilities.

1. On the system that has the NIC whose properties you want tomodify, assume the System Administrator role.
2. (Optional) Display the current status of the property you want to modify.

   # dladm show-linkprop -p property data-link

3. To advertise lower speed capabilities, turn off the higher speed capabilities to preventthem from being advertised.

   # dladm set-linkprop -p property=value1 data-link

Example 2 Disabling Advertisement of a NIC's Gigabit Capabilities

This example shows how you can prevent the link web1 from advertising gigabitcapabilities.

# dladm show-linkprop -p adv_1000fdx_cap web1
LINK     PROPERTY             VALUE     DEFAULT     POSSIBLE
web1     adv_1000fdx_cap      1         --          1,0

# dladm show-linkprop -p adv_1000hdx_cap web1
LINK     PROPERTY             VALUE     DEFAULT     POSSIBLE
web1     adv_1000hdx_cap      1         --          1,0

The properties that advertise the link's gigabit capabilities are adv_1000fdx_cap and adv_1000hdx_cap.To disable these properties from being advertised, you would type the following commands:

# dladm set-linkprop -p adv_1000fdx_cap=0 web1
# dladm set-linkprop -p adv_1000hdx_cap=0 web1

Listing the Ethernet parameter settings would display the following output:

# dladm show-ether web1
LINK     PTYPE       STATE    AUTO  SPEED-DUPLEX             PAUSE
web1     current     up       yes   1G-f                     both

How to Obtain Status Information About NIC Properties

You can obtain information about the NIC driver's properties by displaying either theEthernet parameter settings or the link properties.

1. On the system that has the NIC whose properties you want tomodify, assume the System Administrator role.
2. To obtain information about the Ethernet parameter settings, use the following command:

   # dladm show-ether [-x] data-link

   where the -x option includes additional parameter information about the link. Without the-x option, only the current parameter settings are displayed.

3. To obtain information about all the properties of the link, use the followingcommand:

   # dladm show-linkprop data-link

Example 3 Displaying Ethernet Parameter Settings

This example display an extended list of parameter information about a specified link.

# dladm show-ether -x web1
LINK     PTYPE       STATE    AUTO  SPEED-DUPLEX             PAUSE
web1     current     up       yes   1G-f                     both
--       capable     --       yes   1G-fh,100M-fh,10M-fh     both
--       adv         --       yes   100M-fh,10M-fh           both
--       peeradv     --       yes   100M-f,10M-f             both

With the -x option, the command also displays the built-in capabilities of thespecified link, as well as the capabilities that are currently advertised between thehost and the link partner. The following information is displayed:

• For the Ethernet device's current state, the link is up andfunctioning at 1 gigabits per second at full duplex. Itsautonegotiation capability is enabled and has bidirectional flowcontrol, in which both host and link partner can send and receive pauseframes.

• Regardless of the current setting, the capabilities of theEthernet device are listed. The negotiation type can be set toautomatic, the device can support speeds of 1 gigabits per second, 100megabits per second, and 10 megabits per second, at both full and halfduplex. Likewise, pause frames can be received or sent in bothdirections between host and link partner.

• The following capabilities of web1 are advertised: autonegotiation, speed-duplex, and flow control of pause frames.

• Similarly, the link partner advertises the followingcapabilities: autonegotiation, speed-duplex, and flow control of pauseframes.

Example 4 Displaying Link Properties

This example shows how to list all the properties of a link.If you want to display only a specific property, you use the -poption with the specific property that you want to monitor.

# dladm show-linkprop web1
LINK     PROPERTY             VALUE       DEFAULT     POSSIBLE
web1     speed                1000        --          -- 
web1     autopush             --          --          -- 
web1     zone                 --          --          -- 
web1     duplex               half        --          half,full 
web1     state                unknown     up          up,down 
web1     adv_autoneg_cap      1           1           1,0 
web1     mtu                  1500        1500        -- 
web1     flowctrl             no          bi          no,tx,rx,bi 
web1     adv_1000fdx_cap      1           1           1,0 
web1     en_1000fdx_cap       1           1           1,0 
web1     adv_1000hdx_cap      1           1           1,0 
web1     en_1000hdx_cap       1           1           1,0 
web1     adv_100fdx_cap       0           0           1,0 
web1     en_100fdx_cap        0           0           1,0 
web1     adv_100hdx_cap       0           0           1,0 
web1     en_100hdx_cap        0           0           1,0 
web1     adv_10fdx_cap        0           0           1,0 
web1     en_10fdx_cap         0           0           1,0 
web1     adv_10hdx_cap        0           0           1,0 
web1     en_10hdx_cap         0           0           1,0

The settings for the speed and duplex capabilities of the link aremanually configured on the enabled-speed properties which are labeled en_*_cap. For example, en_1000fdx_cap isthe property for the gigabit full-duplex capability, and en_100hdx_cap is the property for the100 megabits half-duplex capability. The settings of these enabled speed properties are advertisedbetween the host and its link partner by corresponding advertised speed properties, whichare labeled adv_*_cap such as adv_1000fdx_cap and adv_100hdx_cap. Normally, the settings of agiven enabled speed property and the corresponding advertised property are identical. However, ifa NIC supports some advanced features such as Power Management, those features mightset limits on the bits that are actually advertised between the host andits link partner. For example, with Power Management, the values of the adv_*_capproperties might only be a subset of the values of the en_*_cap properties. For more details about the enabled and advertised speed properties, see thedladm(1M) man page.

Configuring Properties Specific to the e1000g Driver

Brussels support for the e1000g driver is available in SXCE build 88. The driver is widely used on many Sun platforms to support the Intel^® PRO/1000 Gigabit NICs, such as the Sun Fire^TM X4200 server, the Sun Netra^TM X4450 server,and the Sun Fire T2000 server.

Previously, the e1000g driver, like many other NIC drivers, was configured by usingthe ndd command and by defining property settings in the driver's e1000g.conf configuration file.This driver is now converted to the GLDv3 framework. Thus, the administration ofits configuration can be uniformly performed just as other similarly converted drivers withthe dladm command.

The same command can be used to adjust parameters that are specific tothe e1000g driver. For example, you might need to configure the driver touse Direct Memory Access (DMA) binding for transmission instead of the bcopy mode.Likewise, you might need to reset certain interrupt parameters to improve the performanceof the driver. See the procedures that show how to configure these parameters.

---

Note - The nxge driver has become Brussels-supported in SXCE build 88. The driver, whichis widely used on many platforms, also has nxge-specific properties. The following procedures aboutconfiguring private properties apply to the e1000g driver. However, the general procedures canalso apply to the nxge driver.

---

Procedures and Example

This section provides procedures and an example that show how to configure selectedprivate properties of the e1000g driver.

How to Set the e1000g Driver to Use Direct Memory Access Binding

Bulk traffic, such as file transfers, normally involves negotiation of large packets acrossthe network. In such cases, you can obtain better performance from the e1000gdriver by configuring it to automatically use DMA binding.

In the following procedure, a threshold is defined for packet fragment sizes. Ifa fragment size surpasses the threshold, then DMA binding is used for transmitting.If a fragment size is within the threshold, then bcopy mode isused, where the fragment data is copied to the preallocated transmit buffer. Formore details about this threshold and other tunables, see the e1000 tunables list in OpenSolaris.

To set the threshold, perform the following steps:

1. On the system that has the NIC whose properties you want tomodify, assume the System Administrator role.
2. Set the appropriate value for the _tx_bcopy_threshold property.

   # dladm set-linkprop -p _tx_bcopy_threshold=value e1000g-data-link

   For this property, the valid values for the threshold range from 60 through2048.

   ---

   Note - As with configuring public properties, the interface must also be unplumbed before privateproperty settings can be modified.

   ---

3. (Optional) Verify the new threshold value.

   # dladm show-linkprop -p _tx_bcopy_threshold e1000g-data-link

How to Manually Set the Interrupt Rate

Parameters that regulate the rate at which interrupts are delivered by the e1000gdriver also affect network and system performance. Typically network packets are delivered tothe upper layer of the stack by generating an interrupt for every packet.In turn the interrupt rate, by default, is automatically adjusted by the GLDlayer in the kernel. However, this mode might not be desirable in allnetwork traffic conditions. For a discussion of this issue, refer to this document(http://www.stanford.edu/class/cs240/readings/mogul.pdf) that was presented at the USENIX technical conference in 1996. Thus, incertain circumstances, setting the interrupt rate manually becomes necessary to obtain better performance.

To define the interrupt rate, you set the following parameters:

• _intr_throttling_rate determines the delay between interrupt assertions regardless of network traffic conditions.

• _intr_adaptive determines whether automatic tuning of the interrupt throttling rate is enabled. By default, this parameter is enabled.

1. On the system that has the NIC whose driver properties you want tomodify, assume the System Administrator role.
2. If necessary, identify the device whose driver property you want to modify.

   # dladm show-phys

3. Disable automatic tuning of the interrupt throttling rate.

   # dladm set-linkprop -p _intr_adaptive=0 e1000g-data-link

   ---

   Note - When automatic tuning of the interrupt throttling rate is enabled, then any valuethat is set for the parameter _intr_throttling_rate is ignored.

   ---

4. Unplumb the network interface.
5. Set the value for the minimum inter interrupt level.

   # dladm set-linkprop -p _intr_throttling_rate=value e1000g-data-link

   ---

   Note - The default value of the _intr_throttling_rate parameter is 550 on SPARC^® based systemsand 260 on x86 based systems. Setting the minimum inter-interrupt level to 0disables the interrupt throttling logic.

   ---

6. Plumb the interface and configure an IP address for the interface.
7. (Optional) Display the threshold's new settings.

Example 5 Configuring for DMA Binding and Setting the Interrupt Throttling Rate

This example uses an x86 based system with an e1000g NIC. The driveris configured with a threshold setting toggle between using DMA binding or thebcopy mode for transmitting packets. The setting for the interrupt throttling rate isalso modified. Further, the e1000g data link has been renamed with acustomized name. Therefore, the configuration is performed on the data link by referringto the customized name, public0.

# dladm show-phys
LINK       MEDIA        STATE     SPEED     DUPLEX     DEVICE
public0    ether        up        100Mb     full       e1000g0

# dladm show-linkprop -p _tx_bcopy_threshold public0
LINK        PROPERTY                VALUE     DEFAULT     POSSIBLE
public0     _tx_bcopy_threshold     512       512         --

# dladm show-linkprop -p _intr-throttling_rate
LINK        PROPERTY                  VALUE     DEFAULT     POSSIBLE
public0     _intr-throttling_rate     260       260         --

# ifconfig public0 unplumb
# dladm set-linkprop -p _tx_bcopy_threshold=1024 public0
# dladm set-linkprop -p _intr_adaptive=0 public0
# dladm set-linkprop -p _intr-throttling_rate=1024 public0
# ifconfig public0 plumb 10.10.1.2/24 up

# dladm show-linkprop -p _tx_bocopy_threshold=1024 public0
LINK        PROPERTY                VALUE     DEFAULT     POSSIBLE
public0     _tx_bcopy_threshold     1024      512         --

# dladm show-linkprop -p _intr_adaptive public0
LINK        PROPERTY           VALUE     DEFAULT     POSSIBLE
public0     _intr-adaptive     0         1           --

# dladm show-linkprop -p _intr-throttling_rate
LINK        PROPERTY                  VALUE     DEFAULT     POSSIBLE
public0     _intr-throttling_rate     1024      260         --

Ongoing Work

The Brussels ndd compat component will provide additional functionality in the Framework for legacysupport of the ndd(1m) commands so that drivers will not have to includecomplex and undocumented interfaces to support the ndd command.

Also, the Brussels Persistence component will add additional functionality to allow the propertysettings to be automatically applied across reboot and driver restart.

Future articles will describe the details of the Framework, as well as themethods used to provide ndd compatibility and persistent property settings.

Work also continues to provide a GUI to administer data links. Aprototype of the GUI is available at the OpenSolaris project page. The prototypeis an interactive GUI for demonstration purposes. Please send feedback to the Brussels discussion list.

Access to the Brussels-supported command-line interfaces is now available in the OpenSolaris 2008.05release.

Changes subject to http://opensolaris.org/os/project/brussels/Documentation/wp/