Overview of Cisco usNIC

Cisco usNIC Deployment Guide for Cisco UCS B-Series Blade Servers,
Release 2.2
Overview of Cisco usNIC 2
Cisco usNIC Prerequisites 3
Configuring Cisco usNIC in Cisco UCS Manager 4
Installing Linux Drivers for Cisco usNIC 9
Manually Loading the Kernel Modules for Cisco usNIC 10
Uninstalling Linux Drivers for Cisco usNIC 11
Verifying the Cisco usNIC Installation 11
Revised: May 30, 2014, OL-30794-01
Overview of Cisco usNIC
The Cisco user-space NIC (Cisco usNIC) feature improves the performance of software applications that run on the Cisco UCS
servers in your data center by bypassing the kernel when sending and receiving networking packets. The applications interact directly
with a Cisco UCS VIC second generation adapter, such as the Cisco UCS VIC-1280, which improves the networking performance
of your high-performance computing cluster. To benefit from Cisco usNIC, your applications must use the Message Passing Interface
(MPI) instead of sockets or other communication APIs.
Cisco usNIC offers the following benefits for your MPI applications:
• Provides a low-latency and high-throughput communication transport.
• Employs the standard and application-independent Ethernet protocol.
• Takes advantage of lowlatency forwarding, Unified Fabric, and integrated management support in the following Cisco data
center platforms:
• Cisco UCS server
• Cisco UCS VIC second generation adapter, such as the Cisco UCS VIC-1280
• 10GbE network
2
Standard Ethernet applications use user-space socket libraries, which invoke the networking stack in the Linux kernel. The networking
stack then uses the Cisco eNIC driver to communicate with the Cisco VIC hardware. The following figure shows the contrast between
a regular software application and an MPI application that uses usNIC.
Figure 1: Kernel-Based Network Communication versus Cisco usNIC-Based Communication
Cisco usNIC Prerequisites
To benefit from Cisco usNIC, your configuration has the following prerequisites:
• Cisco Open Message Passing Interface (MPI) distribution.
• UCS Driver ISO (not inbox).
• RHEL 6.4 support.
3
Configuring Cisco usNIC in Cisco UCS Manager
Before You Begin
Make sure that the following software and hardware components are installed on the Cisco UCS server:
• Red Hat Enterprise Linux (RHEL) version 6.4 or later
• RHEL gcc-c++ and gcc-gfortran RPMs
• RDMA yum install librdmacm-utils and yum install rdma-3.6-1.el6.noarch RPMs
• Cisco UCS VIC second generation adapter, such as the Cisco UCS VIC-1280
Procedure
Step 1
Step 2
Configure the Cisco usNIC properties and BIOS settings using Cisco UCS Manager GUI or Cisco UCS Manager CLI.
Enable the Intel IOMMU driver in the Linux kernel (intel_iommu=on).
You must enable this setting in the kernel boot command to allow user-space applications to properly communicate with
Cisco usNIC.
Step 3
Manually add 'intel_iommu =on' in the grub.conf file (/boot/grub/grub.conf):
KEYBOARDTYPE=pc KEYTABLE=us rd NO DM rhgb quiet intel_iommu=on
Step 4
iommu verification cmd dmesg | grep -i commnad
Verify the configuration.
$ ibv_devinfo
Step 5
Reboot your Cisco UCS server.
You must reboot your server for the changes to take effect after you configure Cisco usNIC.
Step 6
Install the Cisco usNIC Linux drivers.
For more information about installing the drivers, see Installing Linux Drivers for Cisco usNIC, on page 9.
What to Do Next
After you complete configuring Cisco usNIC and installing the Linux drivers, verify that Cisco usNIC is functioning properly. For
more information about how to verify the installation, see Verifying the Cisco usNIC Installation, on page 11.
4
Configuring a Cisco usNIC
Procedure
Step 1
In the Navigation pane, click the LAN tab.
Step 2
Step 3
Step 4
Step 5
On the LAN tab, expand LAN > Policies.
Expand the root node.
Right-click usNIC Connection Policies and choose Create usNIC Connection Policy.
In the Create usNIC Connection Policy dialog box, complete the following fields:
Name
Description
Name field
The name of the policy.
This name can be between 1 and 16 alphanumeric characters. You cannot use spaces
or any special characters other than - (hyphen), _ (underscore), : (colon), and .
(period), and you cannot change this name after the object has been saved.
Description field
A description of the policy. We recommend that you include information about
where and when the policy should be used.
Number of usNICs field
The number of usNICs that you want to create.
Each MPI process running on the server requires a dedicated usNIC. You can create
up to 116 usNICs to sustain 116 MPI processes running simultaneously. Cisco
recommends that you create at least as many usNICs, per usNIC-enabled vNIC, as
there are physical cores on your server. For example, if you have 8 physical cores
on your server, create 8 usNICs.
Adapter Policy drop-down list
The adapter policy that you want to specify for the usNIC. Cisco recommends that
you choose the usNIC adapter policy, which is created by default.
Step 6
In the Navigation pane, click the Servers tab.
Step 7
Step 8
Step 9
On the Servers tab, expand Servers > Policies > root > Adapter Policies.
Click Eth Adapter Policy USNIC.
In the Work pane, click the General tab.
Step 10
(Optional) Modify the details in the Resources and Options sections as needed.
5
Step 11
Step 12
Step 13
Step 14
Step 15
Step 16
For more information about configuring Ethernet adapter policies, see the Cisco UCS Manager Configuration Guide.
On the Servers tab, expand Servers > Service Profiles > Service_Profile_Name.
On the Servers tab, expand Servers > Service Profiles > root.
Expand the service profile node where you want to configure the usNIC and click vNICs.
In the Work pane, click the Network tab.
Step 17
Step 18
Step 19
Step 20
Step 21
In the vNICs area, choose a vNIC and click Modify.
In the Adapter Performance Profile area of the Modify vNIC dialog box, choose Linux from the Adapter Policy
drop-down list.
In the Connection Policies area, click the usNIC radio button.
Choose the usNIC connection policy that you created from the usNIC Connection Policy drop-down list.
Click OK.
Click Save Changes.
In the Navigation pane, click the service profile that you just modified.
Step 22
In the Work pane, click the Policies tab.
Step 23
Step 24
Expand the BIOS Policy bar and choose USNIC in the BIOS Policy drop-down list.
Click Save Changes.
What to Do Next
Enable the Intel IOMMU driver in the Linux kernel (intel_iommu=on).
Creating a usNIC using the Cisco UCS Manager CLI
Before You Begin
You must log in with admin privileges to perform this task.
Procedure
Command or Action
Purpose
Step 1
UCS-A # scope service-profile
serverserver/chassis
Enters the service profile for the specified server and chassis
combination.
Step 2
UCS-A /org/service-profile # show vnic
Displays the vnics that are available on the server. A usNIC vNIC
is available by default when you upgrade to Cisco UCS Manager,
release 2.2.
Step 3
UCS-A /org/service-profile # scope vnic vnic
name
Enters the vNIC mode for the specified vNIC.
Step 4
UCS-A /org/service-profile/vnic # set
adapter-policy Linux
Specifies Linux and the adapter policy for the usNIC.
Step 5
UCS-A /org/service-profile/vnic # enter
Creates the usNIC connection policy reference for the vNIC with
usnic-conn-policy-ref usnic connection policy the specified name. The maximum size for the connection policy
name is 16 characters.
reference name
6
Command or Action
Purpose
Step 6
UCS-A
Commits the transaction to the system configuration.
/org/service-profile/vnic/usnic-conn-policy-ref*
# commit-buffer
Step 7
UCS-A
/org/service-profile/vnic/usnic-conn-policy-ref
# top
Enters the top-level mode.
Step 8
UCS-A # scope org
Enters the root organization mode.
Step 9
UCS-A /org # create usnic-conn-policy usnic
connection policy name
Creates a usNIC connection policy with the specified name.
Step 10
UCS-A /org/usnic-conn-policy* # set
usnic-count number of usnics
Specifies the number of Cisco usNICs to create. Each MPI process
running on the server requires a dedicated usNIC. Therefore, you
might need to create up to 64 usNICs to sustain 64 MPI processes
running simultaneously. Cisco recommends that you create at least
as many usNICs, per usNIC-enabled vNIC, as there are physical
cores on your server. For example, if you have 8 physical cores on
your server, create 8 usNICs.
Step 11
UCS-A /org/usnic-conn-policy* # set
adaptor-profile USNIC
Specifies the usNIC Ethernet adaptor profile for the usNIC
connection policy. This usNIC adaptor profile is created by default
when you upgrade from previous versions of Cisco UCS Manager
to release 2.2.
Step 12
UCS-A /org/usnic-conn-policy* # commit-buffer Commits the transaction to the system configuration.
This example shows how to create a Cisco usNIC and specify its properties:
Server
Server
Server
Server
Server
Server
#
#
#
#
#
#
scope org
create usnic-conn-policy usnic1
set usnic-count-64
set adapter-profile USNIC
commit buffer
top
Server # scope service-profile server 1/1
Server /org/service-profile # show vnic
vNIC:
Name
Fabric ID Dynamic MAC Addr
Virtualization Preference
------------------ --------- ------------------ ------------------------eth0
A
00:25:B5:00:00:8F NONE
eth1
A
00:25:B5:00:00:9F NONE
eth2
A
Derived
NONE
Server /org/service-profile # scope vnic eth0
Server /org/service-profile/vnic # set adapter-policy Linux
Server /org/service-profile/vnic # enter usnic-conn-policy-ref usnic1
Server /org/service-profile/vnic/usnic-conn-policy-ref* # commit-buffer
Server /org/service-profile/vnic/usnic-conn-policy-ref # exit
7
Modifying a usNIC using the Cisco UCS Manager CLI
Before You Begin
You must log in with admin privileges to perform this task.
Procedure
Command or Action
Purpose
Step 1
UCS-A # scope service-profile server chassis /server Enters the service profile for the specified server and chassis
combination.
Step 2
UCS-A /org/service-profile # show vnic
Displays the vnics that are available on the server. A usnic
vnic is available by default when you upgrade to Cisco UCS
Manager, release 2.2.
Step 3
UCS-A /org/service-profile # scope vnic vnic name
Enters the vnic mode for the specified vNIC.
Step 4
UCS-A /org/service-profile/vnic # enter
Specifies the usnic connection policy reference for the vNIC
usnic-conn-policy-refusnic connection policy reference that you want to use.
name
Step 5
UCS-A /org/service-profile/vnic/usnic-conn-policy-ref* Commits the transaction to the system configuration.
# commit-buffer
This example shows how to modify Cisco usNIC properties:
Server # scope service-profile server 1/1
Server /org/service-profile # show vnic
vNIC:
Name
Fabric ID Dynamic MAC Addr
Virtualization Preference
------------------ --------- ------------------ ------------------------eth0
A
00:25:B5:00:00:8F SRIOV USNIC
eth1
A
00:25:B5:00:00:9F NONE
eth2
A
Derived
NONE
Server /org/service-profile # scope vnic eth0
Server /org/service-profile/vnic # set adapter-policy Linux
Server /org/service-profile/vnic # enter usnic-conn-policy-ref usnic2
Server /org/service-profile/vnic/usnic-conn-policy-ref* # commit-buffer
Server /org/usnic-conn-policy # exit
Deleting a usNIC using the Cisco UCS Manager CLI
Before You Begin
You must log in with admin privileges to perform this task.
8
Procedure
Command or Action
Purpose
Step 1
UCS-A # scope service-profile serverserver/chassis
Enters the service profile for the specified server and
chassis combination.
Step 2
UCS-A /org/service-profile # show vnic
Displays the vNICs that are available on the server. A
usNIC vNIC is available by default when you upgrade to
Cisco UCS Manager, release 2.2.
Step 3
UCS-A /org/service-profile # scope vnic vnic name
Enters the vNIC mode for the specified vNIC.
Step 4
UCS-A /org/service-profile/vnic # show
Specifies the usNIC connection policy reference for the
usnic-conn-policy-refusnic connection policy reference vNIC that you want to use.
name
Step 5
UCS-A /org/service-profile/vnic # delete
Deletes the specified usNIC connection policy reference.
usnic-conn-policy-refusnic connection policy reference
name
Step 6
UCS-A /org/service-profile/vnic/usnic-conn-policy-ref* Commits the transaction to the system configuration.
# commit-buffer
This example shows how to modify Cisco usNIC properties:
Server # scope service-profile server 1/1
Server /org/service-profile # show vnic
vNIC:
Name
Fabric ID Dynamic MAC Addr
Virtualization Preference
------------------ --------- ------------------ ------------------------eth0
A
00:25:B5:00:00:8F SRIOV USNIC
eth1
A
00:25:B5:00:00:9F NONE
eth2
A
Derived
NONE
Server /org/service-profile # scope vnic eth0
Server /org/service-profile/vnic # show usnic-conn-policy-ref
USNIC Connection Policy Reference:
USNIC Connection Policy Name
---------------------------usnic0
Server /org/service-profile/vnic # delete usnic-conn-policy-ref usnic0
Server /org/service-profile/vnic* # commit-buffer
Server /org/service-profile/vnic # exit
Installing Linux Drivers for Cisco usNIC
The following section lists the contents of the Cisco usNIC build folder that is included in the UCS Drivers ISO bundle. Documentation
about known issues and installation instructions (README) for Cisco usNIC are also included in the build folder.
Note
To prevent the OS from swapping out the memory that is allocated to the usNIC applications, the installation
software increases the locked memory system setting of the OS to Unlimited.
9
• kmod-usnic_verbs-1.0.2.116-1.x86_64.rpm—Linux kernel verbs driver for the usNIC feature of the Cisco VIC SR-IOV
Ethernet NIC.
• libusnic_usnic_verbs-1.0.1.116-1.x86_64.rpm—User space library libibverbs plugin for usNIC.
• kmod-enic-2.1.1.52-rhel6u4.el6.x86_64.rpm—OOB Linux kernel driver for Enic; must be copied to parent folder (one level
above installation files for Cisco usNIC) for installer script to work properly.
• openmpi-cisco-1.6.5cisco1.0.2.120-1.x86 _64.rpm—Cisco usNIC Open MPI—Open MPI with the Cisco usNIC BTL MPI
transport.
• usnic_tools-1.0.2.120-1.x86_64.rpm—Utility programs for usNIC.
• usnic_installer.sh—Scripts for installing the usNIC packages listed in the section.
• usnic_uninstaller.sh—Script for uninstalling the usNIC packages listed in the section.
Before You Begin
Make sure that you have configured the Cisco usNIC properties in the Cisco UCS Manager (UCSM). For more information about
how to configure the properties, see Configuring Cisco usNIC in Cisco UCS Manager, on page 4.
You must also make sure that the host OS (RHEL 6.4 ) on which you want to install Cisco usNIC has a supported version of the
Cisco Enic driver installed. The Cisco Enic driver is the Linux kernel networking driver for the Cisco VIC SR-IOV Ethernet NIC.
Procedure
Step 1
#./usnic_installer.sh
Execute the installer script from the directory where the installation files are located for Cisco usNIC and then add the
output of the installer script to the parent folder (one level above where installation files are located for Ciusco usNIC.
Note
You require admin privileges to execute the script at the root
(#) prompt.
Step 2
Add the output of the installer script to the parent folder (one level above where installation files are located for Cisco
usNIC.
This is required for a successful installation.
Step 3
# chkconfig rdma on
Enables rdma and, once enabled, it will be started automatically after a system reboot.
Step 4
# service rdma start
Verify that the rdma service is started. This service is required for the usnic_verbs kernel module.
Step 5
Reboot your server for the installation changes to take effect.
If you do not want to reboot your server, you can manually load the kernel modules. For more information about how to
load the modules, see Manually Loading the Kernel Modules for Cisco usNIC, on page 10.
Manually Loading the Kernel Modules for Cisco usNIC
If you do not want to reboot your server, you can manually load the Cisco usNIC kernel modules by using the following steps.
10
Procedure
Step 1
Command or Action
Purpose
# rmmod enic
Unloads the existing Enic driver module.
Note
Make sure that you are not logged into the OS using the network, for
example, via SSH. Otherwise, your network connection might get
permanently disconnected. Alternatively, you can log in to the server
using the UCSM KVM to perform this step.
Step 2
# modprobe enic
Loads the Enic driver module.
Step 3
# modprobe usnic_verbs
Loads the usnic_verbs driver module.
Uninstalling Linux Drivers for Cisco usNIC
Procedure
Step 1
# /opt/cisco/usnic/bin/usnic_uninstaller.sh
Execute the uninstaller script from the bin folder in the directory where the Cisco usNIC installation files are located.
Step 2
Reboot your Cisco UCS server.
You must reboot the server for the changes to take effect.
Verifying the Cisco usNIC Installation
After you install the required Linux drivers for Cisco usNIC, perform the following procedure at the Linux prompt to make sure that
the installation completed successfully.
Procedure
Step 1
Search for and list all kernel modules that have usnic_verbs in their names to verify that the kernel module was loaded
during the OS driver installation.
$ lsmod | grep usnic_verbs
The following details are displayed when you enter the $ lsmod | grep usnic_verbs command. The kernel modules listed
on your console may differ based on the modules that you have currently loaded in your OS.
usnic_verbs
78776 0
ib_core
73747 13
ib_ipoib,rdma_ucm,ib_ucm,ib_uverbs,ib_umad,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad,
11
iw_cxgb4,iw_cxgb3,usnic_verbs
enic
67741 1 usnic_verbs
Step 2
View the configuration of Cisco usNIC-enabled NICs.
$ ibv_devinfo
The following section is a brief example of the results that are displayed when you execute the ibv_devinfo command.
The results may differ based on your current installations. When the results are displayed on your console, make sure that
the state for each of the listed ports are shown as PORT_ACTIVE. The following example shows two ports (usnic_1 and
usnic_0) that are configured on a Cisco UCS VIC adapter adapter. If you configured only one usNIC-enabled vNIC, you
will see a listing for only usnic_0.
Note
The ibv_devinfo command displays the value for the transport parameter as iWARP. However, Cisco usNIC
does not utilize the iWARP transport.
hca_id: usnic_1
transport:
node_guid:
sys_image_guid:
vendor_id:
vendor_part_id:
hw_ver:
board_id:
phys_port_cnt:
port:
1
state:
max_mtu:
active_mtu:
sm_lid:
port_lid:
port_lmc:
link_layer:
hca_id: usnic_0
transport:
node_guid:
sys_image_guid:
vendor_id:
vendor_part_id:
hw_ver:
board_id:
phys_port_cnt:
port:
1
state:
max_mtu:
active_mtu:
sm_lid:
port_lid:
port_lmc:
link_layer:
Step 3
iWARP (1)
0200:00ff:fe00:0000
2657:20ff:fe6f:1100
0x1137
207
0x2
133
1
PORT_ACTIVE (4)
4096 (5)
4096 (5)
0
0
0x01
Ethernet
iWARP (1)
0200:00ff:fe00:0000
2657:20ff:fe6f:1000
0x1137
207
0x2
133
1
PORT_ACTIVE (4)
4096 (5)
4096 (5)
0
0
0x01
Ethernet
Verify that the rdma is on. This service is required for loading the usnic_verbs kernel module.
# chkconfig --level 2345 rdma on
Step 4
Run the usnic_verbs_check script to view the installed RPMs and their versions.
$ /opt/cisco/usnic/bin/usnic_verbs_check
If any errors occurred during the OS driver installation, warnings are generated.
If the usnic_verbs module failed to load, the following brief example shows the warnings that are generated:
$ rmmod usnic_verbs
$ /opt/cisco/usnic/bin/usnic_verbs_check
enic RPM version 2.1.1.52-rhel6u4.el6 installed
usnic_verbs RPM version 1.0.2.116-1 installed
WARNING: usnic_verbs module not loaded
libusnic_verbs RPM version 1.0.1.116-1 installed
Open MPI RPM version 1.6.5cisco1.0.2.120-1 installed
12
WARNING: No usnic verbs devices found
WARNING: No usnic verbs devices found
3 warnings
Step 5
Verify that the Cisco usNIC network packets are being transmitted correctly between the client and server hosts:
a) Determine the name of the Ethernet interface associated with the Cisco usNIC on the server host.
<server>$ cat /sys/class/infiniband/usnic_0/config | grep eth
usnic_0: 0000:07:0.0, eth1, 00:25:b5:c1:b3:10, 32 VFs
b) Determine the IP address for the Ethernet interface.
<server>$ ip addr show dev eth4 | grep "inet[^6]"
inet 10.1.0.1/16 brd 10.1.255.255 scope global eth4
c) Run the ibv_ud_pingpong program on the server host.
<server>$ ibv_ud_pingpong -g 0 -d usnic_0 -s 200
Set the -g flag, which specifies the GID, to 0 for usNIC. For more information about the command line options used
with the ibv_ud_pingpong program, see the ibv_ud_pingpong(1) man page.
d) Execute the ibv_ud_pingpong program on the client host by using the IP address that corresponds to the usNIC on
the server host.
<client>$ ibv_ud_pingpong -g 0 -d usnic_0 -s 200 10.1.0.1
The following example shows the results that are displayed when you run the ibv_ud_pingpong program.
Server-side:
<server>$ ibv_ud_pingpong -g 0 -d usnic_0 -s 200
local address: LID 0x0000, QPN 0x000001, PSN 0x38ffcb: GID fe80::2657:20ff:fe6f:1000
remote address: LID 0x0000, QPN 0x000001, PSN 0x3a9e22, GID fe80::b2fa:ebff:fe72:84fd
400000 bytes in 0.01 seconds = 637.70 Mbit/sec
1000 iters in 0.01 seconds = 5.02 usec/iter
Client-side:
<client>$ ibv_ud_pingpong -g 0 -d usnic_0 -s 200 10.1.0.1
local address: LID 0x0000, QPN 0x000001, PSN 0x3a9e22: GID fe80::b2fa:eb ff:fe72:84fd
remote address: LID 0x0000, QPN 0x000001, PSN 0x38ffcb, GID fe80::2657:20 ff:fe6f:1000
400000 bytes in 0.01 seconds = 617.40 Mbit/sec
1000 iters in 0.01 seconds = 5.18 usec/iter
Step 6
Download, compile, and execute the ring_c test program to validate that the MPI traffic is correctly transmitted between
the client and server hosts.
You can obtain the ring_c test program from this link: http://svn.open-mpi.org/svn/ompi/branches/v1.6/examples/ring_
c.c.
The following example shows how to use the wget utility to obtain, compile, and execute the ring_c. Alternatively, you
can use other methods of obtaining and running the test program.
$ wget http://svn.open-mpi.org/svn/ompi/branches/v1.6/examples/ring_c.c
--2013-08-06 15:27:33-- http://svn.open-mpi.org/svn/ompi/branches/v1.6/examples/ring_c.c
Resolving svn.open-mpi.org... 129.79.13.24 Connecting to
svn.open-mpi.org|129.79.13.24|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2418 (2.4K) [text/plain]
Saving to: “ring_c.c”
100%[======================================>] 2,418
--.-K/s
in 0s
Last-modified header invalid -- time-stamp ignored.
2013-08-06 15:27:33 (10.7 MB/s) - “ring_c.c” saved [2418/2418]
$ mpicc ring_c.c -o ring_c
[no output]
13
$ mpiexec
Process 0
Process 0
Process 0
Process 0
Process 0
Process 0
Process 0
Process 0
Process 0
Process 0
Process 0
Process 0
Process 0
Process 2
Process 1
Process 3
--host host1,host2 -n 4 ./ring_c
sending 10 to 1, tag 201 (4 processes in ring)
sent to 1
decremented value: 9
decremented value: 8
decremented value: 7
decremented value: 6
decremented value: 5
decremented value: 4
decremented value: 3
decremented value: 2
decremented value: 1
decremented value: 0
exiting
exiting
exiting
exiting ...
If the ibv_ud_pingpong program and the ring_c program executed successfully, you should now be able to run MPI applications
over usNIC.
14
© 2013
Cisco Systems, Inc. All rights reserved.
Americas Headquarters
Cisco Systems, Inc.
San Jose, CA 95134-1706
USA
Asia Pacific Headquarters
Cisco Systems (USA) Pte. Ltd.
Singapore
Europe Headquarters
Cisco Systems International BV
Amsterdam, The Netherlands
Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the
Cisco Website at www.cisco.com/go/offices.

Download Report