4X = 4 * Infiniband Link Rate 2.5 Gbits/sec, which is
Equal to (2.5*4)10Gbits/sec, Full Duplex.
SDR= Single Data Rate - 10Gbits/sec
DDR = Double Data Rate - 20Gbits/sec
QDR = Quad Data Rate- 40Gbits/sec
FDR = Fourteen Data Rate - 56Gbits/sec
Check the Infiniband OFED Version
# ofed_info | head -n 1
MLNX_OFED_LINUX-1.4 (OFED-1.4-mlnx8)
InfiniBand Self Test
# /usr/bin/hca_self_test.ofed
---- Performing Adapter Device Self Test ----
Number of CAs Detected ................. 1
PCI Device Check ....................... PASS
Kernel Arch ............................ x86_64
Host Driver Version .................... MLNX_OFED_LINUX-1.5.3-1.0.0 (OFED-1.5.3-1.0.0): 1.5.3-2.6.18_194.el5
Host Driver RPM Check .................. PASS
Firmware on CA #0 HCA .................. v1.2.0
Firmware Check on CA #0 (HCA) .......... NA
REASON: NO required fw version
Host Driver Initialization ............. PASS
Number of CA Ports Active .............. 1
Port State of Port # on CA #0 (HCA)..... UP 4X (IB)
Error Counter Check on CA #0 (HCA)...... PASS
Kernel Syslog Check .................... PASSib_rdma_lat
Node GUID on CA #0 (HCA) ............... 00PORT_ACTIVE:02:c9:02:00:2a:78:b8
------------------ DONE ---------------------
Find the Infiniband Speed Status of the host Topology
# /usr/sbin/iblinkinfo.pl
Switch 0x000b8cffff006c98 MT47396 Infiniscale-III Mellanox Technologies:
21 1[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 22 1[ ] "node1 HCA-1" ( )
21 2[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 2 1[ ] "node2 HCA-1" ( )
21 3[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 1 1[ ] "node3 HCA-1" ( )
21 4[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 3 1[ ] "node4 HCA-1" ( )
21 5[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 4 1[ ] "node5 HCA-1" ( )
21 6[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 5 1[ ] "node6 HCA-1" ( )
Fine the Infniband host Topology
# ibhosts
Ca : 0x0002c902002a78d0 ports 1 "ib-node1 HCA-1"
Ca : 0x0002c902002a78d4 ports 1 "ib-node3 HCA-1"
Ca : 0x0002c902002a78bc ports 1 "ib-node7 HCA-1"
Ca : 0x0002c902002a79a0 ports 1 "ib-node5 HCA-1"
Inifniband Hardware Swith Details.
# ibswitches
Switch : 0x000b8cffff0071cc ports 24 "MT47396 Infiniscale-III Mellanox Technologies" base port 0 lid 9 lmc 0
Infiniband kernel can Configuration Information.
# /etc/infiniband/info
prefix=/usr
Kernel=2.6.18-194.el5
Configure options: --with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mthca-mod --with-mlx4-mod --with-mlx4_en-mod --with-mlx4_ib-mod --with-mlx4_vnic-mod --with-mlx4_fc-mod --with-cxgb3-mod --with-nes-mod --with-qib-mod --with-ipoib-mod --with-sdp-mod --with-srp-mod --with-rds-mod --with-iser-mod
# ibv_devinfo
hca_id: mthca0
transport: InfiniBand (0)
fw_ver: 1.2.0
node_guid: 0002:c902:002a:78b8
sys_image_guid: 0002:c902:002a:78bb
vendor_id: 0x02c9
vendor_part_id: 25204
hw_ver: 0xA0
board_id: MT_03F0110001
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 2048 (4)
active_mtu: 2048 (4)
sm_lid: 1
port_lid: 5
port_lmc: 0x00
link_layer: IB
Status of the Infiniband
# ibstatus
Infiniband device 'mthca0' port 1 status:
default gid: fe80:0000:0000:0000:0002:c902:002a:78b9
base lid: 0x5
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 10 Gb/sec (4X)
link_layer: IB
Infiniband services
openibd - Load the Kernel Module(CLIEN System)
opensmd - Infiniband Subnet Manager -Read /etc/opensm/opensm.conf (MASTER System)
0 Comments