OFED Driver 설치

※ IB 커널 모듈과 Lustre 커널 모듈의 Symbol이 틀려 정상적으로 작동되지 않음

※ 다운로드 된 패키지는 RHEL 6.3(CentOS 6.3) 커널에 맞춰져 있으므로 리빌드 해야함.

※ 다운로드 : http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers

1. 1.5.3-3.1.0

# cd /usr/local/src

# wget http://www.mellanox.com/downloads/ofed/MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.3-x86_64.iso
# wget http://www.mellanox.com/downloads/ofed/MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.3-x86_64.tgz

# tar zxvfp MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.3-x86_64.tgz

# cd MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.3-x86_64

### 기존에 설치된 IB 패키지 삭제

# yum -y remove openmpi-1.5.4-1.el6.x86_64 rocks-openmpi-1.6.2-1.x86_64 rocks-openmpi-modules-1.6.2-1.x86_64 openmpi-devel-1.5.4-1.el6.x86_64

# ./uninstall.sh

This program will uninstall all OFED packages on your machine.

Do you want to continue?[y/N]:y

Removing OFED Software installations

Running /bin/rpm -e --allmatches libibverbs libibverbs-devel libmthca libmlx4 libibumad libibumad-devel libibmad libibmad-devel librdmacm opensm opensm-libs compat-dapl compat-dapl-devel dapl dapl-devel dapl-utils ibutils ibutils-libs infiniband-diags infinipath-psm opensm

Stopping IB Subnet Manager(s)..[  OK  ]

### Infiniband 패키지 리빌드 : 정상적으로 빌드가 완료 되면 /tmp 디렉토리에 MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.3-x86_64.iso가 생성된다.

# ./mlnx_add_kernel_support.sh -i /usr/local/src/MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.3-x86_64.iso 

Note: This program will create MLNX_OFED_LINUX ISO for rhel6.3 under /tmp directory.

      All Mellanox, OEM, OFED, or Distribution IB packages will be removed.

Do you want to continue?[y/N]:y

See log file /tmp/mlnx_ofed_iso.8069.log

Building OFED RPMs. Please wait...

Removing OFED RPMs...

Running mkisofs...

Created /tmp/MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.3-x86_64.iso

2. 1.5.3-4..42

# cd /usr/local/src

# tar zxvfp MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.5-x86_64.tgz

# cd MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.5-x86_64

# ./mlnx_add_kernel_support.sh -m /usr/local/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.5-x86_64 --make-iso

### Infiniband 패키지 설치

# rpm -e --allmatches  fcoe-utils anaconda-13.21.176-1.el6.x86_64 anaconda rdma-3.3-4.el6_3.noarch rdma

# yum install tk

# mount -t iso9660 -o loop /tmp/MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.3-x86_64.iso  /mnt

# cd /mnt

# ./mlnxofedinstall --all --without-depcheck

This program will install the MLNX_OFED_LINUX package on your machine.

Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be removed.

Do you want to continue?[y/N]:y

# cd;umount /mnt

# sync;shutdown -r now

### IP Address 설정

# vi /etc/sysconfig/network-scripts/ifcfg-ib0

DEVICE=ib0

IPADDR=10.6.1.8

NETMASK=255.255.0.0

ONBOOT=yes

MTU=65520

# service network restart

※ 메모리가 16GB 이하인 경우

# cat /etc/modprobe.d/mlx4_en.conf

options mlx4_core pfctx=0 pfcrx=0

options mlx4_core log_num_mtt=20 log_mtts_per_seg=3

# //etc/init.d/openibd restart