I want to set up some point to point links using DPDK to give the advantages of kernel bypass and had lots of “fun” setting things up for this. Probably it’s because I am incapable of following instructions properly that I need to document the process properly so I don’t waste so much time next time. One major issue I discovered (at the time of writing) is that Ubuntu 16.04 isn’t yet supported – 15.10 is the highest version that seems to be supported by the Mellanox OFED. It might be possible to get it working with 16.04 but it seemed easier just to go back to 15.10. Instructions after the jump…
First thing is to install the Mellanox OFED from here. The simplest thing is to select the .iso file for download once the appropriate OS and version has been selected. The user manual explains how to install everything from the .iso but I have repeated it here just to save time. The .iso just needs to be mounted using:
cd ~/Downloads sudo mount -o ro,loop MLNX_OFED_LINUX-3.2-220.127.116.11-ubuntu15.10-x86_64.iso /mnt
This is for the current version – change as required. Then run the install:
sudo /mnt/mlnxofedinstall sudo umount /mnt
That does pretty much everything to install the card drivers.
Next step is to set up huge pages. I just used 2M pages but 1G is also an option for 64 bit processors. Full instructions are here. Edit /etc/default/grub so that the kernel config line looks like:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash hugepages=1024"
This will generate 1024 2M huge pages. To use this configuration, run:
To make the memory available for DPDK, it must be mounted. Enter:
sudo mkdir /mnt/huge
Then edit /etc/fstab and add the line:
nodev /mnt/huge hugetlbfs defaults 0 0
Restart the machine to kick in the new configuration.
DPDK can be downloaded from here. The download can be expanded to somewhere convenient. Before compilation, support for the Mellanox ConnectX-3 Pro cards must be enabled. Full details are here but, to get started, edit <dpdkdir>/config/common_base so that the relevant line looks like this:
Then DPDK can be compiled with:
cd <dpdkdir> make -j8 install T=x86_64-native-linuxapp-gcc
The “-j8” part should be changed to reflect how many cores are to be used for compilation. I get an error complaining about the lack of DESTDIR but it doesn’t seem to matter. A parameter DESTDIR=<something> can be added if required.
To test the build, one or more of the examples can be tried. I used l2fwd which can be found at <dpdkdir>/examples/l2fwd. To try this out, enter:
cd <dpdkdir>/examples/l2fwd export RTE_SDK=<dpdkdir> make cd build/app sudo ./l2fwd -c f -- -p f
This should find the card and start forwarding frames between the ports on the card (does rather assume a dual port card!). My card was connected to another card that was sending pings and I could see these being received and sent out of the other port.