10gbe Cc2 N320e Sr

I finally connected my NAS with my build server with 10 Gigabit Ethernet.

The network card I used is the: Chelsio CC2-N320E-SR. You get this cards around 30$ from ebay and similar places. Plus the direct attach cable needed, which costs around 8-20$, we are looking at a total of less than 100$ to connect 2 computers with 10GbE. Which is very impressive to me.

The coolest part about this cards is that they just work out of the box with FreeBSD. This was harder than expected. Since I put the first card in slot 1 of my Dell T20 and it never showed up. According to the documentation the 1 slot is: One full-height, half-length x16 PCIe Gen3 card slot connected to processor. It is unclear to me why the card never showed up, not in the BIOS, not with pciconf -lv. So I moved it to slot 4: One full-height, half-length x16 (x4) PCIe Gen2 card slot connected to PCH. And it showed up.

When the card is recognized the drive is loaded automatically. And you can check dmesg if the card needs a firmware upgrade.

kernel: cxgbc0: using MSI-X interrupts (9 vectors)
kernel: found old FW minor version(5.0), driver compiled for version 7.11
kernel: cxgbc0: firmware needs to be updated to version 7.11.0

The firmware binary: t3fw-7.11.0.bin is found on https://service.chelsio.com/legacy.html. This can then be unpacked and flashed with the help of cxgbtool. (Which is shipped with FreeBSD but needs to be compiled)

cd /usr/src/tools/tools/cxgbtool
make install
cxgbtool cxgb0 loadfw t3fw-7.11.0.bin

And that's it, reboot and check dmesg again. There is a great blog post by Boris Tassou which explains many of these steps in detail if you are interested.

Now to the big question how fast is it? According to my very primitive iperf testing:

[ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 7.82 GBytes 6.71 Gbits/sec