8 more bytes!

I was recently fortunate enough to move into a place where I could get VDSL (BT’s FTTC product, resold through AAISP). This is a vast improvement over conventional ADSL in that more of the path between you and the Interwebs is fibre rather than copper.

The service is provided with a modem, which you use PPPoE to connect via to your ISP. This supports something called “baby jumbo frames” (RFC4638) which boosts the MTU for your PPP connection from the usual 1492 bytes to 1500 bytes. This means it can carry full-sized 1500 byte IP packets. This also means that the interface hosting the PPP connection needs to support an MTU of 1508 bytes (since PPP has an overhead of 8 bytes).

This is a good thing, and especially so in the IPv6 world where routers don’t fragment packets and you have to rely on ICMPv6 to negotiate end-to-end MTU (in a world where it’s not unknown of for people to disable this mechanism…)

I currently use an Alix 2d3 board running OpenBSD for my router. This is a great little machine with two 10/100 ethernet interfaces (vr(4)). Can easily cope with the 40Mb/10Mb FTTC service. I figured I’d try and get RFC4638 working on my connection.

To do this I needed to take care of two things.

1) Get RFC4638 support in pppoe(4)
2) Get the vr(4) interfaces to support 1508 byte MTU

#1 is easy, simply upgrade to OpenBSD 5.2

I did this by installing OpenBSD 5.2 onto a virtual machine, configuring it correctly and then copying the disk onto the CF card which the router uses as its’ “hard disk”. The latter involves formatting the CF card (fdisk -i, disklabel -e, newfs -O2 (for each disk)) and then using dump/restore to copy the filesystems from the virtual machine to the CF card. Finally following some of the instructions from the “Restoring from tape” section of the manual here to install the boot block.

#2 is a little less straightforward. The vr(4) driver does not support jumbo frames, the maximum MTU is 1500 bytes. We need 8 more bytes out of it. From reading around on the subject it looked like the NIC can cope with larger packets to do VLAN-related things (VLAN long frames support (1518+4bytes)).

So since I don’t care about VLAN support, why not try hacking the driver to see if I can simply boost the MTU and hope it works?

Well the changes are fairly easy to make, so I gave it a go and it appears to work perfectly.

Files to change are:

/usr/src/sys/dev/pci/if_vr.c
/usr/src/sys/dev/pci/if_vrreg.h

(Assuming you’ve got the kernel source in /usr/src/sys/)

First file you need to find the function vr_attach() and add:

ifp->if_hardmtu = 1508;

and comment out:

// ifp->if_capabilities |= IFCAP_VLAN_MTU;

(Just to hammer home the point that this is a hack and thus you probably shouldn’t try to use VLANs!)

Second file you need to change (this number may be a little high, leaving it at the previous default got me up to 1506 MTU fine but was unstable beyond that. Feel free to experiment with lower numbers!):

#define VR_RXLEN  1548

And that’s it! Compile a new kernel with these changes and boot from it. You’ll be able to set the MTU for your vr(4) connections to 1508 (I’ve actually tested this with higher MTUs, but since I have no need for that I figure 1508 is a sensible limit – there’s really little use for anything higher) and thus your pppoe(4) connection to 1500.

I’ve tested this fairly simply using “ping -Ds 1472 somehost.ontheinternet” (-D sets the don’t fragment bit, and 1472 is the highest payload you can cram into a single ICMP packet with a 1500 byte MTU maximum (1472 payload + 8 bytes ICMP header + 20 bytes IP header = 1500 bytes)). Examining the tcpdump capture (tcpdump -pi pppoe0 -w output.pcap) using Wireshark shows 1508 byte PPP frames containing 1500 byte IPv4 packets.

Further testing and stability analysis will come from using it. So far so good though.

So for anyone using OpenBSD on an Alix board with the VT6105M chip to connect to a BT FTTC service you can, fairly easily, have full 1500 byte MTU on your connection.