The Problem
As much as events like World IPv6 day have tried to promote IPv6 and push its adoption, there remain many sites and Internet Service Providers that either don’t support IPv6 or don’t do it well. Luckily, my ISP (Ebox) does it fairly well, but it has not been without its issues. There have been two I have encountered:
- My pfSense router not being able to ping IPv6 sites from the WAN
- Slow or not loading at all websites
The first turned out to be an issue with my ISP, where a quick forum post directed at their technical staff had the issue sorted within about two hours. What would happen is that upon a PPPoE connection (or reconnection), pfSense would be assigned a /64 address on the WAN along with the /56 to delegate to LAN networks. Sometimes, however, traffic sent out – such as pings – from this /64 would never make it past a first hop according to a traceroute, and I would never get a response. A reconnect (or five) would eventually fix this. They never told me what the issue was, but I suspect it involved routing. The internet connection would still function fine, as DNS queries from the pfSense resolver could be done via IPv4 and IPv6 traffic from LAN clients went through just fine. It may have affected some ICMP traffic though, such as IPv6 traceroutes, but I never checked.
The second issue was more irritating, and only occurred with certain websites such as Wikipedia, the Arch Linux wiki, or the pfSense forums, which would load slowly or sometimes timeout, whereupon a page refresh would work fine. Other sites such as Yahoo would not load at all, ever. I was quite confused, as by default pfSense allows essential ICMPv6 traffic, and I was able to pass the ICMP blackhole tests designed to detect this.
As far as I know, the issue occurs only with websites where PMTUD (described well here) is required. Since PPPoE is a (small) tunnel protocol, PMTUD will need to be done on every connection. It can go two ways:
- It works, but adds some delay, as both sides of the connection have to figure out what packet size to use. This is normal, and I assume wasn’t a serious consideration in the 90’s when IPv6 was designed, but as we expect websites to load very fast, this can be irritating.
- It doesn’t work at all when something in the path of the connection breaks it, such as whatever Yahoo is doing.
I noticed that I never had any problem reaching sites such as Facebook or Google, which as far as I know use tunnels internally for load balancing and thus have an even smaller maximum packet size than my PPPoE connection. Thus, they never send packets that are too big for my connection, which avoid any PMTUD issues altogether.
Solution
Use MSS clamping on the pfSense router. That’s it. I believe it is supposed to do this automatically for PPPoE connections, but whatever it is doing by default, it is not working. I find it surprising that in 2021 we still need to manually configure such things, which explains why no one is in a rush to use IPv6.
What MSS on pfSense does is modify TCP packets when a connection is established to have the correct maximum size, thus ensuring that when the connection is established, both client and server don’t use sizes too big for the connection between them. Thus, PMTUD is never needed.
Implementing this solution made me realize there was another issue: pfSense assumes IPv4 when calculating MSS in the GUI:
If a value is entered in this field, then MSS clamping for TCP connections to the value entered above minus 40 (TCP/IP header size) will be in effect.
Thus, in order to set the MSS correctly for IPv6, I needed to set it to 1472 instead of 1492. This had the side effect of making the IPv4 MSS smaller than necessary by 20 bytes. It’s not a serious issue, but a bit irritating if you know about it.
Luckily, my post on the forums got some traction and Viktor, a pfSense dev, wrote a patch to fix the issue. I can now use 1492 as MSS and not worry about it!
The patch can be downloaded from the thread on the forums. Thanks Viktor!
I assume at some point it will be merged into pfSense proper, but no idea when.
I believe the patch should be in pfSense 2.6 per the bug report.
Hello Michael,
Encountert your blog and tried your recommendation.
Sadly this doesn’t solve my pfSense (2.6.0) routing issue.
What I’m trying to figure out is why it all worked perfectly before my ISP changed its IPv6 configuration.
I do get a DHCP config (from the ISP’s Fritz!Box 7590 modem) on my WAN interface and the other NICs get an IPv6 /64 subnet (tracked from WAN) as before.
But trying to ping an external host from other than the WAN interface doesn’t work anymore.
I’d hoped your MSS option would work, but it looks like I have to contact my ISP.
Or do you have some other tricks to debug the system?
I did have a similar problem actually at one point, although I believe it was the other way around – hosts on the /64 subnets could ping, but not Pfsense from the WAN. The problem ended up being on the ISP side and they resolved it once I alerted them. No idea what the problem actually was on their end, and there really isn’t much debugging you can do other than a packet capture on both LAN and WAN to see if the packets are going out.
Pfsense has resolved the bug report I submitted a while back and at least for my configuration there aren’t any Pfsense bugs that affect it with 2.6.0.
yeah I have the smilier issue with the latest 23.01. No idea how to debug it and proof that something is wrong with ISP, but funny enough yahoo indeed doesn’t work when ipv6 is enabled. and the only features I use from yahoo is news and finance.