TRB160 IPv6 not routed to internet from ethernet

I recently received a TRB160. I have inserted a SIM card from a service provider that gives me both IPv4 and IPv6 address. I connect the ethernet port of TRB160 directly to the ethernet port of my laptop (running Fedora with NetworkManager).
The TRB160 connects to a 4G network, receives both IPv4 address and IPv6 address, “Internet Status” in the WebUI shows green Online for both “IPv4 status” and “IPv6” status. When I SSH into the router, I can ping both IPv4 hosts and IPv6 hosts on the internet (google.com) using ping and ping6.
The ethernet interface on the laptop also receives both IPv4 and IPv6 addresses from TRB160. I am currently using TRB160 in NAT mode, but for IPv6 it seems like it does not matter if I use NAT, passthrough or bridge mode.

Despite all these positive signs, I can not reach IPv6 hosts on the internet from the laptop. It seems to resolve IPv6 domains to addresses (or perhaps they are cached from before?), but I can not connect to any IPv6 hosts. Any kind of IPv6 connection just times out. There are no error messages like “unreachable”, etc. The connections are just left hanging (or times out).

Connecting to IPv4 hosts on the internet from the laptop works flawlessly. Both ping, curl and web browsing works well as long as access IPv4 hosts. So it is not a IPv4 problem, only a IPv6 problem. See for instance the following curl output from IPv4:

$ curl -4 api4.ipify.org -v
* Host api4.ipify.org:80 was resolved.
* IPv6: (none)
* IPv4: 104.26.12.205, 104.26.13.205, 172.67.74.152
*   Trying 104.26.12.205:80...
* Connected to api4.ipify.org (104.26.12.205) port 80
> GET / HTTP/1.1
> Host: api4.ipify.org
> User-Agent: curl/8.6.0
> Accept: */*
> 
< HTTP/1.1 200 OK
...

I have poked around in both TRB160 and NetworkManager for countless hours now without getting any closer to a solution. I have tried various WAN modes, various RA/DHCPv6 modes, allowed all traffic through the firewall, and factory reset TRB160 a few times. So now I do not know how to continue the troubleshooting.

The laptop ethernet interface receives the following IPs from TRB160:

$ nmcli
enp1s0f0: connected to Wired connection 2
        "Realtek RTL8111/8168/8211/8411"
        ethernet (r8169), 9C:2D:CD:1A:E2:5D, hw, mtu 1500
        ip4 default, ip6 default
        inet4 192.168.2.231/24
        route4 192.168.2.0/24 metric 100
        route4 default via 192.168.2.1 metric 100
        inet6 2a02:2121:2c9:354f:a349:a19c:4fe6:2e02/64
        inet6 fdf4:3a37:d32:0:8ddf:15aa:f163:41f1/64
        inet6 fe80::3d51:cda6:1498:bf20/64
        route6 fe80::/64 metric 1024
        route6 fdf4:3a37:d32::/64 metric 100
        route6 2a02:2121:2c9:354f::/64 metric 100
        route6 fdf4:3a37:d32::/48 via fe80::2297:27ff:fe0b:7d2 metric 100
        route6 default via fe80::2297:27ff:fe0b:7d2 metric 20100

I can successfully ping the router and SSH to the router from the laptop through the following IPv6 addresses:

$ ping6 fe80::2297:27ff:fe0b:7d2%enp1s0f0
$ ssh root@fe80::2297:27ff:fe0b:7d2%enp1s0f0
$ ping6 2a02:2121:2c9:354f::1
$ ssh root@2a02:2121:2c9:354f::1
$ ssh root@fdf4:3a37:d32::1

I have also looked at ifstatus inside TRB160. I see that mob1s1a1_6 receives the following addresses:

  • IPv6 address: 2a02:2121:2c9:354f:a040:c9e7:b48f:96ae
  • IPv6 prefix: 2a02:2121:2c9:354f::/64
  • It has assigned the same IPv6 prefix to lan: 2a02:2121:2c9:354f::/64

The lan interface receives the following addresses:

  • IPv6 prefix assignment: 2a02:2121:2c9:354f::/64, with local address: 2a02:2121:2c9:354f::1/64.
  • IPv6 prefix assignment: fdf4:3a37:d32::/64, with local address: fdf4:3a37:d32::1/60.

Running a simple curl using IPv6 from the laptop currently looks like this (killed by me after 1 minute):

$ curl -6 api6.ipify.org -v
* Host api6.ipify.org:80 was resolved.
* IPv6: 2607:f2d8:1:3c::4
* IPv4: (none)
*   Trying [2607:f2d8:1:3c::4]:80...
^C

$ curl -6 google.com -v
* Host google.com:80 was resolved.
* IPv6: 2a00:1450:400f:805::200e
* IPv4: (none)
*   Trying [2a00:1450:400f:805::200e]:80...
^C

I have also tried two different APNs on the SIM. One APN that gives me a public IPv4 address (FWA) and another APN that gives med a NATed IPv4 address (for mobile). Both APNs give /64 IPv6 prefixes to TRB160.

TRB160 runs firmware TRB16_R_00.07.08.1.

Does anyone have any tips on how I can continue this troubleshooting? I am completely stuck now, and I need IPv6 to work for the use I intend for it.

It seems like your provider issues you “only” a /64 IPv6 network which means that you cannot “route” the traffic by another hop. They probably assume your device is a phone that uses the addresses by itself.
To have an internal network (ethernet) routed to the IPv6 internet you would require another /64 to be assigned to that interface.
The ISP might assign that when you use a DHCPv6 prefix request, but I do not have experience with that with Teltonika (my ISP Odido lives in the middle ages and does not support IPv6).
It could also be solved by using “bridging” to use that same /64 both on the mobile and ethernet side, but is seems you already tried that.
Another possibility would be to use NAT (urghh) and it looks like the addresses starting with fdf4 are provided to do that.
You could try to force that address using curl --interface fdf4:3a37:d32:0:8ddf:15aa:f163:41f1 and see if that works, and if so try to configure the laptop so that it does not get the public address.

Thank you for the suggestions!

To have an internal network (ethernet) routed to the IPv6 internet you would require another /64 to be assigned to that interface.

It could also be solved by using “bridging” to use that same /64 both on the mobile and ethernet side, but is seems you already tried that.

I made another attempt on changing the WAN from NAT mode to bridge mode using the steps provided here. Bridge mode works fine for IPv4. The laptop is assigned the public IPv4 WAN address through DHCP. For IPv6, it looks like the assigned addresses are pretty much identical in NAT mode and bridge mode. In NAT mode, the laptop is assigned the following addresses:

  • 192.168.2.231/24
  • fe80::3d51:cda6:1498:bf20/64
  • fd1f:c0a:723e::689/128
  • fd1f:c0a:723e:0:97eb:4885:43cf:ae6e/64
  • 2a02:2121:2ca:9e3d::689/128
  • 2a02:2121:2ca:9e3d:dcf1:31ed:acd4:3bf7/64

After I change it to bridge mode and reboot TRB160, the laptop is assigned the following addresses (the IPv6 prefix changes due to the reboot):

  • 78.x.y.35/29
  • fe80::3d51:cda6:1498:bf20/64
  • fd1f:c0a:723e::689/128
  • fd1f:c0a:723e:0:97eb:4885:43cf:ae6e/64
  • 2a02:2121:c1:7b29::689/128
  • 2a02:2121:c1:7b29:e6a8:a2c2:8fee:534c/64

Are these assigned IPv6 addresses expected in bridge mode?
From the WebUI I see that TRB160 still assigns an IPv6 address to itself when I switch to bridge mode. And the “IPv6 status” on the dashboard is still green Online. While for IPv4 the dashboard “IPv4 status” becomes red, which might indicate that it fully bridges the IPv4 interface to the laptop.
So I am not completely convinced that the IPv6 interface is properly bridged. Could that perhaps be a bug in the TRB firmware?

The ISP might assign that when you use a DHCPv6 prefix request, but I do not have experience with that with Teltonika (my ISP Odido lives in the middle ages and does not support IPv6).

What is the difference between a “DHCPv6 prefix request” and the regular prefix delegation that TRB160 already receives? Do you have any other pointers on how to set that up? I have fiddled a bit with the DHCP server settings on TRB160, but I suspect that might be the wrong end for such a request?

Another possibility would be to use NAT (urghh) and it looks like the addresses starting with fdf4 are provided to do that. You could try to force that address using curl --interface fdf4:3a37:d32:0:8ddf:15aa:f163:41f1 and see if that works, and if so try to configure the laptop so that it does not get the public address.

Forcing curl to use the ULA interface does not seem to work any better. The result is the same in both NAT and bridge mode. The connection just hangs.

The difference is that it requests an ADDITIONAL prefix, that will start with 2a02:2121: but will be different from the one the TRB160 currently uses, and that you can then assign that to the LAN interface.
Sorry, I do not have experience with that on Teltonika, I use MikroTik routers and have the Teltonika (in my case a TRB500) only as a dumb 5G bridge in front of it. And while I use DHCPv6 prefix requests in other places, I cannot use it on the installation with the TRB500 because the ISP does not do IPv6.
I only want to explain to you that what you want to do isn’t really possible unless you have extra configuration, cooperation from the ISP, or tricks like NAT.

I managed to get one step closer to the desired behaviour. I decided to get my hands dirty and trawl around the OpenWRT documentation and forums. I am not very well versed on IPv6, and even less so on OpenWRT. So I had to learn quite a bit along the way. The biggest inspiration came from the following forum post: My clients can't access IPv6 hosts - #2 by mk24 - Network and Wireless Configuration - OpenWrt Forum
The idea is to avoid assigning a GUA and the IPv6 prefix to the LAN interface. Then we do not “waste” the IPv6 prefix there. And then we make sure that the WWAN interface and all hosts connecting to LAN gets IPv6 directly from ISP through DHCPv6.
So I stopped using the Teltonika WebUI for configuration and dove directly into the UCI files through SSH. The /etc/config/network file now looks like this:

....

config device 'br_lan'
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0'
        list ports 'usb0'

config interface 'lan'
        option device 'br-lan'
        option area_type 'lan'
        option proto 'static'
        option ipaddr '192.168.2.1'
        option netmask '255.255.255.0'
        option ipv6 '1'           # ADDED
        #option ip6assign '64'    # REMOVED

config interface 'mob1s1a1'
        option proto 'wwan'
        option modem '3-1'
        option metric '1'
        option sim '1'
        option dhcpv6 '1'         # CHANGED from '0' to '1'.
        option ipv6 '1'           # ADDED.
        option pdptype 'ipv4v6'
        option method 'nat'
        option area_type 'wan'
        option pdp '1'
        option apn XXX
        option auto_apn '0'
        option auth 'pap'
        option pref_apn '445'
        option reqaddress 'none'   # ADDED
        option reqprefix 'no'      # ADDED

....

The result of this is that the lan interface does not get a IPv6 GUA anymore, only link local address. Further, ifstatus mob1s1a1_6 does no longer report that the IPv6 prefix is assigned to the lan interface.
The second part is to make sure that DHCPv6 requests are relayed properly. The /etc/config/dhcp file now looks like this:

config dsnmasq
        ...

config dhcp lan
        option interface        lan
        option start            100
        option limit            150
        option leasetime        12h
        option dhcpv6           server
        option ra               'relay'         # Changed from 'server' to 'relay'
        option ndp              'relay'         # Added
        option ra_slaac         1
        list ra_flags           managed-config
        list ra_flags           other-config

# ADDED this entire section.
config dhcp mob1s1a1_6
        option interface        'mob1s1a1_6'
        option ra               'relay'         # Must relay requests all the way from LAN to ISP.
        option ndp              'relay'         # Must relay requests all the way from LAN to ISP.
        option master           '1'

After this, I end up with a IPv6 routing table that looks approximately like this:

default from 2a02:2121:c1:7c32::/64 via fe80::655e:4767:1a0d:8dc1 dev rmnet_data0 proto static metric 1 pref medium
2a02:2121:c1:7c32::/64 dev rmnet_data0 proto kernel metric 256 pref medium
fe80::/64 dev br-lan proto kernel metric 256 pref medium
fe80::/64 dev rmnet_ipa0 proto kernel metric 256 pref medium
fe80::/64 dev rmnet_data0 proto kernel metric 256 pref medium

There is one issue in that routing table. Packets with destination 2a02:2121:c1:7c32::/64 gets routed to the WWAN interface (rmnet_data0). I think this must wrong, since packages that are meant for my laptop will be sent to the internet instead. So I have manually added the following route:

ip -6 route add to 2a02:2121:c1:7c32::/64 metric 100 dev br-lan

I then end up with the following routing table:

default from 2a02:2121:c1:7c32::/64 via fe80::655e:4767:1a0d:8dc1 dev rmnet_data0 proto static metric 1 pref medium
2a02:2121:c1:7c32::/64 dev br-lan metric 100 pref medium         <- ADDED!
2a02:2121:c1:7c32::/64 dev rmnet_data0 proto kernel metric 256 pref medium
fe80::/64 dev br-lan proto kernel metric 256 pref medium
fe80::/64 dev rmnet_ipa0 proto kernel metric 256 pref medium
fe80::/64 dev rmnet_data0 proto kernel metric 256 pref medium

This finally allows me to access IPv6 hosts from the laptop. :slight_smile:
I can also access IPv6 hosts directly from the TRB160.
The entire chain now looks approximately like this (after reboot, so the prefix changed again):

  • TRB160 WWAN (rmnet_data0):
    • fe80::106a:9cff:fe7c:33e2/64
    • 2a02:2121:ca:c145:106a:XXXX:XXXX:33e2/64
  • TRB160 LAN (lan):
    • fe80::2297:27ff:fe0b:7d2/64
  • Laptop LAN:
    • fe80::3d51:cda6:1498:bf20/64
    • 2a02:2121:ca:c145:d695:XXXX:XXXX:b438/64

Great success! :smiley:

This is currently using NAT mode. So I will try to get it working in bridge mode as well.
And since the current solution requires a manual route setup, I need to figure out how to do automatically set that one correctly. When the prefix change, then the route must also change. I guess I need to figure out how OpenWRT prefers routing to be configured. Perhaps I also need to script something.

Additionally, I also stumbled upon the extendprefix option in the IPv6/DHCPv6 configuration page. The following description sounds like it suits my use case perfectly:

On a 3GPP Mobile WAN link, accept a /64 prefix via SLAAC and extend it on one downstream interface - see RFC 7278

So I still have a few iterations left to try out. But at least I got something working.

Ok great!
At least now you can clearly see and understand what the problem is, so that is a good step towards a solution.

I can confirm that it still works in bridge mode. Again, it does not seem like bridge mode has any kind of effect on the IPv6 interfaces and setup. I guess that might be expected, since it has never performed NAT on the IPv6 interface.

Further, I also took a step back to see if could work out a minimum required configuration. With a completely fresh configuration, I started changing the settings that I believe that is most important. I managed to make it work again with a much smaller configuration this time. It turns out that a bunch of settings in /etc/config/network from my previous post were not necessary. The learning from that process is the following:

  • Is is completely fine to use network.lan.ip6assign='64'. We do NOT need to block the lan interface from getting an IPv6 address. By keeping this setting, I also get the necessary route to br-lan for packages in the assigned IPv6 prefix. So I do not need to manually add that route anymore.
  • The network.mob1s1a1 settings reqprefix and reqaddress were not needed either.

The root cause for why the laptop could not reach any internet hosts even though it correctly received an IPv6 address, turned out to be the routing table. The routing table contained the following routes (among a few other ones):

2a02:2121:ca:c1d4::/64 dev rmnet_data0 proto kernel metric 256  pref medium
2a02:2121:ca:c1d4::/64 dev br-lan      proto static metric 1024 pref medium

So it attempts to set up routing to LAN for packets with destination prefix 2a02:2121:ca:c1d4::/64, but it also sets up routing to WWAN for the same packets. This is the actual root cause for why things are not routed to internet from ethernet. When I delete the route to rmnet_data0, everything works as it should.

So it looks like the following steps are the most important ones:

  • Configure network.mob1s1a1.dhcpv6='1'.
  • Configure DHCPv6/RA relay for both dhcp.lan and dhcp.mob1s1a1_6. I guess this relays any kind of DHCPv6/RA requests all the way to ISP, instead of TRB160.
  • Delete the route to rmnet_data0 for packets set to the IPv6 prefix.

After this, I have a fully functional IPv6 interface both on the laptop and TRB160. And I have the following IPs assigned on the different interfaces:

  • rmnet_data0 (WWAN): GUA/64 and link-local.
  • br-lan (LAN): GUA/64, ULA and link-local.
  • Laptop: GUA/64, link-local and public IPv4.

All the GUAs have the same prefix, which is the prefix that is delegated from ISP.

The last challenge now is to figure out how I can avoid that conflicting route in the routing table. I assume it will be re-added whenever there is a network change, so I need to hook into something that automatically does something on network changes. Or even better, figure out why that route is added at all such that it can be avoided.

1 Like

This topic was automatically closed after 15 days. New replies are no longer allowed.