Routing all traffic through a wireguard interface II

[Talleyrand]

Continuing the discussion from Routing all traffic through a wireguard interface:

@flebourse

Several users have recently asked how to route all traffic through a wireguard interface, just setting Allowed IPs to 0.0.0.0/0 + ::/0 (or similar with /1 masks) is doomed to fail because there must be an exception at least for the IP address of the wg server.
A workaround is to play with the metric of a static route to the server, it works but must be actively maintained if the IP address changes.

Here is a much cleaner solution, it is derived from the wg-quick utility.

As the UI doesn’t provide for PostUp and PreDown scripts, the method uses mwan3 in order to act on ifup and ifdown events.

First declare the wg interface in /etc/config/mwan3, like below. Set the name and table value to suit your needs:

config interface 'wgxxx'
        option family 'ipv4'
        option interval '60'
        list flush_conntrack 'connected'
        list flush_conntrack 'disconnected'
        option enabled '1'

then add the following code to /etc/mwan3.user:

table=51820
wgname="wgxxx"

[ x"$ACTION" = x"ifup" ] && [ x"$INTERFACE" = x"$wgname" ] && {
        logger -t "mwan3.user[$]" -p notice " add rules/routes $ACTION interface $INTERFACE device $DEVICE"
        wg set $INTERFACE fwmark $table                                                                     
        ip -4 route add default dev $INTERFACE table $table                                                 
        if [ $(ip -4 rule show 2>/dev/null | grep -c "lookup $table") -eq 0 ]; then                         
                ip -4 rule add not fwmark $table table $table                                               
        fi                                                                                                  
        if [ $(ip -4 rule show 2>/dev/null | grep -c "from all lookup main suppress_prefixlength 0") -eq 0 ]; then
                ip -4 rule add table main suppress_prefixlength 0                                                 
        fi                                                                                                        
        ip -6 route add default dev $INTERFACE table $table                                                       
        if [ $(ip -6 rule show 2>/dev/null | grep -c "lookup $table") -eq 0 ]; then                               
                ip -6 rule add not fwmark $table table $table                                                     
        fi                                                                                                        
        if [ $(ip -6 rule show 2>/dev/null | grep -c "from all lookup main suppress_prefixlength 0") -eq 0 ]; then
                ip -6 rule add table main suppress_prefixlength 0                                                 
        fi                                                                                                        
}

[ x"$ACTION" = x"ifdown" ] && [ x"$INTERFACE" = x"$wgname" ] && {
        logger -t "mwan3.user[$]" -p notice " delete rules/routes $ACTION interface $INTERFACE device $DEVICE"
        while [ $(ip -4 rule show 2>/dev/null | grep -c "lookup $table") -gt 0 ]; do                              
                ip -4 rule delete table $table                                                                    
        done                                                                                                      
        while [ $(ip -4 rule show 2>/dev/null | grep -c "from all lookup main suppress_prefixlength 0") -gt 0 ]; do
                ip -4 rule delete table main suppress_prefixlength 0                                              
        done                                                                                                      
        while [ $(ip -6 rule show 2>/dev/null | grep -c "lookup $table") -gt 0 ]; do                               
                ip -6 rule delete table $table                                                                    
        done                                                                                                      
        while [ $(ip -6 rule show 2>/dev/null | grep -c "from all lookup main suppress_prefixlength 0") -gt 0 ]; do
                ip -6 rule delete table main suppress_prefixlength 0                                              
        done                                                                                                      
}

Allowed IPs must be set to

0.0.0.0/1 + 128.0.0.0/1 + ::/1 + 8000::/1

At the end, restart mwan3:

/etc/init.d/mwan3 restart

and add wireguard_watchdog to cron:

echo '* * * * * /usr/bin/wireguard_watchdog' >> /etc/crontabs/root
/etc/init.d/cron restart

Local traffic (DHCP, DHCPv6, router sollicitation and so on) is still processed normally.

@Talleyrand: could you test this ?

The definition of foolish is to keep doing the same thing and expecting a different outcome. But just to make sure there were no gremlins left in the system I started off the renewed quest for establishing a wireguard tunnel that my Teltonika router will actually use by a factory reset. I then proceeded as in the quoted post.

The result was no internet access: windows indicated there was, but zero bytes in the handshake and timeout for any attempt to open a webpage etc.

Wg, iprule and iproute:

root@RUTX10:~# wg
interface: wgresid
public key: (EDITED)
private key: (hidden)
listening port: 48574

peer: (EDITED)
endpoint:E.N.D.P:48574
allowed ips: 0.0.0.0/1, 128.0.0.0/1, ::/1, 8000::/1
transfer: 0 B received, 444 B sent
persistent keepalive: every 25 seconds
root@RUTX10:~# ip -4 rule show
0: from all lookup local
1002: from all iif wgresid lookup 2
2002: from all fwmark 0x200/0x3f00 lookup 2
2061: from all fwmark 0x3d00/0x3f00 blackhole
2062: from all fwmark 0x3e00/0x3f00 unreachable
3002: from all fwmark 0x200/0x3f00 unreachable
32766: from all lookup main
32767: from all lookup default
root@RUTX10:~# ip -4 route show
default via 192.168.176.1 dev eth1 proto static src 192.168.176.198 metric 2
10.0.138.0/24 dev wgresid proto kernel scope link src 10.0.138.123
192.168.1.0/24 dev br-lan proto static scope link metric 1
192.168.176.0/24 dev eth1 proto static scope link metric 2
root@RUTX10:~#

@flebourse, if you have any new ideas please let me know.

Best regards, T

[flebourse]

Hello,
Looks like there are gremlins hidden somewhere.
From my configuration:

root@lgrrutx:~# wg
interface: wgtls
  public key: (hidden)
  private key: (hidden)
  listening port: 44527
  fwmark: 0xca6c

peer: (hidden)
  preshared key: (hidden)
  endpoint: (public IP of the server):51820
  allowed ips: 0.0.0.0/1, 128.0.0.0/1, ::/1, 8000::/1
  latest handshake: 1 minute, 1 second ago
  transfer: 1.52 GiB received, 142.72 MiB sent
  persistent keepalive: every 25 seconds

So this works fine. Now the rules and route (IPv4 only, IPv6 results are similar):

root@lgrrutx:~# ip -4 rule show
0:	from all lookup local
1001:	from all lookup main suppress_prefixlength 0
1002:	not from all fwmark 0xca6c lookup 51820
1003:	from all iif wgtls lookup 3
2003:	from all fwmark 0x300/0x3f00 lookup 3
2061:	from all fwmark 0x3d00/0x3f00 blackhole
2062:	from all fwmark 0x3e00/0x3f00 unreachable
3003:	from all fwmark 0x300/0x3f00 unreachable
32766:	from all lookup main
32767:	from all lookup default

The rule 1001 doesn’t appear in your configuration. Why ???

root@lgrrutx:~# ip -4 route show
0.0.0.0/1 dev wgtls proto static scope link metric 8 
default via 10.5.200.1 dev wlan1 proto static src 10.5.200.102 metric 4 
10.5.200.0/23 dev wlan1 proto static scope link metric 4 
(IPv4 of the server) via 10.5.200.1 dev wlan1 metric 4 
(IPv4 of the server) dev wlan1 proto static scope link metric 7 
128.0.0.0/1 dev wgtls proto static scope link metric 8 
172.31.248.0/24 dev wgtls proto static scope link metric 8 
192.168.98.0/24 dev br-lan proto kernel scope link src 192.168.98.1 

192.168.98.0/24 is the local lan, 172.31.248.0/24 is the wg network an 10.5.200.0/23 the local wan via wlan1.

Could you try to add the rules manually:

ip -4 rule add not fwmark 48574 table 48574
ip -4 rule add table main suppress_prefixlength 0

Idem for the missing routes.
Another way to do that is to “manually” invoke /etc/mwan3.user:

ACTION=ifup INTERFACE=wgresid /bin/sh /etc/mwan3.user

directly from the shell.

What is the contents of /etc/iproute2/rt_tables on the device ?

Mullvad uses another way to redirect the flow through the wireguard interface, I’ll experiment with it and let you know the results.
Regards,

[Talleyrand]

Since I lost all connectivity yesterday I started to fiddle, forgot what I did and have now reset again to get rid of any silly thing I might have put in there.

Apologies for the wall of text below, but I will give you a detailed description of what worked how/when as I proceeded for error analysis.

First step today was to add a wg instance and peer exactly as per installation file through the webUI. All fine when wg off. Also fine when wg on, but only handshake seems to go through wg interface. This was confirmed by the VPN provider test page which gave an IP address associated with the internet provider, not the VPN provider.

Wg, iproute, iprule:

root@RUTX10:~# wg
interface: wgresid
public key: (EDITED)
private key: (hidden)
listening port: 48574

peer: (EDITED)
endpoint: (PUBLIC IP OF SERVER):48574
allowed ips: 0.0.0.0/0, ::/0
latest handshake: 22 seconds ago
transfer: 184 B received, 520 B sent
persistent keepalive: every 25 seconds
root@RUTX10:~# ip -4 rule show
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
root@RUTX10:~# ip -4 route show
default via 192.168.176.1 dev eth1 proto static src 192.168.176.198 metric 2
10.0.138.0/24 dev wgresid proto kernel scope link src 10.0.138.123
192.168.1.0/24 dev br-lan proto static scope link metric 1
192.168.176.0/24 dev eth1 proto static scope link metric 2
root@RUTX10:~#

@flebourse

What is the contents of /etc/iproute2/rt_tables on the device ?

128 prelocal
255 local
254 main
253 default
0 unspec

After editing config mwan and mwan3.user + restart as above I lost internet. As yesterday Windows thinks it is there, but I cannot open any web pages and there is no new handshake in the tunnel.

Interestingly, a few bytes are received now and then in the existing tunnel but nothing seems to be sent, a behavior I have not noted before.

Wg, iprule and iproute - note that your 1001 rule now appears in 32764:

root@RUTX10:~# wg
interface: wgresid
public key: (EDITED)
private key: (hidden)
listening port: 48574
fwmark: 0xbdbe

peer: (EDITED)
endpoint: (EDITED):48574
allowed ips: 0.0.0.0/1, ::/0, 128.0.0.0/1, 8000::/1
latest handshake: 3 minutes, 15 seconds ago
transfer: 50.08 KiB received, 73.91 KiB sent
persistent keepalive: every 25 seconds
root@RUTX10:~# wg
interface: wgresid
public key: (EDITED)
private key: (hidden)
listening port: 48574
fwmark: 0xbdbe

peer: (EDITED)
endpoint: (EDITED):48574
allowed ips: 0.0.0.0/1, ::/0, 128.0.0.0/1, 8000::/1
latest handshake: 4 minutes, 10 seconds ago
transfer: 50.37 KiB received, 73.91 KiB sent
persistent keepalive: every 25 seconds
root@RUTX10:~# ip -4 rule show
0: from all lookup local
1002: from all iif wgresid lookup 2
2002: from all fwmark 0x200/0x3f00 lookup 2
2061: from all fwmark 0x3d00/0x3f00 blackhole
2062: from all fwmark 0x3e00/0x3f00 unreachable
3002: from all fwmark 0x200/0x3f00 unreachable
32764: from all lookup main suppress_prefixlength 0
32765: not from all fwmark 0xbdbe lookup 48574
32766: from all lookup main
32767: from all lookup default
root@RUTX10:~# ip -4 route show
default via 192.168.176.1 dev eth1 proto static src (PUBLIC ENDPOINT AS EDITED OUT ABOVE) metric 2
10.0.138.0/24 dev wgresid proto kernel scope link src 10.0.138.123
192.168.1.0/24 dev br-lan proto static scope link metric 1
192.168.176.0/24 dev eth1 proto static scope link metric 2
root@RUTX10:~# wg
interface: wgresid
public key: (EDITED)
private key: (hidden)
listening port: 48574
fwmark: 0xbdbe

peer: (EDITED)
endpoint: (EDITED):48574
allowed ips: 0.0.0.0/1, ::/0, 128.0.0.0/1, 8000::/1
latest handshake: 5 minutes, 50 seconds ago
transfer: 50.37 KiB received, 73.91 KiB sent
persistent keepalive: every 25 seconds
root@RUTX10:~#

@flebourse

Could you try to add the rules manually:

ip -4 rule add not fwmark 48574 table 48574
ip -4 rule add table main suppress_prefixlength 0

When trying to add the first line manually the router stopped responding while (or after?) trying to execute. I lost webUI contact and the shell never gave me a new ready signal.

I cut the power for a while and the router came alive again but still no internet. (Windows thought it had, but nothing could be opened etc).

Wg and iprule - note the disappearance of the 1001 rule after the mwan restart:

root@RUTX10:~# wg
interface: wgresid
public key: (EDITED)
private key: (hidden)
listening port: 48574
fwmark: 0xbdbe

peer: (EDITED)
endpoint: (EDITED):48574
allowed ips: 0.0.0.0/1, ::/0, 128.0.0.0/1, 8000::/1
transfer: 0 B received, 148 B sent
persistent keepalive: every 25 seconds
root@RUTX10:~# ip -4 rule show
0: from all lookup local
1000: from all lookup main suppress_prefixlength 0
1001: not from all fwmark 0xbdbe lookup 48574
1002: from all iif wgresid lookup 2
2002: from all fwmark 0x200/0x3f00 lookup 2
2061: from all fwmark 0x3d00/0x3f00 blackhole
2062: from all fwmark 0x3e00/0x3f00 unreachable
3002: from all fwmark 0x200/0x3f00 unreachable
32766: from all lookup main
32767: from all lookup default
root@RUTX10:~# /etc/init.d/mwan3 restart
root@RUTX10:~# ip -4 rule show
0: from all lookup local
1002: from all iif wgresid lookup 2
2002: from all fwmark 0x200/0x3f00 lookup 2
2061: from all fwmark 0x3d00/0x3f00 blackhole
2062: from all fwmark 0x3e00/0x3f00 unreachable
3002: from all fwmark 0x200/0x3f00 unreachable
32766: from all lookup main
32767: from all lookup default
root@RUTX10:~#

@flebourse

Another way to do that is to “manually” invoke /etc/mwan3.user:

ACTION=ifup INTERFACE=wgresid /bin/sh /etc/mwan3.user

directly from the shell.

That yielded:

root@RUTX10:~# ACTION=ifup INTERFACE=wgresid /bin/sh /etc/mwan3.user
RTNETLINK answers: File exists
RTNETLINK answers: File exists
root@RUTX10:~#

Finally here is what I added to my mwan3.user file, every line should be copypasted from your post except for my own table and wgname:

table=48574
wgname=“wgresid”

[ x"$ACTION" = x"ifup" ] && [ x"$INTERFACE" = x"$wgname" ] && {
logger -t “mwan3.user[$]” -p notice " add rules/routes $ACTION interface $INTERFACE device $DEVICE"
wg set $INTERFACE fwmark $table
ip -4 route add default dev $INTERFACE table $table
if [ $(ip -4 rule show 2>/dev/null | grep -c “lookup $table”) -eq 0 ]; then
ip -4 rule add not fwmark $table table $table
fi
if [ $(ip -4 rule show 2>/dev/null | grep -c “from all lookup main suppress_prefixlength 0”) -eq 0 ]; then
ip -4 rule add table main suppress_prefixlength 0
fi
ip -6 route add default dev $INTERFACE table $table
if [ $(ip -6 rule show 2>/dev/null | grep -c “lookup $table”) -eq 0 ]; then
ip -6 rule add not fwmark $table table $table
fi
if [ $(ip -6 rule show 2>/dev/null | grep -c “from all lookup main suppress_prefixlength 0”) -eq 0 ]; then
ip -6 rule add table main suppress_prefixlength 0
fi
}

[ x"$ACTION" = x"ifdown" ] && [ x"$INTERFACE" = x"$wgname" ] && {
logger -t “mwan3.user[$]” -p notice " delete rules/routes $ACTION interface $INTERFACE device $DEVICE"
while [ $(ip -4 rule show 2>/dev/null | grep -c “lookup $table”) -gt 0 ]; do
ip -4 rule delete table $table
done
while [ $(ip -4 rule show 2>/dev/null | grep -c “from all lookup main suppress_prefixlength 0”) -gt 0 ]; do
ip -4 rule delete table main suppress_prefixlength 0
done
while [ $(ip -6 rule show 2>/dev/null | grep -c “lookup $table”) -gt 0 ]; do
ip -6 rule delete table $table
done
while [ $(ip -6 rule show 2>/dev/null | grep -c “from all lookup main suppress_prefixlength 0”) -gt 0 ]; do
ip -6 rule delete table main suppress_prefixlength 0
done
}

I hope this helps in trying to find out where the gremlins are hiding…

Thank you for your patience & best regards,

T

[flebourse]

It seems that /etc/mwan3.user is not called when the wg interface changes state.
Could you enable Failover in Network->Failover for the wgresid interface ?

[flebourse]

Could you check that the add and delete rules/routes lines are present in the logread output:

logread | grep 'rules/routes'

[Talleyrand]

@flebourse

It seems that /etc/mwan3.user is not called when the wg interface changes state.
Could you enable Failover in Network->Failover for the wgresid interface ?

Failover seems to be enabled, but it still says disabled:

Not sure what to make of that.

@flebourse

Could you check that the add and delete rules/routes lines are present in the logread output:

logread | grep 'rules/routes'

Sun Dec 10 22:28:00 2023 user.notice mwan3.user[$]: add rules/routes ifup interface wgresid device wgresid
Sun Dec 10 22:40:33 2023 user.notice mwan3.user[$]: add rules/routes ifup interface wgresid device
Sun Dec 10 22:41:38 2023 user.notice mwan3.user[$]: delete rules/routes ifdown interface wgresid device
Sun Dec 10 22:48:00 2023 user.notice mwan3.user[$]: delete rules/routes ifdown interface wgresid device
Sun Dec 10 22:48:02 2023 user.notice mwan3.user[$]: add rules/routes ifup interface wgresid device wgresid

[flebourse]

@Talleyrand

Sun Dec 10 22:48:02 2023 user.notice mwan3.user[$]: add rules/routes ifup interface wgresid device wgresid

Bingo:

1. /etc/mwan3.user is called with the correct parameters,
2. something is wrong you shouldn’t have an isolated [$] in the output string the initial content is mwan3.user[$$] the $$ is meant to be the pid of the executing process.

So a $ character has been lost somewhere during the copy could you check that all the (single) occurences are present as expected in the script ? (else it will fail miserably …)

[Talleyrand]

I do not read the script with the understanding you do, so I am less likely to catch mistakes, but I cannot find any missing $. I have compared your post with my script and my script with my post above and they look the same to me.

Just to be sure, here is my script again from source (preceded by a bunch of # comment lines and followed by a number of ~ lines:

table=48574
wgname=“wgresid”

[ x"$ACTION" = x"ifup" ] && [ x"$INTERFACE" = x"$wgname" ] && {
logger -t “mwan3.user[$]” -p notice " add rules/routes $ACTION interface $INTERFACE device $DEVICE"
wg set $INTERFACE fwmark $table
ip -4 route add default dev $INTERFACE table $table
if [ $(ip -4 rule show 2>/dev/null | grep -c “lookup $table”) -eq 0 ]; then
ip -4 rule add not fwmark $table table $table
fi
if [ $(ip -4 rule show 2>/dev/null | grep -c “from all lookup main suppress_prefixlength 0”) -eq 0 ]; then
ip -4 rule add table main suppress_prefixlength 0
fi
ip -6 route add default dev $INTERFACE table $table
if [ $(ip -6 rule show 2>/dev/null | grep -c “lookup $table”) -eq 0 ]; then
ip -6 rule add not fwmark $table table $table
fi
if [ $(ip -6 rule show 2>/dev/null | grep -c “from all lookup main suppress_prefixlength 0”) -eq 0 ]; then
ip -6 rule add table main suppress_prefixlength 0
fi
}

[ x"$ACTION" = x"ifdown" ] && [ x"$INTERFACE" = x"$wgname" ] && {
logger -t “mwan3.user[$]” -p notice " delete rules/routes $ACTION interface $INTERFACE device $DEVICE"
while [ $(ip -4 rule show 2>/dev/null | grep -c “lookup $table”) -gt 0 ]; do
ip -4 rule delete table $table
done
while [ $(ip -4 rule show 2>/dev/null | grep -c “from all lookup main suppress_prefixlength 0”) -gt 0 ]; do
ip -4 rule delete table main suppress_prefixlength 0
done
while [ $(ip -6 rule show 2>/dev/null | grep -c “lookup $table”) -gt 0 ]; do
ip -6 rule delete table $table
done
while [ $(ip -6 rule show 2>/dev/null | grep -c “from all lookup main suppress_prefixlength 0”) -gt 0 ]; do
ip -6 rule delete table main suppress_prefixlength 0
done
}

~

Perhaps you could read through and see if any bit is missing and expected $? All occurences, including the initial ones next to the file name are single.

[flebourse]

I didn’t check the text character by character some $ are missing.
Here is a mwan3.user it is a copy of mine I have changed the table and wgname value you can transfer it as-is to the router.
Use scp or putty, or be very careful with copy and paste operations.

[Talleyrand]

Hello,

Thank you - I saw it immediately from your file: there should be two $$ in the logger lines of the script. This is at least now the case in your original post, but if you look at the quote of your original post at the top of this thread there is only one.

I now get an ip rule set very close to yours:

root@RUTX10:~# ip -4 rule show
0: from all lookup local
1000: from all lookup main suppress_prefixlength 0
1001: not from all fwmark 0xbdbe lookup 48574
1002: from all iif wgresid lookup 2
2002: from all fwmark 0x200/0x3f00 lookup 2
2061: from all fwmark 0x3d00/0x3f00 blackhole
2062: from all fwmark 0x3e00/0x3f00 unreachable
3002: from all fwmark 0x200/0x3f00 unreachable
32766: from all lookup main
32767: from all lookup default

Here is iproute, also very similar but I the third line 192 address seems to differ from your yours, which begins with 10. Could this be an indication of where the problem is?

root@RUTX10:~# ip -4 route show
0.0.0.0/1 dev wgresid proto static scope link metric 4
default via 192.168.176.1 dev eth1 proto static src 192.168.176.198 metric 2
10.0.138.0/24 dev wgresid proto static scope link metric 4
128.0.0.0/1 dev wgresid proto static scope link metric 4
192.168.1.0/24 dev br-lan proto static scope link metric 1
192.168.176.0/24 dev eth1 proto static scope link metric 2

And logread rules/routes gives me:

root@RUTX10:~# logread | grep ‘rules/routes’
Thu Dec 14 23:02:58 2023 user.notice mwan3.user[5304]: add rules/routes ifup interface wgresid device wgresid
Thu Dec 14 23:11:04 2023 user.notice mwan3.user[9791]: delete rules/routes ifdown interface wgresid device
Thu Dec 14 23:11:06 2023 user.notice mwan3.user[10628]: delete rules/routes ifdown interface wgresid device
Thu Dec 14 23:11:08 2023 user.notice mwan3.user[11298]: add rules/routes ifup interface wgresid device wgresid

…which I assume looks like it should.

So, perhaps a little closer but still no internet through my router with this installed. I get no reception:

root@RUTX10:~# wg
interface: wgresid
public key: (EDITED)
private key: (hidden)
listening port: 48574
fwmark: 0xbdbe

peer: (EDITED)
endpoint: (EDITED):48574
allowed ips: 0.0.0.0/1, ::/0, 128.0.0.0/1, 8000::/1
transfer: 0 B received, 148 B sent
persistent keepalive: every 25 seconds

The router also becomes unresponsive after a while if I have route allowed IPs turned on.

[Talleyrand]

(Since I saw the $$ change immediately I have not entered your full script but only added to double signs in two places)

[flebourse]

Ok for the $$, the copy-paste issue was at my side.
Two remarks:

@Talleyrand

0.0.0.0/1, ::/0, 128.0.0.0/1, 8000::/1

should be ::/1 instead of ::/0

@Talleyrand

transfer: 0 B received, 148 B sent

At least something is going out of the box.
Can you do a tcpdump -i any -n -v ‘port 48574’ in a cli or ssh console when the router tries to establish the tunnel ? Do you see something coming back from the server ?

[Talleyrand]

@flebourse

@Talleyrand

0.0.0.0/1, ::/0, 128.0.0.0/1, 8000::/1

should be ::/1 instead of ::/0

Oops, sorry, fixed. Thank you for pointing it out.

@flebourse

Can you do a tcpdump -i any -n -v ‘port 48574’ in a cli or ssh console when the router tries to establish the tunnel ? Do you see something coming back from the server ?

Absolutely nothing going anywhere. I run an ssh through windows and I turn the tunnel off and on in the web ui/cli and all I see is the message that it is listening.

Below wg and ip -4 route show from the cli.

root@RUTX10:~# wg
interface: wgresid
public key: (EDITED)
private key: (hidden)
listening port: 48574
fwmark: 0xbdbe

peer: (EDITED)
endpoint: (EDITED, PUBLIC ENDPOINT):48574
allowed ips: 0.0.0.0/1, ::/1, 128.0.0.0/1, 8000::/1
transfer: 0 B received, 148 B sent
persistent keepalive: every 25 seconds
root@RUTX10:~# ip -4 route show
default via 192.168.176.1 dev eth1 proto static src 192.168.176.198 metric 2
10.0.138.0/24 dev wgresid proto static scope link metric 4
192.168.1.0/24 dev br-lan proto static scope link metric 1
192.168.176.0/24 dev eth1 proto static scope link metric 2

Before, when I had a tunnel but no traffic in it (ie I had some incoming traffic in the peer section of the response to the wg command indicating a confirmation of the tunnel but no real traffic flow), I had the public IP address of the VPN server in the row ‘default via…’

In your setup/demonstration above there is a 10.* address. I assume that is because it is being routed to some internal network of yours as the endpoint as I understand 10 addresses are for that kind of use. I now have a 192.168.* address I assume to be internal to my router. If requests are being forwarded to a router address instead of to the VPN server it might not be all that strange there is no response?

As always, apologies if my questions or suggestions are naive, dumb or wrong. I am all too aware of my limited understanding of routing. But they are asked with the intention to help you help me and to perhaps learn something in the process. I am very grateful that you are taking taking time to help.

Best regards,

T

(…and I still have no internet through the router while the mwan3.user file is in place, but that would at least imply that it has taken control of traffic flows…)

[flebourse]

@Talleyrand

it has taken control of traffic flows…

Yes.

root@lgrrutx:/etc/iproute2# ip -4 route show
0.0.0.0/1 dev wgtls proto static scope link metric 8 
default via 10.5.200.1 dev wlan1 proto static src 10.5.200.104 metric 4 
10.5.200.0/23 dev wlan1 proto static scope link metric 4 
public-ipv4-of-the-wg-server via 10.5.200.1 dev wlan1 metric 4 
public-ipv4-of-the-wg-server dev wlan1 proto static scope link metric 7 
128.0.0.0/1 dev wgtls proto static scope link metric 8 
172.31.248.0/24 dev wgtls proto static scope link metric 8 
192.168.98.0/24 dev br-lan proto kernel scope link src 192.168.98.1 

Here 10.5.200.104 and 10.5.200.1 are the addresses of the local wlan1 and the remote access point repectively (a public wifi, unencrypted so I send everything through the tunnel).
I don’t see the public IPv4 address of the wg erver in your ip -4 route show. Something looks wrong.

Instead of sending all traffic through the wg interface, could you set Allowed Ips to 87.248.114.11 (www.yahoo.com) and try to ping this address ? Do the tcpdump at the same time and show the output. Check the counters from the wg output also and from iconfig wgresid. Do you see an increase ?

[Talleyrand]

There is something very funny going on. I shut down my computer but not my router, went away, found your reply and tried to edit allowed but got this in the web UI. (I have edited out the public key of the client, which is what was shown under tunnel name instead of wgresid. But the lack of button to edit and slider to turn on or off is as shown).

In CLI wg got me no response. In network/services wireguard is listed as disabled.

Both the interface wgresid and its peer are still in the etc/config/network. A reboot of the router did not solve the problem. Pulling the plug and waiting a few minutes before putting it back in did.

The ping then got me 56 bytes sent …and an endless wait.

root@RUTX10:~# ping 87.248.114.11
PING 87.248.114.11 (87.248.114.11): 56 data bytes

The tcpdump also got me nothing, see below:

root@RUTX10:~# tcpdump -i any -n -v ‘port 48574’
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes

As far as I can see there is still no traffic to the internet with then mwan3.user script in place.

@flebourse

I don’t see the public IPv4 address of the wg erver in your ip -4 route show. Something looks wrong.

Exactly. Before we entered the mwan script and I got a handshake for the tunnel but no traffic in the tunnel the second ip of that first line would be the public endpoint of the wg server. Now it is not…

Best regards

[flebourse]

@Talleyrand

something very funny going on

Hugh no. Not anymore …

Here is a new mwan3.user with many tests added.
Could you try it and post the output of logread | grep ‘rules/routes’ it will be possible to see which command fails ? Why and how to fix them would be another question of course.

[flebourse]

Could you repost the current contents of /etc/confi/network (the relevant wireguard section only) ?
It seems that the error occurs before mwan3.user is called.

[Raymond]

@flebourse

Here is a much cleaner solution

I don’t know about that. After being frustrated by attempting to get it set up and by the lack of help from Support, I spend $70 on a router and had wireguard set up through their GUI in about 5 minutes.

[Talleyrand]

@flebourse

Could you repost the current contents of /etc/confi/network (the relevant wireguard section only) ?
It seems that the error occurs before mwan3.user is called.

/etc/config/network

config interface ‘wgresid’
option proto ‘wireguard’
option private_key (EDITED)
option public_key (EDITED)
option listen_port ‘48574’
option metric ‘4’
list dns (EDITED, 4 lines of DNS servers ip 4/6 which work fine when straight from the pc wg client)
list addresses ‘10.0.138.123/24’
option disabled ‘0’

config wireguard_wgresid ‘resipeer’
option endpoint_port ‘48574’
option public_key (EDITED)
option persistent_keepalive ‘25’
option endpoint_host (EDITED, in.www.format)
option route_allowed_ips ‘0’ (NOTE: does not matter whether on or off for router behavior, when I had a working tunnel but no traffic in that tunnel turning this on killed the internet in exactly the way the mwan3 file of the suggested solution does)
list allowed_ips ‘0.0.0.0/1’
list allowed_ips ‘128.0.0.0/1’
list allowed_ips ‘::/1’
list allowed_ips ‘8000::/1’

@flebourse

Could you try it and post the output of logread | grep ‘rules/routes’ it will be possible to see which command fails ? Why and how to fix them would be another question of course.

root@RUTX10:~# logread | grep ‘rules/routes’
Tue Dec 19 00:21:45 2023 user.notice mwan3.user[5335]: add rules/routes ifup interface wgresid device wgresid
root@RUTX10:~#

…and for good measure:

root@RUTX10:~# ip -4 rule show
0: from all lookup local
1000: from all lookup main suppress_prefixlength 0
1001: not from all fwmark 0xbdbe lookup 48574
1002: from all iif wgresid lookup 2
2002: from all fwmark 0x200/0x3f00 lookup 2
2061: from all fwmark 0x3d00/0x3f00 blackhole
2062: from all fwmark 0x3e00/0x3f00 unreachable
3002: from all fwmark 0x200/0x3f00 unreachable
32766: from all lookup main
32767: from all lookup default
root@RUTX10:~# ip -4 route show
default via 192.168.176.1 dev eth1 proto static src 192.168.176.198 metric 2
10.0.138.0/24 dev wgresid proto static scope link metric 4
192.168.1.0/24 dev br-lan proto static scope link metric 1
192.168.176.0/24 dev eth1 proto static scope link metric 2

It will be no surprise to you, but whenever I try to run internet through the router with this mwan installed both e-mail and browser think they have internet but time out (sometimes server name can’t be found, sometimes cannot connect in outlook for example). The response to the wg command in the router is the same whether I have the cable to the optical media converter plugged in or not, which speaks volumes on why the tunnel is not established - but I assume this can be seen from the ‘internal’ endpoint issue I noted in a previous post and which can be seen in the response to the route show command.

When I run a successful tunnel directly from my pc through a mobile phone and keep the client window open I note that although the peer port stays at 48574 the server one can be whatever it decides when it starts up.

Hope something of this helps.

[Talleyrand]

@Raymond

I don’t know about that. After being frustrated by attempting to get it set up and by the lack of help from Support, I spend $70 on a router and had wireguard set up through their GUI in about 5 minutes.

I got another Teltonika since the mobile one I use in a remote location where I need it to stay up and restart if it goes down has worked flawlessly. Every single consumer network product of other brands I have used has disappointed and needed regular restarts, dropped traffic, had no range to speak of etc etc.

However, this experience is not encouraging. It is not clear to me if flebourse is doing this privately out of a good heart and as a challenge or if there is an association to Teltonika, but if the former I think this is perhaps the time when Teltonika could or should step in…