I’ve not been able to work this one out yet, and not sure if I can send my troubleshoot logs somewhere for someone to investigate.
I have failover configured with 4G, and two WAN (Starlink) connections.
Two customer networks are separated with configured VLAN ports , Native VLAN 1, and VLAN172.
Policy based routing is used for both, and WAN1 is used as the default route for the main table.
Network 1 uses WAN1 as the default route.
Network 172 uses WAN2 as it’s default route.
Network 1: 192.168.6.0/24
Network 172: 172.6.0.0/24
WAN1: 192.168.2.1/24
WAN2: 192.168.1.1/24
4G: Public addressing
I have failover configured, and it works without an issue if I disable the WAN interface in the Teltonika Web UI. Testing this way works for both WAN connections; the network failover switches to the other WAN, and if both WAN are disabled, they both failover to use the 4G connection.
What isn’t working is that if I reboot the starlink router on WAN1 and monitor traffic via MTR. I can see that the traffic is still using WAN1; and 192.186.2.1 is the last hop before no route to host. The failover is not seeing ping loss to 1.1.1.1, 8.8.8.8 and switching over to 4G.
I have ping method used for all 3; and flush connections enabled for connected,disconnect, interface up and interface down.
I’m using the latest RUTX_R_00.07.18.2, on RUTX11.
Additonal info. I’m curious if this is because the Starlink is not in bypass mode, and my PBR policies are using next hop, rather than interface routes.
root@RUTX11:~# ip rule show
0: from all lookup local
1: from all iif eth0.172 lookup 4
1: from all iif br-lan lookup 1
1001: from all iif eth1 lookup 1
1002: from all iif qmimux0 lookup 2
1004: from all iif eth0.4 lookup 4
1501: from 192.168.2.70 lookup 1
1504: from 192.168.1.38 lookup 4
2001: from all fwmark 0x100/0x3f00 lookup 1
2002: from all fwmark 0x200/0x3f00 lookup 2
2004: from all fwmark 0x400/0x3f00 lookup 4
2061: from all fwmark 0x3d00/0x3f00 blackhole
2062: from all fwmark 0x3e00/0x3f00 unreachable
3001: from all fwmark 0x100/0x3f00 unreachable
3002: from all fwmark 0x200/0x3f00 unreachable
3004: from all fwmark 0x400/0x3f00 unreachable
32766: from all lookup main
32767: from all lookup default
root@RUTX11:~# ip route show table all | grep 192.168
default via 192.168.2.1 dev eth1 table 1 proto static src 192.168.2.70 metric 1
192.168.2.0/24 dev eth1 table 1 proto static scope link src 192.168.2.70 metric 1
192.168.6.0/24 dev br-lan table 1 proto kernel scope link src 192.168.6.200
192.168.6.0/24 dev br-lan table 2 proto kernel scope link src 192.168.6.200
default via 192.168.1.1 dev eth0.4 table 4 proto static
default via 192.168.1.1 dev eth0.4 table 4 proto static src 192.168.1.38 metric 3
192.168.1.0/24 dev eth0.4 table 4 proto static scope link src 192.168.1.38 metric 3
192.168.6.0/24 dev br-lan table 4 proto kernel scope link src 192.168.6.200
default via 192.168.2.1 dev eth1 proto static src 192.168.2.70 metric 1
default via 192.168.1.1 dev eth0.4 proto static src 192.168.1.38 metric 3
192.168.1.0/24 dev eth0.4 proto static scope link src 192.168.1.38 metric 3
192.168.2.0/24 dev eth1 proto static scope link src 192.168.2.70 metric 1
192.168.6.0/24 dev br-lan proto kernel scope link src 192.168.6.200
local 192.168.1.38 dev eth0.4 table local proto kernel scope host src 192.168.1.38
broadcast 192.168.1.255 dev eth0.4 table local proto kernel scope link src 192.168.1.38
local 192.168.2.70 dev eth1 table local proto kernel scope host src 192.168.2.70
broadcast 192.168.2.255 dev eth1 table local proto kernel scope link src 192.168.2.70
local 192.168.6.200 dev br-lan table local proto kernel scope host src 192.168.6.200
broadcast 192.168.6.255 dev br-lan table local proto kernel scope link src 192.168.6.200
root@RUTX11:~#
root@RUTX11:~# mwan3 status
Interface status:
interface wan is online 06h:51m:21s, uptime 06h:52m:57s and tracking is active
interface mob1s1a1 is online 07h:28m:04s, uptime 07h:36m:50s and tracking is active
interface mob1s2a1 is offline and tracking is not enabled
interface wan1 is online 06h:52m:40s, uptime 06h:52m:43s and tracking is active
Current ipv4 policies:
balance_default:
wan1 (33%)
mob1s1a1 (33%)
wan (33%)
mwan_default:
wan (100%)
mwan_vlan:
wan (100%)
policy1:
wan1 (100%)
Current ipv6 policies:
balance_default:
unreachable
mwan_default:
unreachable
mwan_vlan:
unreachable
policy1:
unreachable
Directly connected ipv4 networks:
192.168.1.0/24
100.118.153.110
172.16.1.0/24
192.168.6.0/24
224.0.0.0/3
10.8.0.0/24
127.0.0.0/8
192.168.2.0/24
Directly connected ipv6 networks:
fe80::/64
fd69:a173:de90::/64
Active ipv4 user rules:
32695 3058K - mwan_vlan all -- * * 192.168.6.0/24 0.0.0.0/0
13603 2038K - policy1 all -- * * 172.16.1.0/24 0.0.0.0/0
0 0 - mwan_vlan all -- * * 172.16.2.0/24 0.0.0.0/0
Active ipv6 user rules:
root@RUTX11:~# cat /etc/config/mwan3
config globals 'globals'
option mmx_mask '0x3F00'
option rtmon_interval '5'
option mode 'mwan'
config interface 'wan'
option family 'ipv4'
option enabled '1'
option interval '3'
list flush_conntrack 'connected'
list flush_conntrack 'disconnected'
list flush_conntrack 'ifup'
list flush_conntrack 'ifdown'
config condition
option interface 'wan'
option track_method 'ping'
option reliability '1'
option count '1'
option timeout '2'
option up '3'
option down '3'
list track_ip '1.1.1.1'
list track_ip '8.8.8.8'
config member 'wan_member_mwan'
option interface 'wan'
option metric '1'
config member 'wan_member_balance'
option interface 'wan'
option weight '1'
config policy 'mwan_default'
list use_member 'wan_member_mwan'
list use_member 'mob1s1a1_member_mwan'
list use_member 'mob1s2a1_member_mwan'
option last_resort 'unreachable'
config policy 'balance_default'
option last_resort 'unreachable'
list use_member 'wan_member_balance'
list use_member 'mob1s1a1_member_balance'
list use_member 'wan1_member_balance'
config rule 'default_rule'
option family 'ipv4'
option proto 'all'
list dest_ip '0.0.0.0/0'
option sticky '0'
option use_policy 'mwan_vlan'
list src_ip '192.168.6.0/24'
config interface 'mob1s1a1'
option family 'ipv4'
list flush_conntrack 'connected'
list flush_conntrack 'disconnected'
list flush_conntrack 'ifup'
list flush_conntrack 'ifdown'
option interval '3'
option enabled '1'
config condition
option interface 'mob1s1a1'
option track_method 'ping'
option reliability '1'
option count '1'
option timeout '2'
option down '3'
option up '3'
list track_ip '1.1.1.1'
config member 'mob1s1a1_member_mwan'
option interface 'mob1s1a1'
option metric '2'
config member 'mob1s1a1_member_balance'
option interface 'mob1s1a1'
option weight '1'
config interface 'mob1s2a1'
option interval '3'
option enabled '0'
option family 'ipv4'
config condition
option interface 'mob1s2a1'
option track_method 'ping'
list track_ip '1.1.1.1'
list track_ip '8.8.8.8'
option reliability '1'
option count '1'
option timeout '2'
option down '3'
option up '3'
config member 'mob1s2a1_member_mwan'
option interface 'mob1s2a1'
option metric '3'
config member 'mob1s2a1_member_balance'
option interface 'mob1s2a1'
option weight '1'
option metric '2'
config rule 'rule1'
option sticky '0'
option proto 'all'
option name 'vlan171_rule'
list dest_ip '0.0.0.0/0'
option use_policy 'policy1'
list src_ip '172.16.1.0/24'
config policy 'mwan_vlan'
option last_resort 'unreachable'
list use_member 'wan_member_mwan'
list use_member 'mob1s1a1_member_mwan'
list use_member 'mob1s2a1_member_mwan'
config interface 'wan1'
option interval '3'
option family 'ipv4'
option enabled '1'
list flush_conntrack 'connected'
list flush_conntrack 'disconnected'
list flush_conntrack 'ifup'
list flush_conntrack 'ifdown'
config condition
option count '1'
option timeout '2'
list track_ip '1.1.1.1'
list track_ip '8.8.8.8'
option up '3'
option track_method 'ping'
option reliability '1'
option down '3'
option interface 'wan1'
config member 'wan1_member_mwan'
option interface 'wan1'
option metric '1'
config member 'wan1_member_balance'
option interface 'wan1'
option weight '1'
config rule 'vlan_rule'
option sticky '0'
option proto 'all'
option use_policy 'mwan_vlan'
list dest_ip '0.0.0.0/0'
option name 'vlan172_rule'
list src_ip '172.16.2.0/24'
config policy 'policy1'
option last_resort 'unreachable'
option name 'mwan2_vlan'
list use_member 'wan1_member_mwan'
list use_member 'mob1s1a1_member_mwan'
list use_member 'mob1s2a1_member_mwan'
root@RUTX11:~# mwan3 status
root@RUTX11:~# mwan3 status
root@RUTX11:~# cat /etc/config/mwan3
config globals 'globals'
option mmx_mask '0x3F00'
option rtmon_interval '5'
option mode 'mwan'
config interface 'wan'
option family 'ipv4'
option enabled '1'
option interval '3'
list flush_conntrack 'connected'
list flush_conntrack 'disconnected'
list flush_conntrack 'ifup'
list flush_conntrack 'ifdown'
config condition
option interface 'wan'
option track_method 'ping'
option reliability '1'
option count '1'
option timeout '2'
option up '3'
option down '3'
list track_ip '1.1.1.1'
list track_ip '8.8.8.8'
config member 'wan_member_mwan'
option interface 'wan'
option metric '1'
config member 'wan_member_balance'
option interface 'wan'
option weight '1'
config policy 'mwan_default'
list use_member 'wan_member_mwan'
list use_member 'mob1s1a1_member_mwan'
list use_member 'mob1s2a1_member_mwan'
option last_resort 'unreachable'
config policy 'balance_default'
option last_resort 'unreachable'
list use_member 'wan_member_balance'
list use_member 'mob1s1a1_member_balance'
list use_member 'wan1_member_balance'
config rule 'default_rule'
option family 'ipv4'
option proto 'all'
list dest_ip '0.0.0.0/0'
option sticky '0'
option use_policy 'mwan_vlan'
list src_ip '192.168.6.0/24'
config interface 'mob1s1a1'
option family 'ipv4'
list flush_conntrack 'connected'
list flush_conntrack 'disconnected'
list flush_conntrack 'ifup'
list flush_conntrack 'ifdown'
option interval '3'
option enabled '1'
config condition
option interface 'mob1s1a1'
option track_method 'ping'
option reliability '1'
option count '1'
option timeout '2'
option down '3'
option up '3'
list track_ip '1.1.1.1'
config member 'mob1s1a1_member_mwan'
option interface 'mob1s1a1'
option metric '2'
config member 'mob1s1a1_member_balance'
option interface 'mob1s1a1'
option weight '1'
config interface 'mob1s2a1'
option interval '3'
option enabled '0'
option family 'ipv4'
config condition
option interface 'mob1s2a1'
option track_method 'ping'
list track_ip '1.1.1.1'
list track_ip '8.8.8.8'
option reliability '1'
option count '1'
option timeout '2'
option down '3'
option up '3'
config member 'mob1s2a1_member_mwan'
option interface 'mob1s2a1'
option metric '3'
config member 'mob1s2a1_member_balance'
option interface 'mob1s2a1'
option weight '1'
option metric '2'
config rule 'rule1'
option sticky '0'
option proto 'all'
option name 'vlan171_rule'
list dest_ip '0.0.0.0/0'
option use_policy 'policy1'
list src_ip '172.16.1.0/24'
config policy 'mwan_vlan'
option last_resort 'unreachable'
list use_member 'wan_member_mwan'
list use_member 'mob1s1a1_member_mwan'
list use_member 'mob1s2a1_member_mwan'
config interface 'wan1'
option interval '3'
option family 'ipv4'
option enabled '1'
list flush_conntrack 'connected'
list flush_conntrack 'disconnected'
list flush_conntrack 'ifup'
list flush_conntrack 'ifdown'
config condition
option count '1'
option timeout '2'
list track_ip '1.1.1.1'
list track_ip '8.8.8.8'
option up '3'
option track_method 'ping'
option reliability '1'
option down '3'
option interface 'wan1'
config member 'wan1_member_mwan'
option interface 'wan1'
option metric '1'
config member 'wan1_member_balance'
option interface 'wan1'
option weight '1'
config rule 'vlan_rule'
option sticky '0'
option proto 'all'
option use_policy 'mwan_vlan'
list dest_ip '0.0.0.0/0'
option name 'vlan172_rule'
list src_ip '172.16.2.0/24'
config policy 'policy1'
option last_resort 'unreachable'
option name 'mwan2_vlan'
list use_member 'wan1_member_mwan'
list use_member 'mob1s1a1_member_mwan'
list use_member 'mob1s2a1_member_mwan'
For troubleshooting purposes, we will require more sensitive information from your end, such as the troubleshoot file, which may contain passwords, public IP addresses, serial numbers, and such. To avoid leaking this information, we have sent you a form to fill out, which you will receive in your e-mail inbox that you have registered your account with in the forums. In the Ticket ID field of the form, please enter the ID of this thread, which is 16115.
I’d not heard back on the raised Teltonika ticket I submitted since, but I managed to find one solution that appears to work.
I had to set the Starlink router to Bypass mode. I also had to set the routing policy to ‘From all networks in lan interface to wan interface’ Lookup table 1(which used, interface WAN, target 0.0.0.0, netwmask 0.0.0.0, metric 0, without setting a gateway).
Testing by rebooting the starlink, I see it now failing over to Mobile Sim, and returning once the Starlink connection has re-established.
Why, I assume that icmp destination unreachable are not returned when not in bypass mode for whatever reason; I forgot to run a packet capture to try understand what was occuring. I thought that failover checked for a response within a timeframe, and if there was no response the failover logic would take place. Both ping and wget didn’t work without the Starlink router being in bypass mode.