Very high latency when pinging router on LAN (RUTX10)

Teltonika Networks
1

I am experiencing a very large latency when pining router from clients on LAN (and the other way around).

My network topology is:

10.2.1.1 - RUTX10 router
10.2.1.2 - client A connected to LAN1 on RUTX10
10.2.1.3 - client B connected to LAN2 on RUTX10

Both client A and client B experience very high latency when pining RUTX10, and a similar behaviour is observed when pinging from RUTX10 to A and B

ping 10.2.1.1

PING 10.2.1.1 (10.2.1.1): 56 data bytes
64 bytes from 10.2.1.1: icmp_seq=0 ttl=64 time=348.663 ms
64 bytes from 10.2.1.1: icmp_seq=1 ttl=64 time=295.658 ms
64 bytes from 10.2.1.1: icmp_seq=2 ttl=64 time=72.992 ms
64 bytes from 10.2.1.1: icmp_seq=3 ttl=64 time=12.697 ms
64 bytes from 10.2.1.1: icmp_seq=4 ttl=64 time=27.885 ms
64 bytes from 10.2.1.1: icmp_seq=5 ttl=64 time=32.420 ms
64 bytes from 10.2.1.1: icmp_seq=6 ttl=64 time=63.559 ms
64 bytes from 10.2.1.1: icmp_seq=7 ttl=64 time=129.821 ms
64 bytes from 10.2.1.1: icmp_seq=8 ttl=64 time=229.339 ms
64 bytes from 10.2.1.1: icmp_seq=9 ttl=64 time=260.018 ms
64 bytes from 10.2.1.1: icmp_seq=10 ttl=64 time=0.652 ms
64 bytes from 10.2.1.1: icmp_seq=11 ttl=64 time=0.769 ms
64 bytes from 10.2.1.1: icmp_seq=12 ttl=64 time=0.716 ms
64 bytes from 10.2.1.1: icmp_seq=13 ttl=64 time=233.976 ms
64 bytes from 10.2.1.1: icmp_seq=14 ttl=64 time=347.060 ms

However, both clients can ping each other perfectly:

ping 10.2.1.2
PING 10.2.1.2 (10.2.1.2): 56 data bytes
64 bytes from 10.2.1.2: icmp_seq=0 ttl=64 time=1.300 ms
64 bytes from 10.2.1.2: icmp_seq=1 ttl=64 time=1.125 ms
64 bytes from 10.2.1.2: icmp_seq=2 ttl=64 time=1.045 ms
64 bytes from 10.2.1.2: icmp_seq=3 ttl=64 time=1.049 ms
64 bytes from 10.2.1.2: icmp_seq=4 ttl=64 time=1.149 ms
64 bytes from 10.2.1.2: icmp_seq=5 ttl=64 time=1.285 ms
64 bytes from 10.2.1.2: icmp_seq=6 ttl=64 time=1.025 ms
64 bytes from 10.2.1.2: icmp_seq=7 ttl=64 time=1.342 ms
64 bytes from 10.2.1.2: icmp_seq=8 ttl=64 time=2.206 ms
64 bytes from 10.2.1.2: icmp_seq=9 ttl=64 time=0.991 ms
64 bytes from 10.2.1.2: icmp_seq=10 ttl=64 time=1.133 ms
64 bytes from 10.2.1.2: icmp_seq=11 ttl=64 time=1.203 ms
64 bytes from 10.2.1.2: icmp_seq=12 ttl=64 time=1.055 ms
64 bytes from 10.2.1.2: icmp_seq=13 ttl=64 time=1.442 ms
64 bytes from 10.2.1.2: icmp_seq=14 ttl=64 time=1.026 ms

The later observation should eliminate any hypothesis about damaged network cables. The only way how A and B can communicate is via network switch on RUTX10. Something fishy must be happening in RUTX10 itself.

It also does not seem that RUTX10 is overloaded:

Mem: 162148K used, 85780K free, 524K shrd, 52K buff, 31144K cached
CPU:   2% usr   0% sys   0% nic  95% idle   0% io   0% irq   2% sirq
Load average: 0.04 0.20 0.22 2/120 6017

Other observations: RUTX10 is connected to another client ‘C’ using WireGuard tunnel. It seems that one can ping ‘C’ from within RUTX10 perfectly, much faster than any clients on LAN:

ping 10.1.1.1
PING 10.1.1.1 (10.1.1.1): 56 data bytes
64 bytes from 10.1.1.1: seq=0 ttl=64 time=2.390 ms
64 bytes from 10.1.1.1: seq=1 ttl=64 time=2.624 ms
64 bytes from 10.1.1.1: seq=2 ttl=64 time=1.849 ms
64 bytes from 10.1.1.1: seq=3 ttl=64 time=1.905 ms
64 bytes from 10.1.1.1: seq=4 ttl=64 time=2.449 ms
64 bytes from 10.1.1.1: seq=5 ttl=64 time=1.986 ms
64 bytes from 10.1.1.1: seq=6 ttl=64 time=2.081 ms
64 bytes from 10.1.1.1: seq=7 ttl=64 time=1.792 ms
64 bytes from 10.1.1.1: seq=8 ttl=64 time=2.222 ms
64 bytes from 10.1.1.1: seq=9 ttl=64 time=2.086 ms
64 bytes from 10.1.1.1: seq=10 ttl=64 time=2.083 ms
64 bytes from 10.1.1.1: seq=11 ttl=64 time=2.149 ms
64 bytes from 10.1.1.1: seq=12 ttl=64 time=1.856 ms
64 bytes from 10.1.1.1: seq=13 ttl=64 time=2.413 ms
64 bytes from 10.1.1.1: seq=14 ttl=64 time=2.060 ms
^C
--- 10.1.1.1 ping statistics ---
15 packets transmitted, 15 packets received, 0% packet loss

I simply can wrap my head around - if one assumed that RUTX10 was overloaded, why would it have an easier time pinging remote client C over a WireGuard tunnel than any clients on LAN?

As things stand, at the moment I am clueless as how to troubleshoot the problem. Any help would be much appreciated.

3

Hi audrunas,

Welcome to Teltonika Networks Community!

May I know if any other network is present in RUTX10? This might be caused by IP conflict or older firmware version. Kindly change the LAN IP to a different subnet and let me know if latency changes.

Regards,

4

Hi,
you can try to disable hardware offloading in Network->Firewall, then reboot router.

5

Thank you very much for your response. I have now resolved the problem even though I do not yet know what the exact cause was, but I suspect powerline adapters in my network. Just in case if this turns out to be useful for anyone:

  1. I traced the problem to TP-link powerline adapters within my LAN - rebooting them immediately (but temporarily) solved the symptoms until the problem resurfaced hours later. Rebooting them again helped again.
  2. I tried isolating those powerline adapters within a different subnet, but it did not help as long as that subnet touched the RUTX10 device in question.
  3. I moved those powerline adapters under a different RUTX10 device that was running a newer firmware version (RUTX_R_00.07.08) that was isolating the RUTX10 device in the question; this way powerline adapters ended up on a subnet that did not even touch the original RUTX10 device.
  4. When I saw that the other RUTX10 device was working fine, I also updated firmware of my main router from RUTX_R_00.07.07 to 08.

While I appreciate the the above description did not exactly troubleshoot the problem, I had no time to do all A/B experiments. i stopped tinkering when my home network became stable enough foe me to work. I do not fully understand how powerline adapters work and what kind of virtual switching interface they implement. I was using 3 of them in my network and in theory I was not supposed to have a switching loop, as two of them were connected to clients and one of them was connected to the RUTX10, but who knows.

6

I have resolved the problem by 1) isolating TP-link powerline adapters (three in total) under a different RUTX10 device and 2) upgrading firmware of the original RUTX10 that I was using as a main router. I do not know whether (2) would have been sufficient to fix the problem. What I know for sure is that TP-link powerline adapters were definitely complicit, as the latency problems disappeared repeatedly (albeit only for a few hours) the moment I rebooted them

7

Hi,

Glad to hear that. Let us know if further assistance is needed!

Regards,

8

This topic was automatically closed 40 hours after the last reply. New replies are no longer allowed.