Issue with RUTX50 Device Post-Firmware Update to RUTX_R_00.07.07.1

Hello,

I’m experiencing an issue with my RUTX50 device following its update to firmware version RUTX_R_00.07.07.1. Every time Vodafone Romania disconnects my service, which occurs every four hours, my device encounters a system crash. The crash logs suggest a consistent pattern corresponding with each disconnection.

When the crash happens, the router displays a kernel panic error. Despite the LED indicating normal operation, the Wi-Fi network vanishes from the available networks list, and the device becomes unresponsive to ICMP requests on the LAN.

Currently, the only workaround I’ve discovered is to physically disconnect the router from the power supply and reconnect it. Could you offer any advice or solutions to this problem?

Thank you.

  • Device name: RUTX50
  • Product code: RUTX5000XXXX
  • Bootloader version: 2.3
  • Hardware revision: 0202
  • Batch number: 0018

System

  • Firmware version: RUTX_R_00.07.07.1
  • Kernel version: 5.10.210

Internal modem

  • Model: RG501Q-EU
  • FW version: RG501QEUAAR12A08M4G_04.200.04.200

details about connection:

Registered, home; Vodafone RO; 4G+ (LTE-A); VoLTE

  • Carrier aggregation: Active

  • Bandwidth: 20 MHz / 10 MHz / 10 MHz

  • Connected band: LTE B3 / LTE B3 / LTE B20

sample from /usr/var/log/messages:

Tue May 7 15:36:02 2024 kern.err kernel: [13990.963126] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
Tue May 7 15:36:02 2024 kern.err kernel: [13990.963195] rcu: 2-…!: (1 ticks this GP) idle=762/0/0x1 softirq=272316/272316 fqs=18
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.967896] (detected by 1, t=2105 jiffies, g=599913, q=6416)
Tue May 7 15:36:02 2024 kern.info kernel: [13990.976209] Sending NMI from CPU 1 to CPUs 2:
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.982967] NMI backtrace for cpu 2
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.982976] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.10.210 #0
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.982980] Hardware name: Generic DT based system
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.982984] PC is at memcpy+0x260/0x330
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.982986] LR is at 0x780a52b5
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.982990] pc : [] lr : [<780a52b5>] psr: 20000113
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.982994] sp : c10659ec ip : 00000008 fp : c083449c
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.982998] r10: c1065a30 r9 : ffffe000 r8 : c3188d00
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983002] r7 : c322c540 r6 : 00000000 r5 : c5f96b40 r4 : c1065a30
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983007] r3 : 52b51140 r2 : fffffff4 r1 : c26e4088 r0 : c31bde1c
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983012] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983016] Control: 10c5387d Table: 865a006a DAC: 00000051
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983020] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.10.210 #0
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983023] Hardware name: Generic DT based system
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983027] [] (unwind_backtrace) from [] (show_stack+0x10/0x14)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983032] [] (show_stack) from [] (dump_stack+0x9c/0xb0)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983036] [] (dump_stack) from [] (nmi_cpu_backtrace+0xb0/0xf4)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983042] [] (nmi_cpu_backtrace) from [] (do_handle_IPI+0x5c/0x12c)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983046] [] (do_handle_IPI) from [] (ipi_handler+0x18/0x20)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983051] [] (ipi_handler) from [] (generic_handle_irq+0x34/0x44)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983056] [] (generic_handle_irq) from [] (__handle_domain_irq+0x5c/0xb4)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983060] [] (__handle_domain_irq) from [] (gic_handle_irq+0x80/0x94)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983065] [] (gic_handle_irq) from [] (__irq_svc+0x6c/0x90)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983068] Exception stack(0xc1065998 to 0xc10659e0)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983073] 5980: c31bde1c c26e4088
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983078] 59a0: fffffff4 52b51140 c1065a30 c5f96b40 00000000 c322c540 c3188d00 ffffe000
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983083] 59c0: c1065a30 c083449c 00000008 c10659ec 780a52b5 c060d4e0 20000113 ffffffff
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983087] [] (__irq_svc) from [] (memcpy+0x260/0x330)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983092] [] (memcpy) from [] (ip_frag_next+0xc4/0x194)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983096] [] (ip_frag_next) from [] (ip_do_fragment+0x2ec/0x7bc)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983101] [] (ip_do_fragment) from [] (ip_output+0xc8/0x140)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983107] [] (ip_output) from [] (__ip_queue_xmit+0x150/0x3f4)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983112] [] (__ip_queue_xmit) from [] (l2tp_xmit_skb+0x1d4/0x4c0 [l2tp_core])
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983118] [] (l2tp_xmit_skb [l2tp_core]) from [] (pppol2tp_xmit+0x148/0x25c [l2tp_ppp])
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983123] [] (pppol2tp_xmit [l2tp_ppp]) from [] (ppp_unregister_compressor+0x360/0x69c [ppp_generic])
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983129] [] (ppp_unregister_compressor [ppp_generic]) from [] (ppp_input_error+0x4ec/0x8b0 [ppp_generic])
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983135] [] (ppp_input_error [ppp_generic]) from [] (ppp_output_wakeup+0x50/0xb4 [ppp_generic])
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983140] [] (ppp_output_wakeup [ppp_generic]) from [] (ppp_start_xmit+0x144/0x188 [ppp_generic])
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983146] [] (ppp_start_xmit [ppp_generic]) from [] (dev_hard_start_xmit+0xc8/0x154)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983150] [] (dev_hard_start_xmit) from [] (sch_direct_xmit+0xfc/0x2b8)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983155] [] (sch_direct_xmit) from [] (__dev_queue_xmit+0x50c/0xac0)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983160] [] (__dev_queue_xmit) from [] (neigh_xmit+0xd8/0x214)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983166] [] (neigh_xmit) from [] (nf_flow_offload_ip_hook+0x57c/0x5fc [nf_flow_table])
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983171] [] (nf_flow_offload_ip_hook [nf_flow_table]) from [] (nf_hook_slow+0x38/0xb4)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983176] [] (nf_hook_slow) from [] (__netif_receive_skb_core+0xc24/0xe48)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983182] [] (__netif_receive_skb_core) from [] (__netif_receive_skb_one_core+0x24/0x4c)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983187] [] (__netif_receive_skb_one_core) from [] (process_backlog+0x90/0x144)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983191] [] (process_backlog) from [] (__napi_poll+0x28/0x150)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983196] [] (__napi_poll) from [] (net_rx_action+0xcc/0x200)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983200] [] (net_rx_action) from [] (__do_softirq+0x120/0x2ac)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983205] [] (__do_softirq) from [] (irq_exit+0xac/0xd0)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983209] [] (irq_exit) from [] (__handle_domain_irq+0x60/0xb4)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983214] [] (__handle_domain_irq) from [] (gic_handle_irq+0x80/0x94)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983218] [] (gic_handle_irq) from [] (__irq_svc+0x6c/0x90)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983222] Exception stack(0xc1065f58 to 0xc1065fa0)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983226] 5f40: 00000000 00fb7760
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983231] 5f60: cfd51584 c0313160 ffffe000 c0c04f18 c0c04f58 00000004 00000000 c0c0ef98
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983236] 5f80: 00000000 c0b46fe8 c0c04fd0 c1065fa8 c03075b8 c03075bc 60000013 ffffffff
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983240] [] (__irq_svc) from [] (arch_cpu_idle+0x38/0x3c)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983245] [] (arch_cpu_idle) from [] (default_idle_call+0x2c/0x3c)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983249] [] (default_idle_call) from [] (do_idle+0x1e4/0x208)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983254] [] (do_idle) from [] (cpu_startup_entry+0x18/0x1c)
Tue May 7 15:36:02 2024 kern.warn kernel: [13990.983258] [] (cpu_startup_entry) from [<803016ac>] (0x803016ac)
Tue May 7 15:36:02 2024 kern.err kernel: [13990.983859] rcu: rcu_sched kthread starved for 2045 jiffies! g599913 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
Tue May 7 15:36:02 2024 kern.err kernel: [13991.464938] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
Tue May 7 15:36:02 2024 kern.err kernel: [13991.475336] rcu: RCU grace-period kthread stack dump:
Tue May 7 15:36:02 2024 kern.info kernel: [13991.484275] task:rcu_sched state:I stack: 0 pid: 11 ppid: 2 flags:0x00000000
Tue May 7 15:36:02 2024 kern.info kernel: [13991.489425] [] (__schedule) from [] (schedule+0x5c/0xe0)
Tue May 7 15:36:02 2024 kern.info kernel: [13991.497576] [] (schedule) from [] (schedule_timeout+0x1c0/0x2c4)
Tue May 7 15:36:02 2024 kern.info kernel: [13991.504863] [] (schedule_timeout) from [] (rcu_gp_kthread+0x514/0xd48)
Tue May 7 15:36:02 2024 kern.info kernel: [13991.512589] [] (rcu_gp_kthread) from [] (kthread+0x14c/0x150)
Tue May 7 15:36:02 2024 kern.info kernel: [13991.520665] [] (kthread) from [] (ret_from_fork+0x14/0x2c)
Tue May 7 15:36:02 2024 kern.info kernel: [13991.528205] Exception stack(0xc105ffb0 to 0xc105fff8)
Tue May 7 15:36:02 2024 kern.info kernel: [13991.535322] ffa0: 00000000 00000000 00000000 00000000
Tue May 7 15:36:02 2024 kern.info kernel: [13991.540448] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Tue May 7 15:36:02 2024 kern.info kernel: [13991.548606] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000

Hello,

The messages you shared indicate possible problems with performance or stability in the system. However, there’s no clear indication that these issues are caused by a network disconnection. It’s possible that there’s an additional script or another service running that could enter an execution loop when the connection is abruptly lost? Or any similar thing on the system?

Best regards,

It seems that the device could be short on memory. Maybe you have a memory leak somewhere in a process or the kernel itself.
I would be interesting to monitor the SNMP OID .1.3.6.1.4.1.2021.4.6.0 check that the remaining memory doesn’t decrease dangerously.

Here’s a bit more information about the system configuration that might help isolate the issue:

  • I have configured an L2TP tunnel along with NAT and MSS clamping.
  • DHCP has been set up for LAN clients.
  • The Wi-Fi has been configured with default settings.
  • I have set up the auto-reboot function; the system restarts if there is no response from 8.8.8.8.

Other than these configurations, everything else is set to the default options provided out of the box. Could any of these settings be contributing to the stability issues, especially in conjunction with the network disconnections?

Thank you for your assistance.

It doesn’t appear to be an issue related to memory leak.

2024-05-08 11:09:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 83764 kB
2024-05-08 11:10:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 81924 kB
2024-05-08 11:11:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 83784 kB
2024-05-08 11:12:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 88124 kB
2024-05-08 11:13:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 89836 kB
2024-05-08 11:14:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 92980 kB
2024-05-08 11:15:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 95336 kB
2024-05-08 11:16:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 94212 kB
2024-05-08 11:17:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 92116 kB
2024-05-08 11:18:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 91360 kB
2024-05-08 11:19:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 87920 kB
2024-05-08 11:20:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 89172 kB
2024-05-08 11:21:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 92380 kB
2024-05-08 11:22:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 93036 kB
2024-05-08 11:23:03 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 88676 kB
2024-05-08 11:24:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 86640 kB
2024-05-08 11:25:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 87440 kB
2024-05-08 11:26:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 90072 kB
2024-05-08 11:27:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 93172 kB
2024-05-08 11:28:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 96732 kB
2024-05-08 11:29:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 98212 kB
2024-05-08 11:30:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 97908 kB
2024-05-08 11:31:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 96956 kB
2024-05-08 11:32:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 95644 kB
2024-05-08 11:33:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 93316 kB
2024-05-08 11:34:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 94360 kB
2024-05-08 11:35:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 95620 kB
2024-05-08 11:36:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 99712 kB
2024-05-08 11:37:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 98512 kB
2024-05-08 11:38:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 99764 kB
2024-05-08 11:39:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 102100 kB
2024-05-08 11:40:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 94008 kB
2024-05-08 11:41:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 92120 kB
2024-05-08 11:42:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 93912 kB
2024-05-08 11:43:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 94180 kB
2024-05-08 11:44:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 94128 kB
2024-05-08 11:45:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 93960 kB
2024-05-08 11:46:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 94512 kB
2024-05-08 11:47:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 92836 kB
2024-05-08 11:48:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 93196 kB
2024-05-08 11:49:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 94420 kB
2024-05-08 11:50:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 97808 kB
2024-05-08 11:51:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 97412 kB
2024-05-08 11:52:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 96436 kB
2024-05-08 11:53:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 95680 kB
2024-05-08 11:54:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 97456 kB
2024-05-08 11:55:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 96696 kB
2024-05-08 11:56:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 96268 kB
2024-05-08 11:57:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 97236 kB
2024-05-08 11:58:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 96200 kB
2024-05-08 11:59:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 96088 kB
2024-05-08 12:00:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 95376 kB
2024-05-08 12:01:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 93812 kB
2024-05-08 12:02:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 97356 kB
2024-05-08 12:03:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 98876 kB
2024-05-08 12:04:02 : timeout (device crashed)
2024-05-08 12:10:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 103124 kB

Do you have the last log ?

2024-05-08 15:42:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113004 kB
2024-05-08 15:43:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113244 kB
2024-05-08 15:44:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113268 kB
2024-05-08 15:45:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113400 kB
2024-05-08 15:46:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113256 kB
2024-05-08 15:47:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113260 kB
2024-05-08 15:48:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113512 kB
2024-05-08 15:49:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113948 kB
2024-05-08 15:50:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113148 kB
2024-05-08 15:51:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113624 kB
2024-05-08 15:52:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113800 kB
2024-05-08 15:53:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113592 kB
2024-05-08 15:54:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113340 kB
2024-05-08 15:55:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113076 kB
2024-05-08 15:56:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113272 kB
2024-05-08 15:57:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 112656 kB
2024-05-08 15:58:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 110720 kB
2024-05-08 15:59:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113900 kB
2024-05-08 16:00:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113284 kB
2024-05-08 16:01:02 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 114756 kB
2024-05-08 16:02:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113936 kB
2024-05-08 16:03:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113908 kB
2024-05-08 16:04:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113328 kB
2024-05-08 16:05:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113956 kB
2024-05-08 16:06:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 113936 kB
2024-05-08 16:07:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 114196 kB

CRASH

2024-05-08 16:10:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 136048 kB
2024-05-08 16:11:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 96468 kB
2024-05-08 16:12:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 73276 kB
2024-05-08 16:13:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 84480 kB
2024-05-08 16:14:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 106100 kB
2024-05-08 16:15:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 114760 kB
2024-05-08 16:16:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 115088 kB
2024-05-08 16:17:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 111928 kB
2024-05-08 16:18:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 106828 kB
2024-05-08 16:19:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 102128 kB
2024-05-08 16:20:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 102072 kB
2024-05-08 16:21:01 : UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 108560 kB

Wed May 8 16:07:23 2024 kern.err kernel: [14344.753027] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
Wed May 8 16:07:23 2024 kern.err kernel: [14344.753090] rcu: 2-…: (14 ticks this GP) idle=fe2/0/0x1 softirq=289940/289942 fqs=430
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.757776] (detected by 1, t=2102 jiffies, g=608861, q=4131)
Wed May 8 16:07:23 2024 kern.info kernel: [14344.766107] Sending NMI from CPU 1 to CPUs 2:
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773051] NMI backtrace for cpu 2
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773059] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.10.210 #0
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773064] Hardware name: Generic DT based system
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773068] PC is at skb_copy_bits+0x18/0x350
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773071] LR is at 0xc3055000
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773075] pc : [] lr : [] psr: 40000113
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773078] sp : c10659b0 ip : c6acaa00 fp : c083449c
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773082] r10: c1065a30 r9 : ffffe000 r8 : 000005da
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773087] r7 : c3055000 r6 : 00000000 r5 : 00000000 r4 : c1065a30
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773091] r3 : 00000000 r2 : c6acaa24 r1 : 00000014 r0 : c3055000
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773095] Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773099] Control: 10c5387d Table: 8867406a DAC: 00000051
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773103] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.10.210 #0
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773107] Hardware name: Generic DT based system
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773111] [] (unwind_backtrace) from [] (show_stack+0x10/0x14)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773115] [] (show_stack) from [] (dump_stack+0x9c/0xb0)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773120] [] (dump_stack) from [] (nmi_cpu_backtrace+0xb0/0xf4)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773125] [] (nmi_cpu_backtrace) from [] (do_handle_IPI+0x5c/0x12c)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773129] [] (do_handle_IPI) from [] (ipi_handler+0x18/0x20)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773134] [] (ipi_handler) from [] (generic_handle_irq+0x34/0x44)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773139] [] (generic_handle_irq) from [] (__handle_domain_irq+0x5c/0xb4)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773144] [] (__handle_domain_irq) from [] (gic_handle_irq+0x80/0x94)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773148] [] (gic_handle_irq) from [] (__irq_svc+0x6c/0x90)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773151] Exception stack(0xc1065960 to 0xc10659a8)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773157] 5960: c3055000 00000014 c6acaa24 00000000 c1065a30 00000000 00000000 c3055000
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773162] 5980: 000005da ffffe000 c1065a30 c083449c c6acaa00 c10659b0 c3055000 c07a90d4
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773165] 59a0: 40000113 ffffffff
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773169] [] (__irq_svc) from [] (skb_copy_bits+0x18/0x350)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773174] [] (skb_copy_bits) from [] (ip_frag_next+0xe0/0x194)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773178] [] (ip_frag_next) from [] (ip_do_fragment+0x2ec/0x7bc)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773183] [] (ip_do_fragment) from [] (ip_output+0xc8/0x140)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773188] [] (ip_output) from [] (__ip_queue_xmit+0x150/0x3f4)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773194] [] (__ip_queue_xmit) from [] (l2tp_xmit_skb+0x1d4/0x4c0 [l2tp_core])
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773199] [] (l2tp_xmit_skb [l2tp_core]) from [] (pppol2tp_xmit+0x148/0x25c [l2tp_ppp])
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773204] [] (pppol2tp_xmit [l2tp_ppp]) from [] (ppp_unregister_compressor+0x360/0x69c [ppp_generic])
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773210] [] (ppp_unregister_compressor [ppp_generic]) from [] (ppp_input_error+0x4ec/0x8b0 [ppp_generic])
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773216] [] (ppp_input_error [ppp_generic]) from [] (ppp_output_wakeup+0x50/0xb4 [ppp_generic])
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773221] [] (ppp_output_wakeup [ppp_generic]) from [] (ppp_start_xmit+0x144/0x188 [ppp_generic])
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773227] [] (ppp_start_xmit [ppp_generic]) from [] (dev_hard_start_xmit+0xc8/0x154)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773231] [] (dev_hard_start_xmit) from [] (sch_direct_xmit+0xfc/0x2b8)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773236] [] (sch_direct_xmit) from [] (__dev_queue_xmit+0x50c/0xac0)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773241] [] (__dev_queue_xmit) from [] (neigh_xmit+0xd8/0x214)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773246] [] (neigh_xmit) from [] (nf_flow_offload_ip_hook+0x57c/0x5fc [nf_flow_table])
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773252] [] (nf_flow_offload_ip_hook [nf_flow_table]) from [] (nf_hook_slow+0x38/0xb4)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773257] [] (nf_hook_slow) from [] (__netif_receive_skb_core+0xc24/0xe48)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773262] [] (__netif_receive_skb_core) from [] (__netif_receive_skb_one_core+0x24/0x4c)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773267] [] (__netif_receive_skb_one_core) from [] (process_backlog+0x90/0x144)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773272] [] (process_backlog) from [] (__napi_poll+0x28/0x150)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773277] [] (__napi_poll) from [] (net_rx_action+0xcc/0x200)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773281] [] (net_rx_action) from [] (__do_softirq+0x120/0x2ac)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773285] [] (__do_softirq) from [] (irq_exit+0xac/0xd0)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773290] [] (irq_exit) from [] (__handle_domain_irq+0x60/0xb4)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773294] [] (__handle_domain_irq) from [] (gic_handle_irq+0x80/0x94)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773299] [] (gic_handle_irq) from [] (__irq_svc+0x6c/0x90)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773302] Exception stack(0xc1065f58 to 0xc1065fa0)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773307] 5f40: 00000000 029defe0
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773312] 5f60: cfd51584 c0313160 ffffe000 c0c04f18 c0c04f58 00000004 00000000 c0c0ef98
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773317] 5f80: 00000000 c0b46fe8 c0c04fd0 c1065fa8 c03075b8 c03075bc 60000013 ffffffff
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773321] [] (__irq_svc) from [] (arch_cpu_idle+0x38/0x3c)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773325] [] (arch_cpu_idle) from [] (default_idle_call+0x2c/0x3c)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773330] [] (default_idle_call) from [] (do_idle+0x1e4/0x208)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773334] [] (do_idle) from [] (cpu_startup_entry+0x18/0x1c)
Wed May 8 16:07:23 2024 kern.warn kernel: [14344.773339] [] (cpu_startup_entry) from [<803016ac>] (0x803016ac)

For some reason ppp_start_xmit never terminates there seems to be a bug in the ppp driver.

Thank you for suggestion I believe this is the solution.

I switched from my L2TP tunnel to OpenVPN, and now the router has remained stable for 13 hours so far. Interestingly, the recurring network disconnections every 4 hours have ceased.

I reviewed the logs and found nothing unusual, just the DHCP renewing every hour and the rekeying for OpenVPN also occurring hourly.

Hello @daisi40 ,

We checked and this might be a deeper issue than expected. Could you provide troubleshoot and backup files for investigating this issue?

Kind regards,

Sure, let me know where to send the zip

Hello,

Since it contains private information, we’ll handle it on a separate platform. Instructions for accessing it have been sent to the email you registered for this forum.

Best regards,