Another update.
Connection dropped again later during the evening:
Apr 26 23:10:40 RUTM51 network: IPv4 connectivity started to fail
Apr 26 23:10:59 RUTM51 kernel: [17500.032514] usb 2-1: USB disconnect, device number 8
Apr 26 23:10:59 RUTM51 kernel: [17500.037704] cdc_ncm 2-1:1.0 usb0: unregister 'cdc_ncm' usb-1e1c0000.xhci-1, CDC NCM
Apr 26 23:10:59 RUTM51 netifd: Network device 'usb0' link is down
Apr 26 23:10:59 RUTM51 netifd: Interface 'mob1s1a1_4' has link connectivity loss
Apr 26 23:10:59 RUTM51 netifd: Interface 'mob1s1a1_4' is now down
Apr 26 23:11:00 RUTM51 kernel: [17500.263673] option1 ttyUSB0: GSM modem (1-port) converter now disconnected from ttyUSB0
Apr 26 23:11:00 RUTM51 kernel: [17500.272207] option 2-1:1.2: device disconnected
Apr 26 23:11:00 RUTM51 kernel: [17500.277686] option1 ttyUSB1: GSM modem (1-port) converter now disconnected from ttyUSB1
Apr 26 23:11:00 RUTM51 kernel: [17500.286075] option 2-1:1.3: device disconnected
Apr 26 23:11:00 RUTM51 kernel: [17500.298131] option1 ttyUSB2: GSM modem (1-port) converter now disconnected from ttyUSB2
Apr 26 23:11:00 RUTM51 kernel: [17500.306615] option 2-1:1.4: device disconnected
Apr 26 23:11:00 RUTM51 kernel: [17500.311915] option1 ttyUSB3: GSM modem (1-port) converter now disconnected from ttyUSB3
Apr 26 23:11:00 RUTM51 kernel: [17500.320326] option 2-1:1.5: device disconnected
Apr 26 23:11:00 RUTM51 kernel: [17500.325720] option1 ttyUSB4: GSM modem (1-port) converter now disconnected from ttyUSB4
Apr 26 23:11:00 RUTM51 kernel: [17500.334196] option 2-1:1.6: device disconnected
Apr 26 23:11:00 RUTM51 kernel: [17500.695582] [01-serial-symlink.sh] New device ttyUSB0 at /devices/platform/1e1c0000.xhci/usb2/2-1/2-1:1.2/ttyUSB0/tty/ttyUSB0 appeared!
Apr 26 23:11:00 RUTM51 netifd: mob1s1a1 (15673): Stopping network mob1s1a1
Apr 26 23:11:00 RUTM51 kernel: [17501.021942] [01-serial-symlink.sh] New device ttyUSB1 at /devices/platform/1e1c0000.xhci/usb2/2-1/2-1:1.3/ttyUSB1/tty/ttyUSB1 appeared!
Apr 26 23:11:00 RUTM51 gsmd[4586]: [__write:1300] warning: [CMM] Write to `2-1 [2c7c:0900]` modem failed: errno 5
Apr 26 23:11:00 RUTM51 gsmd[4586]: [common_write_handle:224] warning: [CMM] Failed to send command (0/15 bytes transferred) to `2-1 [2c7c:0900]` modem!
Apr 26 23:11:01 RUTM51 gsmd[4586]: [__write:1300] warning: [CMM] Write to `2-1 [2c7c:0900]` modem failed: errno 5
Apr 26 23:11:01 RUTM51 gsmd[4586]: [common_write_handle:224] warning: [CMM] Failed to send command (0/15 bytes transferred) to `2-1 [2c7c:0900]` modem!
Apr 26 23:11:01 RUTM51 kernel: [17501.330013] [01-serial-symlink.sh] New device ttyUSB2 at /devices/platform/1e1c0000.xhci/usb2/2-1/2-1:1.4/ttyUSB2/tty/ttyUSB2 appeared!
Apr 26 23:11:01 RUTM51 kernel: [17501.599101] [01-serial-symlink.sh] New device ttyUSB3 at /devices/platform/1e1c0000.xhci/usb2/2-1/2-1:1.5/ttyUSB3/tty/ttyUSB3 appeared!
Apr 26 23:11:01 RUTM51 kernel: [17501.885443] [01-serial-symlink.sh] New device ttyUSB4 at /devices/platform/1e1c0000.xhci/usb2/2-1/2-1:1.6/ttyUSB4/tty/ttyUSB4 appeared!
Apr 26 23:11:01 RUTM51 dnsmasq[12405]: read /etc/hosts - 12 names
Apr 26 23:11:01 RUTM51 dnsmasq[12405]: read /tmp/hosts/dhcp.cfg01411c - 4 names
Apr 26 23:11:01 RUTM51 dnsmasq-dhcp[12405]: read /etc/ethers - 0 addresses
Apr 26 23:11:02 RUTM51 gsmd[4586]: [__write:1300] warning: [CMM] Write to `2-1 [2c7c:0900]` modem failed: errno 5
Apr 26 23:11:02 RUTM51 gsmd[4586]: [common_write_handle:224] warning: [CMM] Failed to send command (0/15 bytes transferred) to `2-1 [2c7c:0900]` modem!
Apr 26 23:11:02 RUTM51 network: Mobile data disconnected (internal modem)
Apr 26 23:11:03 RUTM51 dnsmasq[12405]: read /etc/hosts - 12 names
Apr 26 23:11:03 RUTM51 dnsmasq[12405]: read /tmp/hosts/dhcp.cfg01411c - 4 names
Apr 26 23:11:03 RUTM51 dnsmasq-dhcp[12405]: read /etc/ethers - 0 addresses
Apr 26 23:11:03 RUTM51 netifd: mob1s1a1 (15673): Command failed: Request timed out
Apr 26 23:11:03 RUTM51 netifd: mob1s1a1 (15673): Command failed: Not found
Apr 26 23:11:03 RUTM51 network: Mobile data disconnected (internal modem)
Apr 26 23:11:03 RUTM51 netifd: Interface 'mobile_bridge' is now down
Apr 26 23:11:03 RUTM51 netifd: Interface 'mobile_bridge' has link connectivity loss
Apr 26 23:11:15 RUTM51 kernel: [17515.732474] usb 2-1: new SuperSpeed Gen 1 USB device number 9 using xhci-mtk
Apr 26 23:11:15 RUTM51 kernel: [17515.783164] cdc_ncm 2-1:1.0: MAC-Address: ca:d9:13:78:09:f2
Apr 26 23:11:15 RUTM51 kernel: [17515.790426] cdc_ncm 2-1:1.0 usb0: register 'cdc_ncm' at usb-1e1c0000.xhci-1, CDC NCM, ca:d9:13:78:09:f2
Apr 26 23:11:15 RUTM51 kernel: [17515.802086] option 2-1:1.2: GSM modem (1-port) converter detected
Apr 26 23:11:15 RUTM51 kernel: [17515.808947] usb 2-1: GSM modem (1-port) converter now attached to ttyUSB0
Apr 26 23:11:15 RUTM51 kernel: [17515.817980] option 2-1:1.3: GSM modem (1-port) converter detected
Apr 26 23:11:15 RUTM51 kernel: [17515.824946] usb 2-1: GSM modem (1-port) converter now attached to ttyUSB1
Apr 26 23:11:15 RUTM51 kernel: [17515.834048] option 2-1:1.4: GSM modem (1-port) converter detected
Apr 26 23:11:15 RUTM51 kernel: [17515.841275] usb 2-1: GSM modem (1-port) converter now attached to ttyUSB2
Apr 26 23:11:15 RUTM51 kernel: [17515.849815] option 2-1:1.5: GSM modem (1-port) converter detected
Apr 26 23:11:15 RUTM51 kernel: [17515.856677] usb 2-1: GSM modem (1-port) converter now attached to ttyUSB3
Apr 26 23:11:15 RUTM51 kernel: [17515.865299] option 2-1:1.6: GSM modem (1-port) converter detected
Apr 26 23:11:15 RUTM51 kernel: [17515.872077] usb 2-1: GSM modem (1-port) converter now attached to ttyUSB4
Apr 26 23:11:16 RUTM51 kernel: [17516.708891] [01-serial-symlink.sh] New device ttyUSB0 at /devices/platform/1e1c0000.xhci/usb2/2-1/2-1:1.2/ttyUSB0/tty/ttyUSB0 appeared!
Apr 26 23:11:16 RUTM51 kernel: [17516.908631] [01-serial-symlink.sh] New device ttyUSB1 at /devices/platform/1e1c0000.xhci/usb2/2-1/2-1:1.3/ttyUSB1/tty/ttyUSB1 appeared!
Apr 26 23:11:16 RUTM51 kernel: [17517.107729] [01-serial-symlink.sh] New device ttyUSB2 at /devices/platform/1e1c0000.xhci/usb2/2-1/2-1:1.4/ttyUSB2/tty/ttyUSB2 appeared!
Apr 26 23:11:17 RUTM51 kernel: [17517.306773] [01-serial-symlink.sh] New device ttyUSB3 at /devices/platform/1e1c0000.xhci/usb2/2-1/2-1:1.5/ttyUSB3/tty/ttyUSB3 appeared!
Apr 26 23:11:17 RUTM51 kernel: [17517.506173] [01-serial-symlink.sh] New device ttyUSB4 at /devices/platform/1e1c0000.xhci/usb2/2-1/2-1:1.6/ttyUSB4/tty/ttyUSB4 appeared!
Apr 26 23:11:22 RUTM51 gsmd[4586]: [ubus_notify_event:299] error: [MODEM_UBUS] Failed to send ubus notify for `2-1 [2c7c:0900]` modem!
Apr 26 23:11:22 RUTM51 mobifd: [gsm.modem0] Starting modem settings init
Apr 26 23:11:22 RUTM51 mobifd: [gsm.modem0] GSM error in "mb_generic_modem_cfun:1364" (Invalid response)
Apr 26 23:11:22 RUTM51 mobifd: [gsm.modem0] Failed to set CFUN: "2"(full)
Apr 26 23:11:22 RUTM51 kernel: SIM state changed to not inserted (internal modem)
Apr 26 23:11:40 RUTM51 mobifd: [gsm.modem0] Modem settings init completed
Apr 26 23:11:40 RUTM51 mobifd: [gsm.modem0] Starting connection setup
Apr 26 23:11:40 RUTM51 mobifd: [gsm.modem0] APN "IMS" isn't prepared for PDP "11", preparing it
Apr 26 23:11:40 RUTM51 mobifd: [gsm.modem0] GSM error in "set_pdp_ctx:169" (Invalid response)
Apr 26 23:11:40 RUTM51 mobifd: [gsm.modem0] Failed to set PDP "11" context
Apr 26 23:11:40 RUTM51 mobifd: [gsm.modem0] Failed to insert "IMS" APN
Apr 26 23:11:40 RUTM51 mobifd: [gsm.modem0] GSM error in "mb_reset_pdp_context:208" (Invalid response)
Apr 26 23:11:40 RUTM51 mobifd: [gsm.modem0] Failed to reset PDP context
Apr 26 23:11:40 RUTM51 mobifd: [gsm.modem0] Skipping "IMS" APN set
Apr 26 23:11:44 RUTM51 mobifd: [gsm.modem0] Attempting to establish connection to operator with "30s" of timeout
Apr 26 23:12:07 RUTM51 network: DNS resolution started to fail
Apr 26 23:12:44 RUTM51 mobifd: [gsm.modem0] Attempting to establish connection to operator with "60s" of timeout
it proceeds to try during the night:
Apr 26 23:15:40 RUTM51 mobifd: [gsm.modem0] Failed to get IP address from PDP context "1"
Apr 26 23:15:40 RUTM51 mobifd: [gsm.modem0] Modem is not attached to PS
Apr 26 23:15:40 RUTM51 mobifd: [gsm.modem0] Connection establishment timer set to "3600s"
Apr 26 23:15:40 RUTM51 mobifd: [gsm.modem0] Attempting to establish connection to operator with "3600s" of timeout
Apr 26 23:15:40 RUTM51 mobifd: [gsm.modem0] Connected to operator "24007"
Apr 26 23:15:41 RUTM51 mobifd: [gsm.modem0] -CFUN- Functionality: "Full"
Apr 26 23:15:41 RUTM51 mobifd: [gsm.modem0] -COPS- Mode: "Auto", operator: "24007"
Apr 26 23:15:41 RUTM51 mobifd: [gsm.modem0] -CREG- Mode: "Enabled (with location information)", status: "Emergency services only", LAC: "65534", cell ID: "88683827", technology: "E-UTRAN"
Apr 26 23:15:41 RUTM51 mobifd: [gsm.modem0] -CGREG- Mode: "Enabled (with location information)", status: "Not registered", LAC: "", cell ID: "", technology: "Unknown"
Apr 26 23:15:41 RUTM51 mobifd: [gsm.modem0] -CEREG- Mode: "Enabled (with location information)", status: "Emergency services only", LAC: "", cell ID: "88683827", technology: "E-UTRAN"
Apr 26 23:15:41 RUTM51 mobifd: [gsm.modem0] -C5GREG- Mode: "Enabled (with location information)", status: "Not registered", LAC: "", cell ID: "", technology: "Unknown"
Apr 26 23:15:41 RUTM51 mobifd: [gsm.modem0] Failed to get IP address from PDP context "1"
Apr 26 23:15:44 RUTM51 mobifd: [gsm.modem0] Failed to get IP address from PDP context "1"
Apr 26 23:15:47 RUTM51 mobifd: [gsm.modem0] Failed to get IP address from PDP context "1"
Apr 26 23:15:51 RUTM51 mobifd: [gsm.modem0] Failed to get IP address from PDP context "1"
after at scheduled reboot at 04:00 i see that the connection comes back,
but after a short while we see the connection stopping, and this call trace:
Apr 27 05:53:41 RUTM51 kernel: [ 6785.104522] ------------[ cut here ]------------
Apr 27 05:53:41 RUTM51 kernel: [ 6785.109198] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:519 dev_watchdog+0x280/0x288
Apr 27 05:53:41 RUTM51 kernel: [ 6785.117467] NETDEV WATCHDOG: usb0 (cdc_ncm): transmit queue 0 timed out 8400 ms
Apr 27 05:53:41 RUTM51 kernel: [ 6785.124795] Modules linked in: xt_connlimit pppoe ppp_async option nf_conncount l2tp_ppp iptable_nat xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT usb_wwan qmi_wwan pptp pppox ppp_mppe ppp_generic pl2303 nft_redir nft_ct nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_nat nf_flow_table nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast nf_conntrack_amanda nf_conntrack mt7615e mt7615_common mt76_connac_lib mt76 mac80211 ipt_REJECT ftdi_sio ebtable_nat ebtable_filter ebtable_broute cypress_m8 cp210x ch341 cfg80211 cdc_ncm cdc_ether belkin_sa ark3116 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_policy xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_hashlimit xt_esp xt_ecn xt_dscp xt_coova xt_comment xt_TCPMSS xt_NFLOG xt_LOG xt_HL xt_DSCP
Apr 27 05:53:41 RUTM51 kernel: [ 6785.125211] xt_CLASSIFY xfrm_interface wireguard usbserial usbnet ts_kmp ts_fsm ts_bm tla202X slhc sch_cake rfcomm nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_quota nft_objref nft_numgen nft_log nft_limit nft_hash nft_fwd_netdev nft_dup_netdev nft_counter nfnetlink_log nf_tables nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_dup_netdev nf_defrag_ipv6 nf_defrag_ipv4 libcurve25519_generic libcrc32c libchacha20poly1305 iptable_raw iptable_mangle iptable_filter ipt_ah ipt_ECN ip6table_raw ip_tables hwmon hidp hci_uart exfat ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 compat cls_flower ch343 cdc_wdm cdc_acm btusb btrtl btintel bnep bluetooth asn1_decoder pulse_counter tlt_fsb sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact hid evdev input_core tlt_gpio xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport
Apr 27 05:53:41 RUTM51 kernel: [ 6785.213233] ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 msdos ip6_gre ip_gre gre ifb l2tp_ip6 l2tp_ip l2tp_eth l2tp_netlink l2tp_core ipcomp6 xfrm6_tunnel esp6 ah6 xfrm4_tunnel ipcomp esp4 ah4 ip6_tunnel tunnel6 tunnel4 ip_tunnel vrf tun ovpn_dco_v2 udp_tunnel ip6_udp_tunnel xfrm_user xfrm_ipcomp af_key xfrm_algo vfat fat ntfs autofs4 nls_utf8 nls_iso8859_1 nls_cp437 ecdh_generic ecc sha512_generic kpp crypto_hw_eip93 des_generic libdes cmac usb_storage spi_gpio spi_bitbang gpio_74x164 xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd sd_mod scsi_mod tpm_tis_i2c gpio_button_hotplug ext4 mbcache jbd2 tpm_tis_core mii crc32c_generic
Apr 27 05:53:41 RUTM51 kernel: [ 6785.379219] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.10.234 #0
Apr 27 05:53:42 RUTM51 kernel: [ 6785.385520] Stack : 807f6c68 ffffffff 80910000 8008584c 80910000 807f7bfc 00000000 00000000
Apr 27 05:53:42 RUTM51 kernel: [ 6785.393886] 8140ddac 80a70000 8143cbd0 8089ee47 807c41b8 00000001 8140dd50 106da634
Apr 27 05:53:42 RUTM51 kernel: [ 6785.402231] 00000000 00000000 807c41b8 8140dbf0 ffffefff 8090896c 00000000 00000000
Apr 27 05:53:42 RUTM51 kernel: [ 6785.410581] 00000000 00000151 00000000 0005c953 00000000 80910000 00000000 807c0000
Apr 27 05:53:42 RUTM51 kernel: [ 6785.418931] 00000009 000020d0 00000122 ffffffff 00000018 8041489c 00000004 80a70004
Apr 27 05:53:42 RUTM51 kernel: [ 6785.427292] ...
Apr 27 05:53:42 RUTM51 kernel: [ 6785.429742] Call Trace:
Apr 27 05:53:42 RUTM51 kernel: [ 6785.432203] [<80008108>] show_stack+0x30/0x100
Apr 27 05:53:42 RUTM51 kernel: [ 6785.436639] [<803927d4>] dump_stack+0xa4/0xe0
Apr 27 05:53:42 RUTM51 kernel: [ 6785.440998] [<800304cc>] __warn+0xb0/0xd8
Apr 27 05:53:42 RUTM51 kernel: [ 6785.444996] [<80030580>] warn_slowpath_fmt+0x8c/0xac
Apr 27 05:53:42 RUTM51 kernel: [ 6785.449960] [<80587d80>] dev_watchdog+0x280/0x288
Apr 27 05:53:42 RUTM51 kernel: [ 6785.454655] [<800a38d8>] call_timer_fn.isra.30+0x20/0x90
Apr 27 05:53:42 RUTM51 kernel: [ 6785.459946] [<800a42a8>] run_timer_softirq+0x22c/0x4d8
Apr 27 05:53:42 RUTM51 kernel: [ 6785.465069] [<8070c58c>] __do_softirq+0x16c/0x334
Apr 27 05:53:42 RUTM51 kernel: [ 6785.469760] [<800351a0>] irq_exit+0xc4/0xec
Apr 27 05:53:42 RUTM51 kernel: [ 6785.473935] [<803acf9c>] plat_irq_dispatch+0x64/0x104
Apr 27 05:53:42 RUTM51 kernel: [ 6785.478969] [<80003548>] except_vec_vi_end+0xb8/0xc4
Apr 27 05:53:42 RUTM51 kernel: [ 6785.483924] [<804d4fd0>] cpuidle_enter_state_coupled+0x3b0/0x500
Apr 27 05:53:42 RUTM51 kernel: [ 6785.489909] [<804d27f0>] cpuidle_enter+0x54/0xac
Apr 27 05:53:42 RUTM51 kernel: [ 6785.494520] [<80060860>] do_idle+0x268/0x2d8
Apr 27 05:53:42 RUTM51 kernel: [ 6785.498774] [<80060b40>] cpu_startup_entry+0x24/0x34
Apr 27 05:53:42 RUTM51 kernel: [ 6785.503717] [<80004f34>] mips_cpuidle_wait_enter+0x24/0x50
Apr 27 05:53:42 RUTM51 kernel: [ 6785.509179]
Apr 27 05:53:42 RUTM51 kernel: [ 6785.511040] ---[ end trace 781a48e83551f2f1 ]---
Apr 27 05:54:25 RUTM51 network: IPv4 connectivity started to fail
Apr 27 05:54:29 RUTM51 kernel: [ 6832.704892] usb 2-1: USB disconnect, device number 2
Apr 27 05:54:29 RUTM51 kernel: [ 6832.710109] cdc_ncm 2-1:1.0 usb0: unregister 'cdc_ncm' usb-1e1c0000.xhci-1, CDC NCM
Apr 27 05:54:29 RUTM51 netifd: Network device 'usb0' link is down
Apr 27 05:54:29 RUTM51 netifd: Interface 'mob1s1a1_4' has link connectivity loss
Apr 27 05:54:29 RUTM51 netifd: Interface 'mob1s1a1_4' is now down
reconnection attempts continue until 08:15 where i have a modem reboot scheduled:
Apr 28 08:14:59 RUTM51 mobifd: [gsm.modem0] Connection establishment timer set to "3600s"
Apr 28 08:14:59 RUTM51 mobifd: [gsm.modem0] Attempting to establish connection to operator with "3600s" of timeout
Apr 28 08:14:59 RUTM51 mobifd: [gsm.modem0] Connected to operator "24007"
Apr 28 08:15:00 RUTM51 crond[10925]: USER preboot pid 18431 cmd /usr/sbin/reboot_modem.sh
it connects (ip masked to a.b.c.d):
Apr 28 08:15:58 RUTM51 mobifd: [gsm.modem0] IP address on PDP context "1": "a.b.c.d"
in addition, i see a lot of “modem failed” errors (error number 5, 11 and 19) in the log, example:
Apr 27 09:18:15 RUTM51 gsmd[4581]: [__write:1300] warning: [CMM] Write to `2-1 [2c7c:0900]` modem failed: errno 5
Apr 27 12:51:37 RUTM51 gsmd[4562]: [__write:1300] warning: [CMM] Write to `2-1 [2c7c:0900]` modem failed: errno 11
Apr 27 12:51:38 RUTM51 gsmd[4562]: [__write:1300] warning: [CMM] Write to `2-1 [2c7c:0900]` modem failed: errno 19