RUT956 (RUT9M_R_00.07.24.3) – MQTT Modbus Gateway never reconnects after broker-side disconnect; "mosquitto loop" thread terminates, process stays alive

Device

- Model: RUT956

- Firmware: RUT9M_R_00.07.24.3

- Service: MQTT Modbus Gateway (/usr/sbin/modbusgateway)

- Config: TLS enabled (CA certificate, server verification on), port 8883,

keepalive 60 s, QoS 2, username/password auth, fixed client ID

- MQTT-Broker: self-hosted Mosquitto 2.x behind a TLS-terminating TCP proxy

Problem

After the broker side closes the connection (observed after a restart of the

cloud broker infrastructure), the MQTT Modbus Gateway never reconnects. No

data until the service is restarted manually (or the device reboots). We

also see similar behavior on RUT906 and TRB143 units and with Data to Server

(MQTT), but we have not yet measured those units in detail.

Observations on the device in the failed state (before any modbusgateway service restart)

- Process alive, same PID since boot (135 h), state S

- /proc//status: Threads: 1, voluntary_ctxt_switches: 4 (after 135 h)

- TCP socket to the broker stays in CLOSE_WAIT for more than 60 min:

tcp  Recv-Q 1  Send-Q 0  <wan-ip>:57242 -> <broker>:8883  CLOSE_WAIT  modbusgateway

→ FIN received but never read, no PINGREQ sent (keepalive 60 s),

 no reconnect attempt, no conntrack activity, no syslog message

- procd does not respawn the service, because the process does not exit

- Network path and broker were reachable at the same time: another device

behind the same router kept its connection to the same broker

After “/etc/init.d/modbusgateway restart”

- Threads: 2 → “modbusgateway” (main) + “mosquitto loop”

- Main thread voluntary_ctxt_switches: 3 → 3 in 120 s (idle, as expected)

- “mosquitto loop” voluntary_ctxt_switches: 19 → 681 in 120 s

- New connection ESTABLISHED, data flowing immediately

Steps to reproduce

1. MQTT Modbus Gateway connected via TLS (port 8883) to a Mosquitto broker

behind a TLS-terminating TCP proxy

2. Restart the MQTT broker (the proxy closes the client connection cleanly)

3. Check: P=$(pidof modbusgateway); grep Threads /proc/$P/status;

netstat -tnp | grep 8883

→ Threads: 1, socket in CLOSE_WAIT, no reconnect

Questions

1. Is this a known issue? Is the fix "Data to Server: fixed MQTT reconnect

problem" (7.24.5 for TRB1/RUTX) also relevant for the MQTT Modbus Gateway,

and when will it be released for RUT9M?

2. Was the reconnect handling of modbusgateway / data_sender changed in 7.24.x?

Previous firmware on this unit is unknown; the problem was not seen before.

Thank you in advance.

Hi,

can’t say for shure, I haven’t looked as deep into this yet as MasterG87, but I do have very similar issues on two separate TRB143’s.

The TRB143 connect to our HQ through ipsec tunnels. The mosquitto server here uses plain mqtt, no TLS (but is passed through the tunnel). Data is collected from m-bus and sent on by data to server.

Mqtt traffic will work for around 15-20h then fail and will not recover. I’ve had to rebbot, did not try to restart processes yet. Devices behind the TRB (also sending mqtt to our HQ) are not affected.

At first I thought I had received a “bad” unit, it was sold as a demo unit, older silver one. So I tried with a brand new one straight out of the box with the same results. Both units run firmware version TRB_R_00.07.24.5.

This is quite a problem. I have 16 other similar installations, all running without issues.

Greetings,

@MasterG87, communication regarding your initial report will continue through our Help Desk system, where we will assist you with further investigation.

@OscarB, could you please confirm whether the issue persists with the latest firmware version? Please share an update on your findings so we can analyse the issue further if it remains unresolved.

Kind regards,
T.

To be honest I actually downgraded to 00.07.23.7, I haven’t seen any issues on that version and it worked as I expected with this version. So I left it like that.

I cant have too large gaps in telemetry, it messes with my statistict. I could try the latest firmware but I need to set aside a few hours in case I run into trouble. I can have gaps in telemetry during a day but not between two days.

It would be nice if you can confirm the working order. I cant set up the ipsec tunnel myself and I loose config during downgrades. If I make a backup before upgrading I should be able to restore that if I end up downgrading again (to the same version) 7.23.7)?