RMS is down again: devices failing to connect

Continuing the discussion from Notification of planned RMS maintenance & downtime:

Same problem again from the failed maintenance a few days ago. We have about 90 devices not connecting to RMS. What is going on? We have critical systems down again for the second time in a few days. This is totally unacceptable. Where are the comms from Teltonika??? Where is the ETA? Why is there no status page?

Please respond with some answers.

5 Likes

Hi @brody

around 50% of our devices are red in RMS but I can’t connect to
the green ones either. See various other threads and postings.
Seems to be a global problem with RMS.

Regards,
Timelapse Admin

2 Likes

Hi all,

Teltonika staff is either not aware of this problem or too busy fixing it.
Hopefully the latter.

Regards,
Timelapse Admin

2 Likes

Hi all,

I’m getting a lot of “came back online” email alerts at the moment.
Still can’t access the devices though.

Regards,
Timelapse Admin

1 Like

In my case, more than a hundred are still red. Three devices returned for 3 days. There is no access to green ones.

Regards

1 Like

Hi all,

green devices seem to be reachable again.
Now trying to get just one of the red ones online
via SMS command.

Regards,
Timelapse Admin

Woke up to the same. Around 50% of managed devices down/red in RMS. Hopefully there will be an official comment from Teltonika (one that includes a postmortem from the failed maintenance a few days ago would be even nicer…)

4 Likes

Hi all,

would be great to hear from the Teltonika staff whether the system is stable again.
I don’t want to send multiple SMS’s just do discover that they don’t do anything.
Still some 30% red although they seem to be online.

Regards,
Timelapse Admin

3 Likes

Seems to exactly status quo… Nothing has changed from a user-perspective.

Teltonika has not even acknowledged this issue. I would, therefore, assume that they are not yet working on it. I would be happy to be proven wrong!

In the past, they at least acknowledged the issues, even though they did not provide an ETA or any explanation.

1 Like

Hi all,

our devices are coming back online in (roughly) 30min intervals, like at
15:58, 16;21, 17:33, and the most recent one at 17:54

Those which have been green are accessible at the moment. A few are
totally stuck, i.e. respond to SMS but throw errors in the log.

Either the system is about to fix itself, or they do not dare to tell us what’s
going on :slight_smile:

Regards,
Timelapse Admin

1 Like

Routers still stuck. They will not come back even with SMS cmd…
This is a joke

2 Likes

Hi all,

the remaining five percent of our devices try to connect as I see
in the logs - but they don’t succeed. I’ll wait until tomorrow and
hope to hear something from Teltonika.

I want to avoid un-registering and re-registering all of them.

This is really, really bad!

Regards,
Timelapse Admin

2 Likes

Is anyone else experiencing more degraded performance from about an hour ago. I was down to 8 devices offline. We are back up to 21 devices offline and a bunch on SMS failover.

I thought we were through the worst of if but seems like it is getting worse again.

Is anyone aware of any formal acknowledgement of the issues or if Teltonika is even working on a fix. This is very disheartening for me and my teams who have been working through the night trying to resolve operational issues for our affected customers.

The lack of communication from Teltonika is astounding.

3 Likes

Thanks all for detailing this.

I am in the process of registering a large number of units for deployment and have wasted a huge amount of time and credits troubleshooting this. re registering devices etc thinking this issue was on my end.

FYI no new units can be registered so far (Over 30) and as [timelapsesystems] has recommended, I will not be attempting or wasting anymore time or credits in doing so.

The fact there is no public face for this by Teltonika is unreal.

1 Like

This is a bad outage, doesn’t seem like Teltonika knows what is going on or there would be an update somewhere. Half of my devices are unable to connect to RMS as of this morning.

Hi @jacksonshanks and all the others,

our green-dot-devices seem to be OK for now, and
on the red-dot-devices I can see the last registration
attempt just ten minutes ago in the respective log
in RMS. So something is at least partially working.

As @nrobin just posted:

doesn’t seem like Teltonika knows what is going on

this might be true - the system coughing through a
backlog of tens of thousands of connection attempts.

Somewhat unbelievable that there is no weekend staff
and no systems monitoring. After all we are all here…

[Edit: last connection attempt was now two minutes ago]

Regards,
Timelapse Admin

1 Like

Hi all,

just wanted to let you know that another router just came back to green
about 75 minutes after I sent an rms_connect.

[sorry for posting so much, just want to inform about what’s happening]

Regards,
Timelapse Admin

Status from our side:

  • Most routers still offline
  • 2 routers went online

That’s it - the issue is not nearly resolved, and we are going into another forced nightshift after working on Sundays too. I feel compelled to emphasize that our teams are exhausted. Continuously requiring staff to work excessive overtime is not only costly for the company but also unsustainable for our employees’ well-being.

Oh boy - I am on-site after so far spending 4 hours trying to get a new device registered into RMS. A search found this.

I’m guessing this is what I’m banging my head against? I literally had the customer go out and buy a new unit which is doing the same thing!

Add the device in RMS, set up from brand new, ran through the wizard, then I get a notification in the target device that I must register the device in RMS first (done, and re-done, and re-done…)

The error I get on the target device is:

Failure (Server refused connection. The device may be blocked or unidentified)

So TWO brand new devices both with the exact same issue. I have an older RUT240 under the same RMS account working fine.

I have 2 unused credits sitting in the RMS management.

In short - there’s nothing I can do? This thing is supposed to go out to site tomorrow!!!