I came to ask status, I had marked my calendar for 11AM UTC as per the last email communication I received. Now the whole system is down many hours later. I know “something went wrong” but please can Teltonika share some up-to-date info with its customers?
Hey all,
Quick update on the current situation - the main unexpected issues that occurred during the maintenance have not been resolved yet. As a result, we will be reverting the internal system changes that were implemented during the maintenance. This process may take up to 3 hours from now, after which the system should be reachable once we feel confident it is in a stable state.
Good luck, sorry to hear this. Do we need to do anything to the endpoints (restart rms_mqtt etc) or will services restore automatically?
Hi Teltonika,
Do you have any progress updates on the rollback and how long it’ll be until the system is in a stable state? We have 135 devices offline or in failover mode and my team needs to plan ahead and manage instability as the RMS returns online. Thanks.
Also, having a 3 reply limit on threads for new users during a major outage is not helpful.
2 Likes
Whats the latest @Teltonika-support-UN. Please provide an ETA.
1 Like
This is unacceptable and unprofessional. Have you got any test environment or you try your chances on our installations? We want to know when the RMS and its features will be enable not for fun, but because we trusted in your knowledge, hardware and software. And such as situations successfully deflated our opinon about Teltonika’s environment.
3 Likes
Attn: Other impacted RMS users - we are a long standing customer of Teltonika and our company has now had well over $100,000 of trading impact as a result of this outage. We would like to get in contact with other affected Teltonika customers as we feel this has not been handled well and is in breach of contractual arrangements. If you have been affected please feel free to text message me on +17206940697 so we can decide on a joint response.
1 Like
This is absolutely UNACCEPTABLE. Your promised “3-hour” downtime has been completely shattered, with systems down for CONSIDERABLY LONGER periods. This marks yet another devastating service disruption that is directly threatening our business operations.
The impact of your failure:
- Teams working extended shifts since 1:00 PM trying to maintain basic operations
- Night shift staff stretched to breaking point
- Business continuity severely compromised
- Customer deliverables at risk
- Revenue loss mounting by the hour
Your repeated failures to meet SLA commitments are beyond inexcusable. Our entire operation has been forced into crisis management mode, with staff working exhaustive hours to compensate for YOUR system failures.
We categorically REJECT this level of service. The continued pattern of extended outages has become intolerable. Our business cannot and will not sustain operations under these conditions.
Awaiting your immediate response with ANY ETA!!!
6 Likes
Estexs - We share your view. We are looking at exploring a joint action - please feel free to message me as per our above note.
This is the update now on the portal…
" We’re currently working to restore the system as soon as possible. Thank you for your patience, and we apologize for the extended inconvenience.
Last updated: 08:45 EET; next update: in 2 hours"
Its been 20hours and your next update (not even the ETA for resolving it) will be in 2hours!?
How can you have no contingency to roll back?
1 Like
Teltonika’s website is also down. The scheduled maintenance seems to be out of control…
Next update in 2 hours, stating the maintenance site. Are you guys kidding?
My trust in Teltonika has been shattered…
Hello everyone,
As per last @Aurimas message, our RnD team is still working on restoring system functionality. The maintenance did not go as planned and we were forced to revert the update. While we do not have any ETA yet, we are working as hard as we can to restore the functionality.
We are aware of maintenance overextension and are giving all the effort to bring RMS back as soon as possible.
RMS maintenance page will be updated every two hours for the period of this maintenance.
Once again, we deeply apologize for the extended inconvenience caused.
A bit more additional information that should shed some light on the current status:
Currently the main reason for the system being offline is, in essence, issues with device firmware. Older firmware versions were unable to handle connection rejections properly during the downtime, resulting in abnormal connection requests that cannot be properly handled, leading to device connection issues across the system.
Our highest priority is getting the system back online as the team is actively developing a fix for this behavior. We deeply appreciate any and all patience during this time, and we will be updating you on the situation as soon as we are able.
3 Likes
Maybe restoring entire system from backup (you have one, yes?) is the simple way to solve the problem ? Next two hours ETA is announced… it is not acceptable. Realy not acceptable.
It looks like there is progress. I am able to log in, and my devices are now available. I can successfully connect to my devices.
Status Update
The underlying issue has been mitigated, and the system is now back online. Some functionalities may experience intermittent issues; specifically, new devices may take longer than usual to establish connections.
Our team is closely monitoring system stability and working to ensure RMS gets fully operational as soon as possible.
We appreciate your patience and understanding throughout this situation.
1 Like
Any ETA on when this will be? For the devices that “appear offline” what can be done about this if anything bearing in mind that the physical hardware is deployed with customers in the field and is not readily accessible!