Hello Teltonika,
I would like to inform and report that newly released software RutOS 7.5 has very high CPU usage, even at idle rising up to 100% with Teltonika RUT240/241.
I’m not experiencing such thing as with 7.4 and 7.4.5 line software (usually I have steady 90% CPU usage display at Overview page).
Now I test RUT240 with firmware: RUT2_R_00.07.05_WEBUI
This behavior causes that I am very often (or even constantly) automatically logged out of WebUI, as I get many times even 98,5-100% CPU usage, sometimes it drops for a moment to 63-77.5% - device sits on idle, doing nothing.
Other behavior is that when I try to navigate and switch between tabs it hangs with “Loading” and I have to wait longer time to display anything or even accidently logging out of GUI, by itself.
There were moments, even at loging into device, it could not even build up welcome window and display correctly Overview page or doing it very slowly (especially view the usage of CPU parameter).
Basically, I have resetted device to factory settings and only configured simple 4G LTE SIM connection to my Povider.
I have not enabled any other CPU-consuming services, WiFi turned off - only remote access via SSH/HTTPS, nothing else.
I have checked out 'System->‘Maintenance’->‘Events Log’ but here it doesn’t show any possible reason what’s the issue of logging out of WebGUI.
BTW offtopic: I’m surprised that no one was complaining yet at the Forum, as you guys have broken out and screwed up SSH when removing RSA-support for creating keys, I cannot any longer use the most popular on Windows SSH-Client “Putty” (no support for ECDSA signatures in PuTTY suite) to access RUT240 via CLI (below RutOS 7.4.3 it works great, but higher version not, which is frustrating) - now it refuses access to it via SSH (I use latest 0.79 - PuTTY Fatal Error: Server’s host key is invalid - so it couldn’t agree on host key algorithm). Now, I have to use other non-intuitive Clients, but it’s not comfortable and convenient for me to install other Apps - like TerraTerm - only to get access to Teltonika’s.
Anyway…besides only regretting and crying out loud, below I paste you verified CLI “top” command to view current usage of processes at RUT240.
There are moments when I see its rising up to almost 97-99% when it’s refreshing data.
Any ideas or option to find source of the problem?
Or maybe newer software is too heavy and resource-consuming for RUT240 with newer features to handle it properly (too weak CPU, too short of RAM) ?
I have recently conducted a series of tests on the RUT240 device running RutOS 7.5 firmware. I observed that the CPU load remains consistently high, hovering around 100% when the device’s WebUI is active and on the overview page. However, I noticed that when navigating to other WebUI pages, excluding the overview page, the CPU load decreases, ranging from 60% to 80%. Moreover, when I tested the CPU load with the device’s WebUI turned off, the CPU load dropped significantly, reaching as low as 20%. Additionally, I did not encounter frequent disconnections from WebUI during these tests.
Further, I would recommend you to try bootloader method to solve the constantly logging off issue. This involves putting the router into bootloader menu state and attempting an update from there.
Also, for the time being, I recommend using RutOS 7.4.5 firmware version, as RutOS 7.5 is currently the most hardware demanding firmware release to this date. Additionally, I’d like to inform you that we have escalated this issue to our Research and Development (R&D) team, who will be investigating potential solutions.
Moreover, I’d like to provide you some information in reference to the Putty issue you’ve encountered. SSH RSA has indeed been disabled, but ECDSA key generation has been added. As a result, you should be able to use Putty without any issues. I would recommend to try to reset the device and change the SSH port on your device from the standard 22 to a less commonly known port, such as 9999 or another non-well-known port.
If you have additional questions, please do not hesitate to get in touch with us.
And you really recommend 7.4.5?
Why doesn’t Teltonika delete the 7.5 from the download area? Or fixes the bugs in a rush? This is really hard for me to understand
So I have to load 7.4.5 again. Is it possible to downgrade without any problems?
Hi Julius,
thank you very much for your feedback.
Well, its also worth to mention that I have at least 5-6 seconds lag (visible delay) between clicking mouse buttons and really navigating WebUI between tabs etc. to see its doing it.
I tested this on Windows 10 with: Edge, Firefox and Chrome Browser - everywhere that looked the same (Edge was even less responsive I had to click 5-10 times and wait longer for something to happen).
In RutOS 7.4.5 its much better time like 1,5-2sec delay.
I tried to Upgrade RUT240 via bootloader method procedure, but it’s not possible I get: Update Failed - Error: Incorrect checksum. (I tried with many different OS’es with same fail result, but with GUI same images are OK then can be loaded correctly to device).
Perhaps bootlader itself is too old, but I don’t know how to check it.
I also noticed that problem with high CPU utilization is not limited only to displaying WebUI, but generally speaking its affecting routing proceses and engine, as well.
I’m not sure if you guys apply, by default, some kind of CPU-defender against pinging ICMP and this is typical and usual behavior - even inside LAN, so if not, I’m experiencing some delay-issue.
There are times that I see significant increase in delay with pinging LAN segment network between local device plugged with UTP cable to RUT240.
Here I will give you another simple example:
I have configured RUT240 to Passthrough mode and plugged-in Rut240 LAN-port into my other router Juniper SRX300 behind it.
Now, although I have connected local small public subnet/30 between two devices, in fact I’m trying to ping my local-LAN segment with ICMP Echo.
This, I guess, gives same results with RutOS 7.5 and similarly with 7.4.5 – perhaps its intended by the software, but I’m not sure and convinced here.
If I had RUT240 in normal NAT mode: my PC plugged to LAN port would have: 192.168.1.[100-200] and trying to ping Teltonika gateway: 192.168.1.1.
Topology looks like this (it’s from my other threat, see only pictures from Visio from 2nd or 3rd post):
Pasting my picture for better undersanding simple topology:
RutOS 7.5
So, I’m pinging with source IP: 46.77.101.129 [JuniperSRX] to 46.77.101.130[RUT240]
I see on the statistics something like this - pasting only times when delay is higher, other 1ms ICMPs are skipped:
admin@Router> ping 46.77.101.130 source 46.77.101.129
64 bytes from 46.77.101.130: icmp_seq=28 ttl=64 time=49.612 ms
64 bytes from 46.77.101.130: icmp_seq=47 ttl=64 time=36.847 ms
64 bytes from 46.77.101.130: icmp_seq=48 ttl=64 time=58.954 ms
64 bytes from 46.77.101.130: icmp_seq=66 ttl=64 time=32.293 ms
64 bytes from 46.77.101.130: icmp_seq=85 ttl=64 time=33.798 ms
64 bytes from 46.77.101.130: icmp_seq=104 ttl=64 time=24.264 ms
64 bytes from 46.77.101.130: icmp_seq=123 ttl=64 time=29.564 ms
64 bytes from 46.77.101.130: icmp_seq=136 ttl=64 time=1.305 ms
64 bytes from 46.77.101.130: icmp_seq=142 ttl=64 time=17.331 ms
64 bytes from 46.77.101.130: icmp_seq=161 ttl=64 time=28.038 ms
64 bytes from 46.77.101.130: icmp_seq=181 ttl=64 time=24.427 ms
64 bytes from 46.77.101.130: icmp_seq=200 ttl=64 time=36.679 ms
64 bytes from 46.77.101.130: icmp_seq=258 ttl=64 time=36.046 ms
64 bytes from 46.77.101.130: icmp_seq=316 ttl=64 time=40.609 ms
64 bytes from 46.77.101.130: icmp_seq=335 ttl=64 time=15.638 ms
64 bytes from 46.77.101.130: icmp_seq=354 ttl=64 time=30.484 ms
64 bytes from 46.77.101.130: icmp_seq=373 ttl=64 time=22.999 ms
64 bytes from 46.77.101.130: icmp_seq=374 ttl=64 time=1.224 ms
64 bytes from 46.77.101.130: icmp_seq=375 ttl=64 time=1.226 ms
64 bytes from 46.77.101.130: icmp_seq=376 ttl=64 time=1.350 ms
64 bytes from 46.77.101.130: icmp_seq=412 ttl=64 time=44.337 ms
64 bytes from 46.77.101.130: icmp_seq=431 ttl=64 time=14.591 ms
64 bytes from 46.77.101.130: icmp_seq=451 ttl=64 time=44.754 ms
…
— 46.77.101.130 ping statistics —
1651 packets transmitted, 1651 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.005/3.309/558.929/15.720 ms
RutOS 7.4.5 - Passthrough, same config only earlier OS, same issue only paste fragment of it
admin@Router> ping 46.77.101.130 source 46.77.101.129
PING 46.77.101.130 (46.77.101.130): 56 data bytes
64 bytes from 46.77.101.130: icmp_seq=2 ttl=64 time=43.502 ms
64 bytes from 46.77.101.130: icmp_seq=21 ttl=64 time=19.297 ms
64 bytes from 46.77.101.130: icmp_seq=79 ttl=64 time=48.403 ms
64 bytes from 46.77.101.130: icmp_seq=98 ttl=64 time=24.480 ms
64 bytes from 46.77.101.130: icmp_seq=137 ttl=64 time=39.981 ms
64 bytes from 46.77.101.130: icmp_seq=156 ttl=64 time=15.528 ms
64 bytes from 46.77.101.130: icmp_seq=176 ttl=64 time=14.116 ms
RutOS 7.4.5 - default factory reset settings same scenario I use NAT mode and have default subnet: 192.168.1.0/24 between devices 192.168.1.140 [Juniper] and 192.168.1.1 [RUT240]:
admin@Router> ping 192.168.1.1 source 192.168.1.140
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=45.608 ms
64 bytes from 192.168.1.1: icmp_seq=24 ttl=64 time=27.838 ms
64 bytes from 192.168.1.1: icmp_seq=43 ttl=64 time=11.411 ms
64 bytes from 192.168.1.1: icmp_seq=62 ttl=64 time=16.227 ms
64 bytes from 192.168.1.1: icmp_seq=82 ttl=64 time=42.047 ms
64 bytes from 192.168.1.1: icmp_seq=101 ttl=64 time=34.094 ms
64 bytes from 192.168.1.1: icmp_seq=120 ttl=64 time=11.109 ms
64 bytes from 192.168.1.1: icmp_seq=140 ttl=64 time=14.754 ms
64 bytes from 192.168.1.1: icmp_seq=159 ttl=64 time=39.448 ms
…
…
— 192.168.1.1 ping statistics —
1601 packets transmitted, 1601 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.996/2.615/54.832/6.237 ms
This should not look like this, because LAN IP traffic should almost always be between <1-3ms if device is not saturated and bandwidth utilized to maximum, values like: 50-100ms, ocasionally even ~500ms - proves that software-CPU is in fact overloaded and traffic is delayed, after all. Maybe I am missing something. I dunno. I have also checked ‘Network’ → ‘Firewall’ → Attack Prevention, here REMOTE ICMP REQUESTS section, where I have default settings:
Enable ICMP requests is On
Enable ICMP limit if Off
Also, ‘Firewall’ → ‘General Settings’ → Software flow offloading is On → but nothing changes.
BTW: Thanks as well for looking into SSH-Putty matter. I already changed port from 22 to 9022 as less common and security purpose.
But, my Putty is not by default negotiating ECDSA keys (even I think it has option to handle it) during first connections with RUT’s - it refuses connection and closing session, automatically.
So I guess first, I would need to generate such keys at RUT’s, export them to PC and import again at Putty to properly handle next connection - this overall process is challenging and inconvenient,
rather than using other SSH-Client (TeraTerm) that negotiate ECDSA 256 automatically during first SSH session and works without any problem.
Anyway, thanks for your interest and providing some information.
Firstly, Robert, I would like to thank you for providing valuable and informative insights regarding the CPU issue. As mentioned earlier, this matter has been registered, and we will investigate this case. Your patience and cooperation in resolving this matter are greatly appreciated.
Additionally, regarding the Putty issue, I recommend attempting to resolve it by uninstalling Putty completely and then reinstalling the latest version (0.79), it’s possible that there may be some corruption in the Putty configuration settings themselves. I can assure you that Putty should establish a successful session with our devices without requiring any additional key import/export processes.
In response to Markus inquiry, I kindly request further details about the issues you have encountered. Are you experiencing the same issues as mentioned by Robert in the original post, or do you encounter any other specific CPU-related issues? Additionally, may I inquire about the frequency at which these issues occur for you? Your input is highly valuable to us as we work to address this concern effectively.
Hello Julius,
one more thing. I made some additional tests. I believe there is active some kind of CPU-defence system (in the background). This may be observed with large (or even all) RutOS’s, but I didn’t notice this earlier, so didn’t test with released firmware 7.1-7.4.
When I start pinging from Router (Juniper) → Router (RUT240) through ‘LAN’ port into Teltonika I get every 20’-30’ pings randomly single delay [usually 20-50ms].
When I start pinging from my PC (Win7) → Router(RUT240) through ‘LAN’ port into Teltonika I experience no delay, at all. Its clean from about 1500+ packets.
If you can, please consult that with Developers is there some logic and mechanism behind it.
Sometimes, such behavior are observed as deployed anti-DDoS mechanism to protect-CPU and let outside attacker think that large number of flow has hitted target and success to decrease performance of device.
ICMP packets can be here handled at different (better or worst level) by CPU, depend of applied-policy and other factors (source its being received).
What I see only difference here with ICMP’s generated from PC and Router is packet size=32bytes(PC) vs 64bytes(Router). TTL is 64 the same.
I don’t bother to capture detailed ICMP packet (Router vs PC) and compare them both to recognize in Wireshark to check if there are some other differences.
Anyway, my ticket will probably will autoclose within 2-3 days without extension.
So, once you get answer from Devs please re-open case (if u can) past information and close it.
Thank you very much.
PS.
I used my laptop (Win7 with setup IP 192.168.1.133/24; Gateway RUT240: 192.168.1.1 direct-connected cable from Ethernet adapter to LAN Rut240 port) to make ping test.
At summary you see that averege time and maximum is always <1ms.
C:\Users\Robert>ping -t 192.168.1.1
Testing 192.168.1.1 with 32 bytes of data:
Response from 192.168.1.1: bytes=32 time<1 ms TTL=64
Response from 192.168.1.1: bytes=32 time<1 ms TTL=64
Response from 192.168.1.1: bytes=32 time<1 ms TTL=64
Response from 192.168.1.1: bytes=32 time<1 ms TTL=64
…
Ping statistics for 192.168.1.1:
Packets: Sent = 1658, Received = 1658, Lost = 0
(0% loss),
Estimated packet wandering time in milliseconds:
Minimum = 0 ms, Maximum = 1 ms, Average time = 0 ms
I have consulted with our R&D team regarding your inquiry about the CPU-defense system, and I can affirm that there are no CPU-defense systems in place that could cause the CPU load to reach 100% or disrupt routing or any other functionalities. However, I would like to inform you that RUT240 has a weaker hardware than other devices, so it takes up a lot more CPU resources to run RUTOS. Device needs more time and resources to process data in the system. Also, I can inform you that with RutOS 7.6 the logging off issue, which you experienced previously, will be fixed and there will be improvements regarding data collecting part, in result, CPU load won’t get up to 100%.
