NTP001 <-> Siemens Automation PLC

Hello,

I have some kind of strange behaviour of a NTP001, which communicates with a few Siemens PLC S7-300 and S7-400. At first I have to say, that the time synchronisation with a Windows 10 is working. The messages from Windows are sometimes wrong (time synchronisation did not work but was successful), but the NTP telegrams are ok.

I made a wireshark capture an there I can see a pattern:

  1. request from Siemens PLC - time stamp from PLC included
  2. reply from NTP001 - time stamp from NTP001 included
  3. however time synchronisation did not work
  4. request from Siemens PLC - time stamp from PLC included
  5. reply from NTP001 - time stamp from PLC(!!!) included

These are the hex dump of the ntp telegrams - starting with the tag for internet protocol v4:

4500004c8c2e00001e117a94c0a80a49c0a80a45c004007b00388faee30004fa000100000001000000000000000000000000000000000000000000000000000000000000ed93439e7916872b

45b8004c2172400040118298c0a80a45c0a80a49007bc0040038b034240104ef000000000000004447505300ed93442d61a99b5aed93439e7916872bed934430b5b73047ed934430b5d5d5cf

4500004c8c3300001e117a8fc0a80a49c0a80a45c004007b003869cbe30004fa000100000001000000000000000000000000000000000000000000000000000000000000ed93439e7a1cac08

45b8004c2173400040118297c0a80a45c0a80a49007bc00400381396e40004fa0001000000010000524154450000000000000000ed93439e7a1cac08ed93439e7a1cac08ed93439e7a1cac08

Why the first correct telegram is ignored is also a question. But my question here is: why the NTP001 replies the time stamp from the PLC with the request time stamp? And had anyone the same issue with Siemens PLC?

thx

Greetings, @sersch ,

Sincerest apologies for a late response,

To investigate this further, we would appreciate if you could provide the following information:

  • Does the issue occur with all PLCs or only specific devices?
  • Is the PLC configured to use standard NTP or SNTP?
  • Does the issue persist after rebooting the NTP001?
  • Can you reproduce the issue after rebooting the NTP001?

As an additional test, if another NTP server is available, please verify whether the Siemens PLC synchronizes successfully with it. This will help determine whether the behavior is specific to the NTP001 implementation or related to the PLC configuration.

Once we receive the requested information, we will continue investigating the issue in greater detail.

Kind regards,
V.

Hello,

after a review of the wireshark captures we found the RATE signal in the answer of NTP001. This means that two requests from the same client are too close.

This behaviour can not be solved by rebooting the NTP001. And there is no option to switch off the blocking of fast requests. The PLC uses SNTP and there are no options to configure the ntp client.

So the NTP001 is not suitable for Siemens PLC S7-400. I don’t know if the NTP001 ist working with the new Siemens PLC S7-1500 series, but the S7-400 series is still active.

Our solution is to connect the NTP001 to a raspberry. There is Chrony installed which is working fine with Siemens PLC.

thx

Hello,

Thank you for sharing your insights and the workaround you implemented.

I have submitted a feature request to allow configuration of the request frequency. Our R&D team will review your suggestion, and I will get back to you with an update as soon as I receive their feedback.

All the best,
V.

Hello,

I have received an update from our R&D team.

At this time, the NTP001 does not support changing or disabling the NTP request rate.

To help us better understand your requirements, could you please provide more details about your use case? I will forward this information to our R&D team so they can evaluate the scope and intended use of this functionality. Your feedback may be taken into consideration for future firmware releases.

If this functionality is essential for your deployment, please note that we also offer the possibility of a custom development project tailored to your requirements.

Best regards,
V.

Hello,

I am only a user of Siemens S7-400. I can only switch on NTP client and set an IP-address. There is no documentation about the NTP client on S7-400. I have heard there is implemented a SNTP client but an AI-analysis of wireshark capture identifies the traffic as NTP protocol.

I don’t know why, but the wireshark capture shows after the first request of NTP client and correct reply from NTP001 there is a second request immediately after NTP001 reply. Maybe the time difference was too big and the second request was for verification of the NTP001 reply but this is only a guess and I have no background information.

best regards

C.

Hello, @sersch,

Thank you for sharing your inquiry.

After consulting with our R&D team, I can confirm that this feature is not currently planned for any upcoming firmware releases.

If this functionality is essential for your deployment, you are welcome to submit a request through the Contact Us form on the Teltonika Networks website to discuss the possibility of a custom development project.

All the best,
V.

Hello @sersch and @Vilius,

Apologies for me commenting also this, but I believe I have a valid excuse that perhaps you can approve.

I used to do quite abit of maintaining NTP servers & clients long time. I think there is reason to believe the culprit is that Siemens PLC.

The SNTP client minimum request period is 15 seconds, no client should try polling more often than that. Anyone needing more accurate clock synchro should use full NTP instead of SNTP, which is just a dumbed down version of the original.

Below you can find summary what I got asking from Google AI “What is minimum and maximum request (poll) rates allowed by the NTP and SNTP protocols?” Followed question to point out detailed location of details specified and asking it to reply in markdown format.

So I’m making educated* guess that Teltonika NTP001 is using some commonly used NTP server implementation which all usually follow rigorously the NTP spec for good reason.

*) NTP servers are not trivial to implement.

NTP software which implements limits are there to to protect its time keeping and synchronisation capabilities. When NTP-server software is run non-realtime platformss like Linux without its soft realtime extensions. It cannot keep good time if they don’t guard enough processing capability all the time they run NTP server and interact with their time sources.

NTP is a delicate process.

It can reach quite easily up to 10 ms (0.01s) accuracy over long periods and good time sources. But certainly not reach that system where it runs cannot provide process scheduling does not meet what is needed to keep good time.

Anything that affects it’s timely scheduling latency, jitter (variation in latency) etc. are really bad for NTP process. This is clearly visible if you try to run NTP service in virtualised hosts (vm guests) and when there is even bit more load from other VM’s or things Host (virtualisation platform ie. the physical server) manages and have greater priority like any IO disks, network etc. NTP services in virtualised is more or less hit and miss, often suffer to keep even 100ms (0.1s) accuracy. That time resolution might be good for many services in that VM, but do not expect good time if you try to run your datacenter primary NTP servers in virtualised. Busy clustered services and especially the ones relying simultaneously access storage or database resources that is not good enough.

Now, possibly this case having single badly behaving SNTP client doesn’t become problem that kind of sort, but if you had large numbers of that kind of clients it certainly becomes problem at some point and creates DDoS like situation. Thus NTP protocol and service do not allow it.

(There was about 10 years, back large scale publicly available NTP/SNTP vulnerability, where source address spoofed NTP-request were used to cause third party DDoS attacks. That was and would be even today possible because NTP/SNTP as connectionless protocol can be used that kind of malice especially from client sites where unfortunately no sane egress ip-filters are not in place because lack of local governmental laws and policies covering network service providing. See more info from this Wikipedia page)

I don’t know if Teltonika is willing to implement a workaround for their system and let shorten minimum times with caveats known. They may do that or not, it’s up to them.

But my point with this reply is that you @sersch should definitely try talking to someone at Siemens and let them know they have an issue with their Siemenc PLC’s, because their SNTP implementation is not following NTP specification as they should.

Here is what Google AI got me, when asking from this matter:

— Google AI reply —

The protocol specifications define request rates as exponents of 2 ($2^x$ seconds). For NTP, the allowed request rate ranges from a minimum interval of 16 seconds ($2^4$) to a maximum of 36.4 hours ($2^{17}$). For SNTP, the minimum allowed request interval is strictly 15 seconds, while the protocol enforces no absolute maximum interval.

The precise locations of these rules within the official Internet Engineering Task Force (IETF) Request for Comments (RFC) documents are detailed below.


1. SNTP Protocol Rates (RFC 4330)

The 15-Second Minimum Limit

  • Location: Page 23, Section 10 (“Best Practices”), Bullet 2.
  • Exact Text/Context: The document states:

    “A client MUST NOT transmit a request to a given server at intervals less than 15 seconds.”

  • Explanation: This is a hard protocol rule designed to prevent badly coded embedded devices from accidentally executing a Distributed Denial of Service (DDoS) attack against public time servers.

The Server “Poll Interval” Field

  • Location: Page 6, Section 4 (“NTP Packet Message Format”), under the “Poll Interval” description.
  • Exact Text/Context:

    “This field is significant only in SNTP server messages, where the values range from 4 (16 s) to 17 (131,072 s – about 36 h).”

  • Explanation: While a client can technically wait as long as it wants between updates (meaning there is no true protocol maximum limit for clients), the server uses this field to suggest a healthy polling window to the client.

2. NTP Protocol Rates (RFC 5905)

Full NTP regulates request rates dynamically using two system parameters: MINPOLL and MAXPOLL.

Protocol Limits (MINPOLL and MAXPOLL)

  • Location: Page 31, Section 7.3 (“Global Parameters”).
  • Exact Text/Context: The protocol defines the boundary constraints as:
    • MINPOLL: 4 (which means $2^4 = \mathbf{16\ seconds}$).
    • MAXPOLL: 17 (which means $2^{17} = 131,072\ seconds$, or roughly (\mathbf{36.4\ hours})).
  • Explanation: The local NTP engine dynamically scales its request rate up and down within these bounds based on network conditions and clock stability.

Suggested Default Operating Limits

  • Location: Page 15, Section 4 (“Data Types”), under the “Poll” definition.
  • Exact Text/Context:

    “Suggested default limits for minimum and maximum poll intervals are 6 and 10, respectively.”

  • Explanation: While the protocol allows shifts between 16 seconds and 36.4 hours, standard implementations (like Linux ntpd or Cisco IOS) safely restrict their default behavior between 64 seconds ($2^6$) and 1,024 seconds ($2^{10}$, or ~17 minutes).

— Google AI reply —

ps. There is an old advise I heard long time ago: “If your face looks dirty while looking at mirror, trying fixing it wiping mirror is wrong way to approach problem wash your face instead”.

That advice lesson is, find the problem source (root cause) and fix it right there, not by trying to mend things afterwards places where problem becomes visible.

Hello @mesrik,

thank you for your really detailled answer. I am in contact with Siemens Support, but I am only an user of Siemens products and at the support are working specialists but no developers.

Anyway, I work really long with Siemens products and there I thought a couple of times: “now I found a bug in an implementation” but always the remote partner had the problem. I am sure Siemens has more than one departement only for reading specifications. And sometimes, when I have really detailled information, the support can connect to development such as here.

The Siemens NTP implementation uses the burst mode, because this mode delivers the most precise results. Described here:

Poll Process

Obviously the NTP001 does not support burst mode and send the kiss-of-death paket to Siemens NTP client in order to inform NTP client to slow down requests. The KoD is also part of NTP specification for NTP servers with not enough computing power to support NTP functions like burst mode.

The problem is now the NTP001 wants to switch to NTP standard mode because of a lack of computing power. And the Siemens PLC insists on the more precise burst mode. So Siemens PLC and NTP001 do no fit together. NTP001 is not suitable for Siemens PLC or Siemens PLC is not suitable for NTP001. The result is the same - it does not work.

Hello @sersch,

Thanks for your thoughtful reply. I understand your position, you are just a customer of Siemens AG, a huge conglomerate and minor customer complaints probably are just a buzz down somewhere grass level that large behemoth hardly hears. I know, i’ve been same situation many times and had two find a workaround just to get job done like you.

Also those S7-300 and S7-400 seem to be already have legacy mark some sources I got searching and trying to get glimpse what kind of devices those are. So your luck getting them much attention to this is probably quite low.

Anyway. about NTP in general and time keeping a bit more. I write this not just for you, but perhaps someone else reading this thread also.

With time keeping always it depends greatly what kind of accuracy is needed.

  • SNTP synchronisation good enough, when distinct SNTP-clients do not have need much less than quarter minute time coordination difference issue to them. SNTP is been a while more a consumer and low accuracy time needing devices thing. It was created early 90’s by Prof. David L. Mills, same guy who created NTP protocol and started working network time sync problem from 1977, for then (90’s) low computing capability and power needs.

  • NTP is oldest still used original internet protocols because it was fortunately then really well designed. A proper time keeper and fills needs guide well with good time sources up to that already mentioned 10ms which it can fulfil even with moderate link speeds. When you are able to monitor it “ntpq -p” (I wish Teltonika would add ntpq tool to RutOS) or like you can see that it usually is even better just few milliseconds if good time sources are close enough and stable. NTP is good for general use data processing and services and syncing time over WAN interfaces too.

  • PTP (Precision Time Protocol, IEEE 1588) is newer and much more accurate than NTP. That is a thing that today where more than that 10ms accurate time needing industries should already be transitioning. PTP can keep good time sync even nanoseconds accuracy, with appropriate hardware. Most use of PTP is usually LAN only applicable.

  • And then when there are even more demanding environments like in physical sciences field which they need better than any of above protocols can provide. One of those is WR (CERN White Rabbit Project). I did not work any of that but was following interested when i knew guy who did tests and built Finnish NREN (FUNET) PTP WR network solution. Because at the time our physics department was considering particle accelerators (2 cyclotrons) next option instead having own time base when that would need to be upgraded. But I retired before that happened. Mikes in that pdf is Mittakeskus, the national metrology center and Metsähovi is a radio astronomy observatory that were primary users of that solution then 5 years ago.

OK, then just few more words about NTP usage, what I learned decades using it.

When you set up a site, which needs NTP sync. Don’t ever rely on just one time NTP server and nor two of them. It doesn’t matter if that time NTP servers time sources are GNSS or whatever.

You need alway three NTP services (site NTP servers or remote NTP) to make a proper synchronised time available for your NTP / SNTP clients.

Why?, you may ask?

Right, if you think about it a while only one NTP time source is clearly a single point of failure (SPoF) which is not good. Then OK, you tink I’ve have two of them and that will be good?

No, it isn’t. Why? Because if you think of it when you have two any kind of clocks how can you tell if which one of them is drifting?

The thing is there is no reliable way to know from two clocks which one is showing correct time and which one isn’t. Therefore you need three clocks as then you statistically very unlikely do not have two clocks drifting at same time and you can observer which one is drifting or broken.

Therefore always build a site with either three NTP servers, and they can be either local or remote as long as delay and jitter (variiation of delay) are within NTP protocol acceptable ranges.

Its a good working practise to add all three of your site NTP servers to each others as “peer” and any remote as “server” time sources. NTP protocol is quite smart, it will take care of rest. It monitor and follow which are at any time best stable time sources and decide to sync on these. It does not make your NTP kept time suddenly jump any direction. Any correction will be done either little by little slowing time if it needs to be done and same if it needs to catch up and was behind before. You don’t need to adjust any ways, just let those run and you are good when you have good multiple time sources with all of your NTP servers.

Also when you use remote NTP servers as time sources, do not configure all with exactly same time sources (other NTP servers, GNSS) if you can (are allowed and capable doing that), then you have much better change not to run in the situation that none of your site NTP servers have good time source(s) available.

If and when your site can access time sources from internet, you have multiple very likely decent NTP service pools available which using you get good time sources quite easily and it doesn’t cost you any more than the traffics you are paying using your site connection. Check www.ntp.org and their NTP pool project sources about what is available.

And please, do yourselves a favour do not run you own NTP servers in virtualised services in any kind of VM platform if you need or even appreciate in any ways good time with your systems. Local NTP-service is OK to run and keep time for that single system if it’s better than VM-agent (guest running daemon) time sync can provide. VM’s can work for a while, but may surprise you when the VM Host us under heavy load some later time.

Another thing with time to consider is what time your system should, if there is an option, select to run time internally? This writing is so long already that I do not go in to that now, but that is another time keeping related matter which can bite occasionally unaware.

Above is what I think applies most use cases NTP is used. Most important thing is not to just set one NTP-server and think that is all you need. It may be OK for a home users devices time source, but not much of anything else in my experience.

I had once a manager who thought I try to overdo this time thing when applying money upgrading time servers then recession time. But when asked back that how much he thinks would it cost if consequence of not enough accurate time for other time depending services were malfunctioning due NTP-service outages he understood and approved my request.

And I’m sorry anyone feeling bad, that I wrote so long post. But I did it because I good time keeping meaning and how NTP-services should be run are bit too little understood matter. Not that I everyone here this forum do not understand. Sure there are many, but as I see there are quite many with less experience they may think nah, I just set one NTP-server and be good. With that decision industrial and commercial environments you are most likely playing with luck more than you think you really are, if anything using that single device failure accumulate bit more than you did not think first. If that single NTP-server was not your decision you are bare network engineer, installer and not one designed it. Tell these things upwards who designs so that he/she can do educated guess and upgrade the thing when it is appropriate and would be needed. I know few examples when this was done and who I notified was later very happy I did this. He had not thought and understood this before I brought this to his attention. He was just folloiwing a customer request which mentioned in singular NTP-service ie. not asked many and redundant solution.

I think that’s enough for me. Again, apologies writhing so long post.