Modbus Communication Link Disruption At Random Times

Please describe to the forum how the various telephone calls to the equipment suppliers have gone so far, and why these suppliers have been unable to provide information that led to solving the problem to date?

Both DCS and package vendors have done their checks and found nothing unusual. As mentioned in the original post, the communication is working fine with other package units. Even for the above mentioned package, link goes down randomly. So, problem lies somewhere in the middle.

<...This link down error sometimes comes twice in a day...> Does this correspond with the tidal minima or maxima?

No. Until now, we can't co-relate this communication error frequency with any factor. We have not even found any single message or alarm in the diagnostic log of switch for that. This link down error is coming only in DCS at random times.

It is unlikely that the forum is going to be able to sort this problem, as it cannot be seen from here.

Hire a local qualified and experienced Electronics Engineer and take it from there.

Sounds like a software program issue....that unit might have a different version program, could be spyware maybe...could be a newer version board with added feature....I think I would do a temperature scan first, to look for any anomalies....eliminate any overheating issues....

Thank you for the comment. Software versions, security and temperature have already been checked.

I can't help you because I have no knowledge of how redundant communications operates, but if I were tasked to do this, I would be asking these questions:

Exactly what did a DCS "vendor check" consist of? What was in their written report?
Exactly what did a package "vendor check" consist of? What was in their written report?

Who has the spreadsheet confirming firmware versions on all devices at both ends and in-between? Your review of that spreadsheet confirms that all three sets of devices are running identical firmware versions?

Exactly how is the "Link Down" error is generated? Can different conditions trip a Link Down error? What test equipment is needed to view the generation of a Link Down error? What test equipment is needed to analyze and trap traffic that creates a Link Down error? Was that equipment used in the vendor checks?

What are the details of the comm protocol for redundant comm? Where are the details published?

>Both DCS and package vendors have done their checks and found nothing unusual. As mentioned in the original post, the communication is working fine with other package units. Even for the above mentioned package, link goes down randomly. So, problem lies somewhere in the middle.

Given the firmware nature of communication links and the number of versions and revisions to versions for things like comm modules and network switches, I find your assessment that the problem "is in the middle" to be an assumption without proof, but it's your gear, have at it.

To me, the prospect that the vendor's comm redundancy was 'tested' for a single and maybe, just maybe, two channels/packages, and then assumed to work on three or more is highly likely and you're the unofficial Beta site. Without knowing exactly what a Link Down error is and how it's generated, I'm as suspicious of the vendor ends as of the middle.

All the basic checks which you mentioned were already performed. Both DCS & package vendors have done all the checks related to configuration, software versions and settings. I already have validated all the checks with other members. They both are still here and trying to find the root cause.

"Link Down" error is generated on DCS when DCS COM module sends request but does not receive response within given amount of time.When checked on the switch diagnostics, we found some packets collisions but switches in other two cabinets show similar behavior (collisions reported sometime) but do not result in Link Down Error. Response time parameter has been increased on DCS side and Package side but result the same random error.

The reason for me to post here is to get some out-of-box suggestions, opinions which might have been overlooked during checks.

<...They both are still here and trying to find the root cause...>

Then keep them there until they do, and please post back here with the findings, as these may well be of value to other forum users.

<unsubscribes>

Maybe the wiring is longer on one, or possibly some noise interference...loose or defective wire someplace maybe...

It sounds like you bought the wrong switches. Deterministic or redundant networks do not work well with cheap off the shelf switches. Those devices are expecting a fixed packet length. But if you are sending larger packets, those devices have to break apart your data, then transmit it, then reassemble it on the other end (which takes too long for a deterministic network).

Is there a microwave oven in the break room right next to the cable tray?

I've seen this issue before, one of the sender or receiver units on the way out. All tested fine on several occasions. Have you tried swapping them out with known good ones? In our case, the issue was not resolved until the component gave up, and a replacement was fitted.

Swap two of the package Modbus TCP/IP modules. If the problem moves it’s the module, if it doesn’t move it’s the cable.

You wrote: "The package internally generating Modbus RTU data"

Do the three packages communicate via a EIA-485 serial cable to a single switch (Modbus router) or does each package have its own switch to raise the RTU to TCP/IP?

Did you monitor the RTU signal at the problem package and analyze the signal at the time of a failure?

Does the EIA-485 link use 2-conductor cable or 3-conductor? If it's 2-conductor and only one router, are all three packages connected to same power xformer? Are their grounds equal?

Zvi

Have you tried using a packet sniffer to trap the address which does not respond before timeout? It might also point to a malformed request or a request to an unknown receiver. The proper tools in the hands of a knowledgeable tech will point to the problem much quicker than random guessing.