CR4® - The Engineer's Place for News and Discussion®


Previous in Forum: Gravitational Lensing   Next in Forum: Virtual Tour of Bletchley Park
Close
Close
Close
19 comments
Anonymous Poster #1

Modbus Communication Link Disruption At Random Times

06/07/2017 3:11 AM

Greetings CR4 community,

I have come here to get some suggestions, ideas to analyze the issue we are currently facing. We have Yokogawa Centum VP DCS getting Modbus data from one package via Modbus TCP/IP redundant links. The cable used for communication is CAT-6. The package internally generating Modbus RTU data which is sent to Westermo Lynx redundant switches converting and forwarding Modbus TCP/IP to Centum VP redundant COM modules. We have three package units working on the same principle.

Now, the issue is sometimes the Stand-by (not active) communication link in one of the unit gets disrupted randomly (Link down) and goes back to normal after some time (1 min to 2min). This link down error sometimes comes twice in a day, some times comes five times a week at irregular intervals.We have checked the cables, ports, Switches configuration, DCS COM module configuration (exact replica of other units); and found nothing unusual.

Kindly share your ideas if we can do anything else to find the root cause.

Reply
Interested in this topic? By joining CR4 you can "subscribe" to
this discussion and receive notification when new comments are added.

Comments rated to be Good Answers:

These comments received enough positive ratings to make them "good answers".

Comments rated to be "almost" Good Answers:

Check out these comments that don't yet have enough votes to be "official" good answers and, if you agree with them, rate them!
Guru
United Kingdom - Member - Indeterminate Engineering Fields - Control Engineering - New Member

Join Date: Jan 2007
Location: In the bothy, 7 chains down the line from Dodman's Lane level crossing, in the nation formerly known as Great Britain, and now disconnecting as Little England and Wales (not too sure about Wales bit, either). Kettle's on.
Posts: 26689
Good Answers: 700
#1

Re: Modbus Communication link disruption at random times

06/07/2017 3:27 AM

Please describe to the forum how the various telephone calls to the equipment suppliers have gone so far, and why these suppliers have been unable to provide information that led to solving the problem to date?

__________________
"Did you get my e-mail?" - "The biggest problem in communication is the illusion that it has taken place" - George Bernard Shaw, 1856
Reply Score 1 for Off Topic
Anonymous Poster #1
#2
In reply to #1

Re: Modbus Communication link disruption at random times

06/07/2017 3:38 AM

Both DCS and package vendors have done their checks and found nothing unusual. As mentioned in the original post, the communication is working fine with other package units. Even for the above mentioned package, link goes down randomly. So, problem lies somewhere in the middle.

Reply
Guru
United Kingdom - Member - Indeterminate Engineering Fields - Control Engineering - New Member

Join Date: Jan 2007
Location: In the bothy, 7 chains down the line from Dodman's Lane level crossing, in the nation formerly known as Great Britain, and now disconnecting as Little England and Wales (not too sure about Wales bit, either). Kettle's on.
Posts: 26689
Good Answers: 700
#3

Re: Modbus Communication link disruption at random times

06/07/2017 3:49 AM

<...This link down error sometimes comes twice in a day...> Does this correspond with the tidal minima or maxima?

__________________
"Did you get my e-mail?" - "The biggest problem in communication is the illusion that it has taken place" - George Bernard Shaw, 1856
Reply
Anonymous Poster #1
#4
In reply to #3

Re: Modbus Communication link disruption at random times

06/07/2017 4:15 AM

No. Until now, we can't co-relate this communication error frequency with any factor. We have not even found any single message or alarm in the diagnostic log of switch for that. This link down error is coming only in DCS at random times.

Reply
Guru
United Kingdom - Member - Indeterminate Engineering Fields - Control Engineering - New Member

Join Date: Jan 2007
Location: In the bothy, 7 chains down the line from Dodman's Lane level crossing, in the nation formerly known as Great Britain, and now disconnecting as Little England and Wales (not too sure about Wales bit, either). Kettle's on.
Posts: 26689
Good Answers: 700
#5

Re: Modbus Communication link disruption at random times

06/07/2017 5:00 AM

It is unlikely that the forum is going to be able to sort this problem, as it cannot be seen from here.

Hire a local qualified and experienced Electronics Engineer and take it from there.

__________________
"Did you get my e-mail?" - "The biggest problem in communication is the illusion that it has taken place" - George Bernard Shaw, 1856
Reply Score 1 for Off Topic
Guru

Join Date: Mar 2007
Location: at the beach in Florida
Posts: 19210
Good Answers: 1125
#6

Re: Modbus Communication link disruption at random times

06/07/2017 7:30 AM

Sounds like a software program issue....that unit might have a different version program, could be spyware maybe...could be a newer version board with added feature....I think I would do a temperature scan first, to look for any anomalies....eliminate any overheating issues....

__________________
Life is like riding a bicycle. To keep your balance you must keep moving. A.E.
Reply
Anonymous Poster #1
#8
In reply to #6

Re: Modbus Communication link disruption at random times

06/07/2017 8:44 AM

Thank you for the comment. Software versions, security and temperature have already been checked.

Reply
Power-User

Join Date: Nov 2009
Posts: 289
Good Answers: 55
#7

Re: Modbus Communication Link Disruption At Random Times

06/07/2017 8:31 AM

I can't help you because I have no knowledge of how redundant communications operates, but if I were tasked to do this, I would be asking these questions:

Exactly what did a DCS "vendor check" consist of? What was in their written report?
Exactly what did a package "vendor check" consist of? What was in their written report?

Who has the spreadsheet confirming firmware versions on all devices at both ends and in-between? Your review of that spreadsheet confirms that all three sets of devices are running identical firmware versions?

Exactly how is the "Link Down" error is generated? Can different conditions trip a Link Down error? What test equipment is needed to view the generation of a Link Down error? What test equipment is needed to analyze and trap traffic that creates a Link Down error? Was that equipment used in the vendor checks?

What are the details of the comm protocol for redundant comm? Where are the details published?

>Both DCS and package vendors have done their checks and found nothing unusual. As mentioned in the original post, the communication is working fine with other package units. Even for the above mentioned package, link goes down randomly. So, problem lies somewhere in the middle.

Given the firmware nature of communication links and the number of versions and revisions to versions for things like comm modules and network switches, I find your assessment that the problem "is in the middle" to be an assumption without proof, but it's your gear, have at it.

To me, the prospect that the vendor's comm redundancy was 'tested' for a single and maybe, just maybe, two channels/packages, and then assumed to work on three or more is highly likely and you're the unofficial Beta site. Without knowing exactly what a Link Down error is and how it's generated, I'm as suspicious of the vendor ends as of the middle.

Reply
Anonymous Poster #1
#9
In reply to #7

Re: Modbus Communication Link Disruption At Random Times

06/07/2017 9:42 AM

All the basic checks which you mentioned were already performed. Both DCS & package vendors have done all the checks related to configuration, software versions and settings. I already have validated all the checks with other members. They both are still here and trying to find the root cause.

"Link Down" error is generated on DCS when DCS COM module sends request but does not receive response within given amount of time.When checked on the switch diagnostics, we found some packets collisions but switches in other two cabinets show similar behavior (collisions reported sometime) but do not result in Link Down Error. Response time parameter has been increased on DCS side and Package side but result the same random error.

The reason for me to post here is to get some out-of-box suggestions, opinions which might have been overlooked during checks.

Reply
Guru
United Kingdom - Member - Indeterminate Engineering Fields - Control Engineering - New Member

Join Date: Jan 2007
Location: In the bothy, 7 chains down the line from Dodman's Lane level crossing, in the nation formerly known as Great Britain, and now disconnecting as Little England and Wales (not too sure about Wales bit, either). Kettle's on.
Posts: 26689
Good Answers: 700
#10
In reply to #9

Re: Modbus Communication Link Disruption At Random Times

06/07/2017 10:59 AM

<...They both are still here and trying to find the root cause...>

Then keep them there until they do, and please post back here with the findings, as these may well be of value to other forum users.

<unsubscribes>

__________________
"Did you get my e-mail?" - "The biggest problem in communication is the illusion that it has taken place" - George Bernard Shaw, 1856
Reply
Guru

Join Date: Mar 2007
Location: at the beach in Florida
Posts: 19210
Good Answers: 1125
#12
In reply to #9

Re: Modbus Communication Link Disruption At Random Times

06/07/2017 12:29 PM

Maybe the wiring is longer on one, or possibly some noise interference...loose or defective wire someplace maybe...

__________________
Life is like riding a bicycle. To keep your balance you must keep moving. A.E.
Reply
Guru
Popular Science - Cosmology - New Member Technical Fields - Technical Writing - New Member Engineering Fields - Energy Engineering - New Member Engineering Fields - Electrical Engineering - New Member Engineering Fields - Control Engineering - New Member Engineering Fields - Electromechanical Engineering - Old Member, New Association

Join Date: Apr 2008
Location: Lexington, KY
Posts: 1553
Good Answers: 72
#19
In reply to #9

Re: Modbus Communication Link Disruption At Random Times

06/27/2017 8:25 AM

It sounds like you bought the wrong switches. Deterministic or redundant networks do not work well with cheap off the shelf switches. Those devices are expecting a fixed packet length. But if you are sending larger packets, those devices have to break apart your data, then transmit it, then reassemble it on the other end (which takes too long for a deterministic network).

__________________
A great troubleshooting tip...."When you eliminate the impossible, whatever remains, however improbable, must be the truth." Sir Arthur Conan Doyle
Reply
2
Guru
Engineering Fields - Electrical Engineering - Been there, done that, still doing it. Engineering Fields - Control Engineering - New Member

Join Date: Dec 2008
Location: Long Island NY
Posts: 12214
Good Answers: 774
#11

Re: Modbus Communication Link Disruption At Random Times

06/07/2017 11:35 AM

Look directly at the Modbus serial communication with an intelligent oscilloscope and an intelligent mind to identify the chatter. I suspect you have a marginal communication with multiple requests for a repeat transmissions. Occasionally this high volume of chatter causes excessive data collisions that cannot be recovered. You might even have a simple miswiring across twisted pairs in just one cable producing this frequent need for re-transmissions.

__________________
"Don't disturb my circles." translation of Archimedes last words
Reply Good Answer (Score 2)
2
Power-User

Join Date: Jul 2016
Posts: 371
Good Answers: 20
#17
In reply to #11

Re: Modbus Communication Link Disruption At Random Times

06/08/2017 6:45 PM

You may find a solution with a NetDecoder TCP sniffer (physically connected in line on the affected cable) and running their software. Buy their hardware adapter, the software comes with it.

http://www.fte.com/products/NetDecoder.aspx

I've used this to detect high traffic due to poor design and resource allocation on systems that grew over time, nailed it in less than 3 days on a 6 node system, while running the plant. On another system, determined media/hardware problems based on excluding other causes, and reading (through NetDecoder) garbage packets that passed through transceivers. Every now and then, the garbage would somehow trigger something, and the comm line would shut down, like it passed a shutdown command. Could go weeks without a failure, but the upsetting factor was active all the time.

Can't say this enough, check your grounds between the different locations. AC coupled voltage (ground potential difference) measured between switches can wreak havoc on the sensing thresholds of the transceivers.

Reply Good Answer (Score 2)
Guru
Hobbies - DIY Welding - Wannabeabettawelda

Join Date: May 2007
Location: Annapolis, Maryland
Posts: 4174
Good Answers: 235
#13

Re: Modbus Communication Link Disruption At Random Times

06/07/2017 6:52 PM

Is there a microwave oven in the break room right next to the cable tray?

Reply Score 1 for Good Answer
Guru

Join Date: Jan 2007
Location: Edinburgh, Bonnie Scotland
Posts: 1174
Good Answers: 12
#14

Re: Modbus Communication Link Disruption At Random Times

06/08/2017 1:16 AM

I've seen this issue before, one of the sender or receiver units on the way out. All tested fine on several occasions. Have you tried swapping them out with known good ones? In our case, the issue was not resolved until the component gave up, and a replacement was fitted.

__________________
Madness is all in the mind
Reply Score 1 for Good Answer
Power-User

Join Date: Sep 2006
Location: Houston Texas
Posts: 289
Good Answers: 16
#15

Re: Modbus Communication Link Disruption At Random Times

06/08/2017 10:35 AM

Swap two of the package Modbus TCP/IP modules. If the problem moves it’s the module, if it doesn’t move it’s the cable.

__________________
Specializing in Dynamic Weighing Systems for Powder and Bulk Solids Handling
Reply
Power-User

Join Date: Jun 2012
Posts: 105
Good Answers: 6
#16

Re: Modbus Communication Link Disruption At Random Times

06/08/2017 1:11 PM

You wrote: "The package internally generating Modbus RTU data"

Do the three packages communicate via a EIA-485 serial cable to a single switch (Modbus router) or does each package have its own switch to raise the RTU to TCP/IP?

Did you monitor the RTU signal at the problem package and analyze the signal at the time of a failure?

Does the EIA-485 link use 2-conductor cable or 3-conductor? If it's 2-conductor and only one router, are all three packages connected to same power xformer? Are their grounds equal?

Zvi

Reply Score 1 for Good Answer
Guru
Engineering Fields - Power Engineering - New Member

Join Date: May 2007
Location: NYC metropolitan area.
Posts: 2993
Good Answers: 416
#18

Re: Modbus Communication Link Disruption At Random Times

06/10/2017 8:56 PM

Have you tried using a packet sniffer to trap the address which does not respond before timeout? It might also point to a malformed request or a request to an unknown receiver. The proper tools in the hands of a knowledgeable tech will point to the problem much quicker than random guessing.

__________________
Curious minds want to know, engineering minds get answers....
Reply
Reply to Forum Thread 19 comments
Interested in this topic? By joining CR4 you can "subscribe" to
this discussion and receive notification when new comments are added.

Comments rated to be Good Answers:

These comments received enough positive ratings to make them "good answers".

Comments rated to be "almost" Good Answers:

Check out these comments that don't yet have enough votes to be "official" good answers and, if you agree with them, rate them!
Copy to Clipboard

Users who posted comments:

Anonymous Poster (4); Brave Sir Robin (1); Carl_E (1); Delmar (1); GM1964 (1); NotUrOrdinaryJoe (1); PWSlack (4); RAMConsult (1); redfred (1); rwilliams (1); SolarEagle (2); Zvi (1)

Previous in Forum: Gravitational Lensing   Next in Forum: Virtual Tour of Bletchley Park

Advertisement