Wisdom Through Troubleshooting Circuits

We had a system at a local runway that probably fits in here.

The controller for a large fixed gas extinguishing system was in a trouble state on start up. The system had to be on line ASAP as it protected the runway lights backup power for a new runway at Orlando International Airport, and this new runway was being placed in operation.

T=2 days until runway ops.

First technician on scene noticed the problem seemed to be the power supply. He ordered a replacement and left the site.

T=1 day until runway ops.

Second technician goes to site, replaces power supply and declares it bad out of the box. Calls in to order another power supply which would get on site the day of runway ops.

I put the brakes on the project and went to the site. It turns out that the power transformer which supplies power to the power supply board had been improperly tapped at the factory. I corrected the tap and the problem disappeared.

The simplest possible problem was overlooked. The assumption that "if the factory did it, it HAD to be good" almost gave us a black eye.

One thing which is very hard to explain to technicians is that a component was not necessarily faulty just because when they changed it for a new one the problem went away. There could easily be a fundamental problem which is sometimes made worse by the tolerance of components. This is especially true of interface problems.

I seem to have gone through a long period of my life ironing out problems with RS422/RS485 signalling. This is just a way to convert TTL (or other logic technologies) into differential signals to transmit over a cable so that noise causes less problems. Nearly all of the problems which the test guys on the shop floor thought they had fixed by replacing an RS485 transceiver turned out to be something subtle in the way the signalling was turned on and off, or, the way the address of the receiving component got "phantomed". But one of the best was a pure test problem.

The signalling is often done with a clock pair and data pair: the data changes at the negative going clock edge and is clocked in at the far end on the positive going clock edge:-

In the setup which had the problem the test equipment had one set of clock and data pairs and the Unit Under Test (UUT) had another set. The test guys knew that they had some "really good" pairs of transceivers that always worked in the UUT and some that were "compatible" with some and not with others.

As soon as I looked at the clock and data at the receiving end of the link I knew what the problem was:-

The test engineer who built the test equipment had swapped the neg. and pos. leads of the clock differential pair. The positive clock edge now coincided with the point at which the data was changing. The "really good" pairs of transceivers were ones which had very different propagation delays through them so that the clock edge always fell just after or just before the data change. When the combined propagation delay through the the transceivers at both ends of the link added up to the same delay the clock was sometimes received before the data change and sometimes after so the data got really badly scrambled.

Trying to figure out what was going on for the test technicians by swapping out components made for an unfathomable puzzle. Looking at the signals with a scope and it was easy.

That is really an interesting problem. Had to give you a GA (good answer for those of you who don't know). I would have pulled out most of my hair trying to figure out that one!