Your Greatest Troubleshooting Feat

Nice one Milo. I've also been puzzled by people who collect data and then don't use it properly.

Personally I like to draw diagrams. Often, just as a sketch (clearly showing the various relationships) is being finished the problem becomes apparent and can be solved (which shows doing the drawing was unnecessary).

I agree. Collecting data without analyzing the results is nutty to me. Personally, I will put together histograms and do a Pareto analysis, or time-series evaluations or whatever seems appropriate. Collecting data if you dont plan to use it is not cost effective.

Sometimes the most vexing problems, won't be intermittent.

Many years back, I was involved in a very high gain audio bandwidth amplifier. Myself and the engineer in charge knew that noise would be a problem with such a high gain amplifier. We carefully decoupled every amplifier stage from the presumed 60 hertz residing on the power supply rails. Calculated the effect of the Johnson noise of each input resistor and after several iterations, constructed what we believed a suitable amplifier. Our noise spectral analysis showed exactly the predicted noise energy of the front end resistors with a little shot noise at the lowest frequencies from the amplifiers' bias currents. Then we added the RG58 coaxial input cable and up popped a 60 hertz spike some +40dB above the noise floor. Logistics of our design required some 30 meters of coaxial cable from transducer (ion chamber) to amplifier but our test was just adding the cable, not the ion chamber and interference now became a problem. Trading the cable capacitance for a fixed capacitance, removed this noise so an unexpected resonant tank circuit was not our likely problem. Somehow 60 hertz was entering a coaxial cable. We then tried various grounding and un-grounding schemes to connect the shield and center conductor one at a time to earth. None of these approaches worked in our electromagnetically pristine environment of our lab. We were convinced that the high EMI noise environment of the research facility would render this approach unusable until our room mate slid his chair. This was not a quite glide, but a startling grinding sound of a steel chair sticking to a recently waxed floor and the amplifier picked it up. Increasing the oscilloscope gain we noticed our speech riding on top of the 60 hertz noise. It was then that we noticed the air conditioner in the room always hummed. A flick of the air conditioner power and the 60 hertz spike dropped into the expected noise floor. We had made a condenser microphone out of the RG58 input cable. Now that the real mechanism was found, the rest was simple iterative research to find a suitable cable.

I've rated this thread as a 5 star, for it is a great question.

Milo's story reminded me of Technical organizational tools used to make movies.

The First Assistant Director uses a "Strip Sheet" that works a bit like a slide rule for complex scheduling.

The Script Supervisor keeps track of what actually happened.

Much of my working life has had a minimum of routine, and often that meant for me that I'd just go from one problem to the next.

My last job, I kept for too long, but it wasn't a happy trip, it was a money trip.

Boss had an equipment shed where I was supposed to go in the mornings and collect tools and other supplies to distribute to often three different sites. There was no bathroom. He told me not to use the bathroom in his house on the property.

Intermittent problems are fun!

It's the everyday artificially maintained ones that wear you down.

The "people element" always turns all.

Things learned from life and death situations do tend to sharpen the lessons.

Just as in War, we sometimes just need more power, better weapons, for the moment, but everyday we need to work together, and listen to each other, and respect each other, like the time my ground went hot because the box tied into wasn't grounded as we normally would expect and all phase legs were maxed amp draw.

My electrician and I and a couple of other guys on the crew had to have a meeting.

And then I remember the boss wanted to go for miles to find water for the masons, and I pointed out that there was a pond and a creek in plain sight 100 yards from where we stood.

No wonder he wanted to fire me.

"Intermittent problems are fun!"

Not when they only occur on your customers site when you're not there, they aren't.

For me the most technically challenging thing is not the electrical, mechanical or logistical problems. Its simply getting the people in charge to take a serious look at what is often a simple root cause of an on going problem.

Convincing the know it all hire up decision makers that the "Its too simple to be the problem" is in fact the problem, is often the greatest trouble shooting I have ever done.

I have seen $50,000 motors get taken out and sent off for rewinding because the $800 master breaker that ran it burned up one leg and was causing a single phasing situation. I have also seen diesel engines get full over hauls because a single fuel injector clogged and caused a bad miss. I have also seen vehicles get overhauls simply because the owner couldn't believe a spark plug wire could go bad. And the bad miss and backfiring must be from burned valves or a cracked piston or head.

I could list several million dollars worth of perfectly good equipment that was rebuilt or trashed because someone in charge could not accept that a simple item was the actual root cause of the problem and not something far more serious and expensive.

Trouble shooting people is my greatest challenge!

"I have also seen vehicles get overhauls simply because the owner couldn't believe a spark plug wire could go bad."

Curious, why didn't you just replace the wire?

"Sometimes you need to step back and look at the issue from a different perspective."

This is what caught my attention 35 years ago, while troubleshooting electronic systems.

If you are troubleshooting a problem, and you keep coming back to the same point - STOP! Go take a break, get a cup of coffee, whatever. But flush the entire problem out of your brain. If your running around in circles, it is a good bet that somewhere in that chain, you have assumed a signal was bad that wasn't, or assumed a signal was good that wasn't. Unfortunately, once you make that assumption, you are more likely to make it again - and again...

The only way to break that loop is to start fresh, or get someone else to help (and keep your assumptions to yourself).

Agreed, enlist a fresh pair of eyes and say as little as possible.

Years ago (when VCRs had a lot more serviceable stuff inside) I was vexed at how to set the timing without access to an oscilloscope. Then I thought about strobes and set the open box to PLAY in front of the TV. By looking at the reflection of the screen on the video play head I was able to get the timing set perfectly by adjusting for a "stationary" video head.

Nice trick!

Some great posts on this thread.
It's having the experience to know you have the experience...listen to the little voice telling you 'No that wasn't right...think again, or double check it'
Yeah, I've changed a clutch when a wheel was just spinning on the half shaft. I saw the propshaft was turning but ignored the little voice.

Del

Not a great tough bit of troubleshooting, but simple elegant solution.
I'd built this prototype hand held Spirometer (Measures respiratory function) It worked nicely, but the output would drift if you squeezed or twisted it as you held it. It had a hybrid amp, printed resistors on a ceramic substrate all DC coupled, as a human breath is very low frquency.
It was fairly obvious the problem was the amplifier, it had legs down each side of the substrate like a big IC. The printed resistors were acting like strain gauges!
I solved the problem by thinking about how you stand on a moving train or bus...you have your knees bent! I made a jig which put a nice zig zag bend in all the legs.
Problem disapeared.
That job never got into production tho' . Some big boss told purchasing not to buy stuff for the pre-production batch, but neglected to tell the Chief designer/project manger (me).
I left not long after that...All the top management playing politics and jostling for position.

Del

There I was two weeks on my new job and I am tasked with my very first "it won't preset" issue.

The day crew changed out a bad motor and left the clean-up for the next guys. Those guys replaced all of the guards, reset all the limit and safety switches then tried to preset the panel. No luck. The panel would not preset. Mechanics and electronic technicians worked on it for hours with on success.

My boss tells me to "go take a look" and he'll send someone to help me in a couple of hours. I was nervous. I went to the panel ( I had to borrow a key and a meter, I was that new) and opened it up. I looked in the panel (did I tell you that I had NEVER done anything like this before in my life? I mean this is 480V in this panel!) and was astonished by the number of wires and components in there.

The first thing I do is check the disconnect. Ok, 480V going in,great. Next I pick the 3 main panel fuses out of the bottom of the panel and test them for continuity,also good. I then put the fuses into their holders,turn on the disconnect and press the preset button and... it presets. THE END.

This is a true story, and boy were those other guys mad!