Login | Register


Electrical Components

The Electrical Components Blog is the place for conversation and discussion about power generation, distribution and protection; connectors and relays; sensors, RFID & passive components; and magnetics and transformers. Here, you'll find everything from application ideas, to news and industry trends, to hot topics and cutting edge innovations. This blog is inspired by the Electrical Components newsletter from GlobalSpec, which you can subscribe to here.

Previous in Blog: Supercapacitors: A New Use for Paper?   Next in Blog: Does RFID Scare You?
Close

Comments Format:






Close

Subscribe to Discussion:

CR4 allows you to "subscribe" to a discussion
so that you can be notified of new comments to
the discussion via email.

Close

Rating Vote:







16 comments

Your Greatest Troubleshooting Feat

Posted July 28, 2009 7:39 AM

Troubleshooting an intermittent problem challenges your test creativity. Sometimes you need to step back and look at the issue from a different perspective. But even the toughest problems eventually get solved with a combination of persistence and technical acumen. If you've been working in the tech industry for any length of time, you certainly have some troubleshooting war stories of your own. What's the toughest technical problem you ever solved, and what was the key that unlocked your solution?

The preceding article is a "sneak peek" from Electrical Components, a newsletter from GlobalSpec. To stay up-to-date and informed on industry trends, products, and technologies, subscribe to Electrical Components today.


Interested in this topic? By joining CR4 you can "subscribe" to
this discussion and receive notification when new comments are added.

Comments rated to be Good Answers:

These comments received enough positive ratings to make them "good answers".

Comments rated to be "almost" Good Answers:

Check out these comments that don't yet have enough votes to be "official" good answers and, if you agree with them, rate them!
2
Guru

Join Date: Jul 2006
Location: "Dancing over the abyss."
Posts: 3202
Good Answers: 163
#1

Re: Your Greatest Troubleshooting Feat

07/28/2009 9:35 PM

Chem lab at steel plant. 2meter straight! light path 30 year old Baird optical emission spectrometer. Analog and a handfull of "pixie tube" clocks for readout. Air conditioned room. Great standards. 20 year seniority operator. Flakey readings. I had been brought in from engineering to become lab supervisor a couple weeks earlier. What the heck could I know?

This spectrometer analyzed all pipe samples for shipment, All billet samples for bar mills rolling, and all customer claim samples. 3 shifts a day . 7 days week.

MONEY ON THE LINE- an entire steel plants worth of production. every shift.

Dept superintendent supervised operator and electricians substituting of clocks and channel drawers (electronics) for each elemental channel. They struggled with it for a week and couldnt get it stable. I told them it was optical problem. ( i had watched them swap out good clocks and bad clocks enough to get a strong feeling that the good clocks were good, just not in certain spots.) They just sniffed at the new guy. and kept sending the samples and the operator up to the BOF lab at other end of mill,(about 2 miles west) to run them between heat analysis for steel melting.

I came in on afternoon turn following week and took all their scrap paper notes and put them on a big piece of green ledger paper (what accountants used before excel). by channel. by clock, by drawer.

When I was finished I could tell them which clocks worked, which drawers worked, and which element channels were problematic regardless of drawer or clock or combo- that they clearly needed to check optics.

"When you put your data into rows and columns, it will talk to you!"-milo

They opened it up and as they started to remove the photomultiplier tubes, they noticed that the mirrors glued to the tubes and racks were falling off when they touched the tube. 30 years and the glue had dried out and was essentially failed.

Four lessons:

1) Authority and rank don't make you smart, it just makes you the boss.

2) Seniority and time on task don't make you an expert, it just suggests familiarity.

3) Why take notes if you don't know how to make them tell the story?

4) When you put the data into rows and columns, it truly does speak to you. THIS IS THE KEY! It allows you to employ digital thinking- the problem IS/ISNOT. Clearly. Convincingly. No ANALOG thinking allowed!

Personal Benefit /accomplishment: When you can solve a million dollar process problem by sitting down with a pencil and paper, you know you have a great future in management.

Thats my best that didn't require a DOE.

milo

__________________
People say between two opposed opinions the truth lies in the middle. Not at all! Between them lies the problem, what is unseeable,eternally active life, contemplated in repose. Goethe
Good Answer (Score 2)
Power-User

Join Date: May 2007
Location: Geelong, Australia
Posts: 429
Good Answers: 19
#3
In reply to #1

Re: Your Greatest Troubleshooting Feat

07/29/2009 12:19 AM

Nice one Milo. I've also been puzzled by people who collect data and then don't use it properly.

Personally I like to draw diagrams. Often, just as a sketch (clearly showing the various relationships) is being finished the problem becomes apparent and can be solved (which shows doing the drawing was unnecessary).

__________________
If there's something you don't understand...Then a wizard did it. As heard on "The Simpsons".
Guru
Engineering Fields - Retired Engineers / Mentors - New Member

Join Date: Dec 2006
Location: Brecksville, OH
Posts: 1362
Good Answers: 17
#8
In reply to #3

Re: Your Greatest Troubleshooting Feat

07/29/2009 6:46 PM

I agree. Collecting data without analyzing the results is nutty to me. Personally, I will put together histograms and do a Pareto analysis, or time-series evaluations or whatever seems appropriate. Collecting data if you dont plan to use it is not cost effective.

__________________
"Stupidity got us into this mess, then why can't it get us out?" : Will Rogers
Guru

Join Date: Dec 2008
Location: Long Island NY
Posts: 939
Good Answers: 51
#2

Re: Your Greatest Troubleshooting Feat

07/28/2009 11:52 PM

Sometimes the most vexing problems, won't be intermittent.

Many years back, I was involved in a very high gain audio bandwidth amplifier. Myself and the engineer in charge knew that noise would be a problem with such a high gain amplifier. We carefully decoupled every amplifier stage from the presumed 60 hertz residing on the power supply rails. Calculated the effect of the Johnson noise of each input resistor and after several iterations, constructed what we believed a suitable amplifier. Our noise spectral analysis showed exactly the predicted noise energy of the front end resistors with a little shot noise at the lowest frequencies from the amplifiers' bias currents. Then we added the RG58 coaxial input cable and up popped a 60 hertz spike some +40dB above the noise floor. Logistics of our design required some 30 meters of coaxial cable from transducer (ion chamber) to amplifier but our test was just adding the cable, not the ion chamber and interference now became a problem. Trading the cable capacitance for a fixed capacitance, removed this noise so an unexpected resonant tank circuit was not our likely problem. Somehow 60 hertz was entering a coaxial cable. We then tried various grounding and un-grounding schemes to connect the shield and center conductor one at a time to earth. None of these approaches worked in our electromagnetically pristine environment of our lab. We were convinced that the high EMI noise environment of the research facility would render this approach unusable until our room mate slid his chair. This was not a quite glide, but a startling grinding sound of a steel chair sticking to a recently waxed floor and the amplifier picked it up. Increasing the oscilloscope gain we noticed our speech riding on top of the 60 hertz noise. It was then that we noticed the air conditioner in the room always hummed. A flick of the air conditioner power and the 60 hertz spike dropped into the expected noise floor. We had made a condenser microphone out of the RG58 input cable. Now that the real mechanism was found, the rest was simple iterative research to find a suitable cable.

__________________
There are 10 types of people. People who understand binary and those who don't.
Score 1 for Good Answer
Guru
Hobbies - Musician - New Member

Join Date: Oct 2007
Location: Transcendia
Posts: 1746
Good Answers: 30
#4

Re: Your Greatest Troubleshooting Feat

07/29/2009 12:23 AM

I've rated this thread as a 5 star, for it is a great question.

Milo's story reminded me of Technical organizational tools used to make movies.

The First Assistant Director uses a "Strip Sheet" that works a bit like a slide rule for complex scheduling.

The Script Supervisor keeps track of what actually happened.

Much of my working life has had a minimum of routine, and often that meant for me that I'd just go from one problem to the next.

My last job, I kept for too long, but it wasn't a happy trip, it was a money trip.

Boss had an equipment shed where I was supposed to go in the mornings and collect tools and other supplies to distribute to often three different sites. There was no bathroom. He told me not to use the bathroom in his house on the property.

Intermittent problems are fun!

It's the everyday artificially maintained ones that wear you down.

The "people element" always turns all.

Things learned from life and death situations do tend to sharpen the lessons.

Just as in War, we sometimes just need more power, better weapons, for the moment, but everyday we need to work together, and listen to each other, and respect each other, like the time my ground went hot because the box tied into wasn't grounded as we normally would expect and all phase legs were maxed amp draw.

My electrician and I and a couple of other guys on the crew had to have a meeting.

And then I remember the boss wanted to go for miles to find water for the masons, and I pointed out that there was a pond and a creek in plain sight 100 yards from where we stood.

No wonder he wanted to fire me.

__________________
You don't get wise because you got old, you get old because you were wise.
Guru

Join Date: Sep 2007
Location: Defreestville, NY
Posts: 642
Good Answers: 48
#13
In reply to #4

Re: Your Greatest Troubleshooting Feat

07/30/2009 8:25 PM

"Intermittent problems are fun!"

Not when they only occur on your customers site when you're not there, they aren't.

__________________
Enjoy every sandwich - Warren Zevon
Guru

Join Date: Jun 2009
Location: South of Minot North Dakota
Posts: 524
Good Answers: 47
#5

Re: Your Greatest Troubleshooting Feat

07/29/2009 12:59 PM

For me the most technically challenging thing is not the electrical, mechanical or logistical problems. Its simply getting the people in charge to take a serious look at what is often a simple root cause of an on going problem.

Convincing the know it all hire up decision makers that the "Its too simple to be the problem" is in fact the problem, is often the greatest trouble shooting I have ever done.

I have seen $50,000 motors get taken out and sent off for rewinding because the $800 master breaker that ran it burned up one leg and was causing a single phasing situation. I have also seen diesel engines get full over hauls because a single fuel injector clogged and caused a bad miss. I have also seen vehicles get overhauls simply because the owner couldn't believe a spark plug wire could go bad. And the bad miss and backfiring must be from burned valves or a cracked piston or head.

I could list several million dollars worth of perfectly good equipment that was rebuilt or trashed because someone in charge could not accept that a simple item was the actual root cause of the problem and not something far more serious and expensive.

Trouble shooting people is my greatest challenge!

__________________
The best hire the best. The second best hire the third best. -- tcmtech
Guru

Join Date: Sep 2007
Location: Defreestville, NY
Posts: 642
Good Answers: 48
#15
In reply to #5

Re: Your Greatest Troubleshooting Feat

07/30/2009 8:40 PM

"I have also seen vehicles get overhauls simply because the owner couldn't believe a spark plug wire could go bad."

Curious, why didn't you just replace the wire?

__________________
Enjoy every sandwich - Warren Zevon
Guru

Join Date: Aug 2007
Location: Anyplace over 100 Deg F
Posts: 543
Good Answers: 21
#6

Re: Your Greatest Troubleshooting Feat

07/29/2009 4:06 PM

"Sometimes you need to step back and look at the issue from a different perspective."

This is what caught my attention 35 years ago, while troubleshooting electronic systems.

If you are troubleshooting a problem, and you keep coming back to the same point - STOP! Go take a break, get a cup of coffee, whatever. But flush the entire problem out of your brain. If your running around in circles, it is a good bet that somewhere in that chain, you have assumed a signal was bad that wasn't, or assumed a signal was good that wasn't. Unfortunately, once you make that assumption, you are more likely to make it again - and again...

The only way to break that loop is to start fresh, or get someone else to help (and keep your assumptions to yourself).

__________________
I could never join a club which would have me as a member - Groucho Marx
Guru

Join Date: Sep 2007
Location: Defreestville, NY
Posts: 642
Good Answers: 48
#14
In reply to #6

Re: Your Greatest Troubleshooting Feat

07/30/2009 8:27 PM

Agreed, enlist a fresh pair of eyes and say as little as possible.

__________________
Enjoy every sandwich - Warren Zevon
Associate

Join Date: May 2009
Posts: 44
Good Answers: 2
#7

Re: Your Greatest Troubleshooting Feat

07/29/2009 5:15 PM

Years ago (when VCRs had a lot more serviceable stuff inside) I was vexed at how to set the timing without access to an oscilloscope. Then I thought about strobes and set the open box to PLAY in front of the TV. By looking at the reflection of the screen on the video play head I was able to get the timing set perfectly by adjusting for a "stationary" video head.

Guru

Join Date: Sep 2007
Location: Defreestville, NY
Posts: 642
Good Answers: 48
#10
In reply to #7

Re: Your Greatest Troubleshooting Feat

07/30/2009 1:41 PM

Nice trick!

__________________
Enjoy every sandwich - Warren Zevon
Guru
Popular Science - Weaponology - New Member United Kingdom - Member - New Member

Join Date: May 2007
Location: Harlow England
Posts: 10332
Good Answers: 207
#9

Re: Your Greatest Troubleshooting Feat

07/30/2009 12:32 PM

Some great posts on this thread.
It's having the experience to know you have the experience...listen to the little voice telling you 'No that wasn't right...think again, or double check it'
Yeah, I've changed a clutch when a wheel was just spinning on the half shaft. I saw the propshaft was turning but ignored the little voice.

Del

__________________
health warning: These posts may contain traces of nut.
Guru
Popular Science - Weaponology - New Member United Kingdom - Member - New Member

Join Date: May 2007
Location: Harlow England
Posts: 10332
Good Answers: 207
#11

Re: Your Greatest Troubleshooting Feat

07/30/2009 2:43 PM

Not a great tough bit of troubleshooting, but simple elegant solution.
I'd built this prototype hand held Spirometer (Measures respiratory function) It worked nicely, but the output would drift if you squeezed or twisted it as you held it. It had a hybrid amp, printed resistors on a ceramic substrate all DC coupled, as a human breath is very low frquency.
It was fairly obvious the problem was the amplifier, it had legs down each side of the substrate like a big IC. The printed resistors were acting like strain gauges!
I solved the problem by thinking about how you stand on a moving train or bus...you have your knees bent! I made a jig which put a nice zig zag bend in all the legs.
Problem disapeared.
That job never got into production tho' . Some big boss told purchasing not to buy stuff for the pre-production batch, but neglected to tell the Chief designer/project manger (me).
I left not long after that...All the top management playing politics and jostling for position.

Del

__________________
health warning: These posts may contain traces of nut.
2
Guru
Popular Science - Weaponology - New Member United Kingdom - Member - New Member

Join Date: May 2007
Location: Harlow England
Posts: 10332
Good Answers: 207
#12

Re: Your Greatest Troubleshooting Feat

07/30/2009 2:49 PM

Oh, I took great delight in simply saying...
'Paint it black'
To a PhD guy who was griping about noise in his prototype...
Yup, his optical problem disappeared.
Multiple reflections off the machined aluminium structure were screwing it up.
Del

__________________
health warning: These posts may contain traces of nut.
Good Answer (Score 2)
Guru

Join Date: Dec 2008
Location: nj,usa
Posts: 672
Good Answers: 14
#16

Re: Your Greatest Troubleshooting Feat

08/02/2009 3:51 PM

There I was two weeks on my new job and I am tasked with my very first "it won't preset" issue.

The day crew changed out a bad motor and left the clean-up for the next guys. Those guys replaced all of the guards, reset all the limit and safety switches then tried to preset the panel. No luck. The panel would not preset. Mechanics and electronic technicians worked on it for hours with on success.

My boss tells me to "go take a look" and he'll send someone to help me in a couple of hours. I was nervous. I went to the panel ( I had to borrow a key and a meter, I was that new) and opened it up. I looked in the panel (did I tell you that I had NEVER done anything like this before in my life? I mean this is 480V in this panel!) and was astonished by the number of wires and components in there.

The first thing I do is check the disconnect. Ok, 480V going in,great. Next I pick the 3 main panel fuses out of the bottom of the panel and test them for continuity,also good. I then put the fuses into their holders,turn on the disconnect and press the preset button and... it presets. THE END.

This is a true story, and boy were those other guys mad!

__________________
" vini, vidi, dormivi" I came, I saw, I took a nap (Orbes volantes exstare )
16 comments
Interested in this topic? By joining CR4 you can "subscribe" to
this discussion and receive notification when new comments are added.

Comments rated to be Good Answers:

These comments received enough positive ratings to make them "good answers".

Comments rated to be "almost" Good Answers:

Check out these comments that don't yet have enough votes to be "official" good answers and, if you agree with them, rate them!
Copy to Clipboard

Users who posted comments:

agua_doc (1), Del the cat (3), ffej (1), Kilowatt0 (1), Milo (1), not so smart (1), redfred (1), stevem (4), tcmtech (1), Transcendian (1), ZenZeddmore (1)

Previous in Blog: Supercapacitors: A New Use for Paper?   Next in Blog: Does RFID Scare You?
You might be interested in: Borescopes, PIN Diodes, Thin-section Bearings