Share Your Maintenance Nightmares!

How about a newly installed mainframe computer, on the east coast of England in the '70s that only worked on certain days of the week? It either worked or we got massive software errors.....!

It took a week, but I found out what it was...taught me a great deal.

At a satellite tracking facility we had a a newly commissioned digital link that would arbitrarily produce a high bit error rate. Everything was checked and checked and all was as it should be.

The test equipment at this stage was permanently logging the BER to see if any odd correlations could be observed. The audible error annunciator was left on once.

From time to time I would examine the data and on one night I heard a strange beep that was synchronised to my footsteps as I approached the rack. WTF!!

Step, beep, step, beep, step step step, beep beep beep.

I followed the noise to the BER tester and I could then see the errors accumulating. Test stomp produced errors. Found it!!

It turned out that a local oscillator was microphonic and it took a quiet night shift to realise it wasn't arbitrary at all.

Kudos from the boss? Not really, just another credibility maintenance miracle.

When you march to the beat of a different drum just being left alone can be reward in itself.

Funnily enough, part of the way I found my problem was with a medium wave radio sitting on the main memory as it produced tones as the memory was used....

I wrote a few lines of code to produce a loop, put the radio on and waited.

Then I started hearing a completely different tone level, with a perfect timing of coming and going, taking 4 or 5 (if I remember correctly!) seconds for a cycle.

I must here mention that it was the first day since the problem without fog.......

I was thinking and looking out the window and saw several miles away, a DEW Radar turning, and I laughed (without thinking why?) that it was "synced" with my strange "tone".

A few seconds later I realised it was the cause!!! A powerful Distant Early Warning Radar was upsetting the memory.......

We had to individually attach all the frame doors to a new "REAL" ground via good dimensioned cables, and the problem was as good as solved. The customer did a few things too, a Faraday screen I believe on that side of the computer room (its a long time ago), also grounded with heavy cables.... and that was that.....

We had no proper test software from the manufacturer in those days, simply nothing, so we always wrote us a simple routine to find a particular problem in those days (not that test software would have told us that much about this error!), and the medium wave transistor radio was often used....

A colleague used to write music as well, a routine for each note....l!!!! Star wars theme was one that I remember!!!

Cyclic interference and radar. Classic!

Seems that "signature" tone recognition for data is now a lost art.

Was a time when you could hear what the training sequence of a modem was actually doing or listening for the beat frequency of an FM data carrier from an offset tuned receiver as a spacecraft came over the horizon...

All is lost my friend, all is lost.

Where is the art?

You are so right.

I put it down to test software being made available, that turned us all into amateurs. i worked for another company that wrote "Boot-up" software for all of its products, that only cycled through when everything was perfect.....without giving any proper information back!!

We got to know later what was what.

I spent many hours writing requests for software testing that would just run the built in processor and as long as that was OK, then run tests on all the rest of the box with proper error indications......it took YEARS for the "experts" to agree and change things....

Its the difference between a logic designer (nerd) and a field engineer, and never will the twain meet!!!

New on the job and highly motivated, a customer complained of a pitch oscillation in his aircraft during climbs.

Beginning of a holiday weekend, we did a test flight; then we instrumented the aircraft and did another. Feeling all alone and out of my depth, I couldn't get another engineer or my boss on the phone.

Then we went through and bonded and cleaned and grounded and did another flight with identical results; nothing dangerous or even noticeable from the back, just a mild pitch oscillation.

About the time we are about to start ~really~ spending money (because $3,500 a flight hour and $100 per tech hour wasn't spending money) I get a page from my boss.

Him - "They all fly like that to varying degrees, quit testing it and apologize for wasting the customer's time."

Me - "You couldn't write that down somewhere?"

The customer was quite gracious about the whole thing.

From Sleepy,

A very old story and if you have heard it before......

In an engineering role in a lab I had to build an oscillator whose basic design parameter was it's frequency and long term stability.

It's stability looked fine after a short bedding in process but then I put a counter, a D/A converter and a chart recorder on the last 3 digits and left it on overnight.

Absolute disaster overnight between 2200 hrs and 0600 hrs. Scratch head, looked at everything that I could think of, got the securtity guard to check on power systems overnight and left it again. Same performance, absolutely awful overnight.

In the end I brought my sleeping bag in and watched as the transition time of 2200 hrs approached. An external street light came ON, counter and oscillator remained stable. 24;00 hrs counter went beserk as outside street light had switched OFF. All the Lab lights had been off since everyone else had gone home. Found an Anglepoise Light switched it on and pointed it at the test equipment, OK ish but the real improvement came when I pointed the light straight at the counter display! Rock solid.

I told the security Guard that this had to stay on all night and that I would remain there to check it; rolled myself up in my sleeping bag and went to sleep. Oscillator performed well overnight - as did the counter!!

This then became the long term set up (less me and my sleeping bag!) and the oscillator/counter ran for the requisite number of weeks or months like that and yards and yards of paper with a straight line was the result! Obviously it was also tested against environmental parameters and I prayed that the bulb would outlive the measurement as some idiot might have asked for the test to be run again if the bulb died overnight!

I had a discussion with the Test Equipment supplier who were most embarrased but they were and are still a very reputable firm and I am sure that they have learned that lesson albeit they did not come back to me to advise what they had done!

Sleepy

Every story deserves a GA!!!!

Oh, a water treatment plant that wouldn't backwash properly. On arrival, a quick glance at the equipment in one of the concrete boxes revealed two chambers that had a complete pair of level electrodes and the third where one of the two was missing. Gone. Down into the gloop never to return.

Tell the Client.

Raise invoice.

Bu%%er off....

Not really a story of me solving something, but I like to think I was the catalyst!

When I worked for Uncle Henry, our department was seated in two adjoining 4-seat cubicles. The temperature here was always significantly higher that around about. When you were sitting at your desk all day, it felt overly warm to the point of drowsy, but you got used to it. When you left the area, you were struck by the "chill" in the corridor area and when you came back you were hit by the wall of heat.

Being engineers, we were prohibited from touching the temperature controls, so we regularly called the maintenance team, who would come and fiddle, the temperature would drop for a bit, and then slowly return to where it had been.

In the end, I wrote an email to the head of maintenance, saying that his guys had been very helpful but that I thought there was something more fundemental wrong with the system, could he check.

The following Monday, we came in to a comfortable working environment and I to an email explaining that the maintenance team had taken a long hard look at the HVAC system, and discovered that the mixing boxes for our area had been installed, 30 years previously, upsidedown! They sorted this over the weekend and hoped all was well.

I sent a heartfelt "thank you"

Great story to get the ball rolling Andy!
Many many years ago (35-40?), there was a very slowwwww automated data logging burn in rack on production which was giving problems and locking up after long runs.
I was a bit low on work and my wise old boss knew that if he threw me in there I'd eventually sort it out.
To cut a long story short the main problem was the reading and writing onto big fat tape cassettes which were formatted into blocks. They always read and wrote to the same few blocks of tape which then started to wear and cause errors.
It was all in written in some variant of basic (HP basic?) anyhow I just organised it to cycle through to the next block on each run, stuck in nice new cassettes and the problem evaporated.
Del

I thought they made sauces. I didn't know they did coding as well...

Hi Del,

Wear on tapes was a large part of my work over my last 10 years in the company, even many tape drives write the directory at the same place each time. This wears the tapes out in a fundamentally important area. but at least the data is not lost even if the directory is unreadable, but it must be searched for the old way - reading the tape from front to back many times as the data is written maybe 30 times front to back, depending upon the type of drive.

Takes hours!!

at least with the later µcode versions, when a tape is re-used, it writes the directory in a new spot if there is any wear noted.....

The main problem is that some customer's data rates are simply far too slow for modern streaming tape drives, which almost all are nowadays....some customers simply never learn!!!