UPS Issue

Wow that must have been one hellova short, what caused this?...or are you still trying to figure that out?...If I was just to make a wild guess, I would say a possible cross phase short at the output main lines....

There was absolutely no short, arc, unintended ground, etc. anywhere in the facility.

This is one of the main reasons why I suspect an internal fault, whether hardware or software, in the UPS.

im still looking for similar experiences from the crowd...

My guess would be a cross phase at the unit discharge....or as you say internal....

More than likely the 3 phase bridge control timing circuit(s) failed to synchronize causing a phase-to-phase fault.

Most 3 phase UPS units utilize Phase Locked Loop timing (PLL) to synchronize the firing circuits and control/shape the reconstituted waveforms.

This circuitry is very sensitive to noise and ground fault generated current which can cause "ghost" faults to occur until the cause is identified and removed.

As far as the error log history file being deleted by the technician goes:

Some equipment error logs have a limited amount of file storage and will not allow new DATA to be stored if the register becomes full which can cause important DATA to be lost if the register is not being cleaned regularly.

Other equipment error logs are FIFO (First-In-First-Out) storage registers with the oldest recorded event DATA automatically being purged from the record as new DATA is entered/created.

If the machine has been thoroughly checked and no apparent fault(s) are evident, the technician will often clear the records before repeating a test so that only new, valid DATA is present for troubleshooting purposes.

I am unsure as to how APC's modules do their work, as the schematics and/or functional block diagrams are impossible to get a hold of. Apparently only franchisees, contractors, or company personnel have any access, purchased or leased.

Their parent seems to have a lock on a very large portion of the industry. Most of the materials in the power modules I inspected are protected by European copyright and mfd. over there.

Two entries from the log:

'Critical' , 'Output voltage waveform error detected','IO enclosure level 2'

This was explained as meaning a fault on phase-2 of the output. However, a similar error later begs the question: did the programmer just use unfortunate language or is APC somehow manufacturing phases on the fly?

Later:

'Power module synchronization error','Power enclosure 1 level 1'

'Power module synchronization error','Power enclosure 1 level 2'

'Power module synchronization error','Power enclosure 1 level 3'

'Power module synchronization error','Power enclosure 1 level 4'

'Power module synchronization error','Power enclosure 1 level 5'

weird...

The error log entries you supply support my earlier statements in that unit enclosure 1 failed to synchronize, it faulted phase-to-phase, causing an overload on the primary input supply and the resultant output waveform was/is distorted.

We have a very large quantity of UPS units and we have been through your scenario numerous times.

In many cases we have identified the root-cause as being: the initializing sequence by an operator/technician as described in the IOM was not followed correctly. Proper sequence of input and output circuit breaker closure and/or system start initialization is critical on many/most UPS units.

In a few cases the fault lies in the control circuitry and requires control module(s) replacement by the factory technician.

Funny, APC pointed the blame on operator error for the second shutdown...due to operator error.

Two problems I have with that:

1) The second shutdown was identical in scope to the first...i.e., no operator intervention. The first time saw a period of three days before the event. The second happened about 1/2 hour after the system was brought back up.

2) There is no room for error in re-starting the system, in that the operator must follow the on-screen prompts (5 to be exact) to restart the system. If conditions are not met according to system expectations (breakers closed, breakers open) then the system will not start.

okay, okay...three things:

3) I was the operator. I may be taking it personally.

OOPS!

It may be due to a loose connection or a bad solder joint on a control card socket or control card component socket.

Or; A due to bad terminal end on a wire or wires. (B.O. Pressure fitting/crimp.)

It is very possible that the UPS filter network is not filtering all of the noise out of the control circuitry and you might want to use an oscilloscope to probe the input, output, and control circuit test points to identify if there are any issues.

Good luck and stay safe.

What concerns me about this is:

1. the UPS is purchased to provided as its name says, uninterruptible power for critical applications. it failed its intend use, and was effectively useless in action.

2. we have no solutions or lessons learned.

3. could / should a different manufacturer be considered, or a different type UPS? what should we - UPS purchasers - do diffferently?