Over Voltage Ground Relay

..."look for electrical distribution systems in which one leg of a 3-phase supply powers both single-phase and 3-phase loads. You may find single phase loads not evenly balanced across the phases. Or, look for in-line reactors installed to correct imbalances. These reactors usually have taps for adjustment, and somebody may have adjusted them. Or, the imbalance they originally corrected may have shifted over time. Circuits with tapped reactors rarely stay in balance indefinitely."...

http://ecmweb.com/content/basics-voltage-imbalance

"The phase to phase voltages are consistent with each other. The phase to neutral voltages are not constant. Our system is ungrouded system."

If you have an UNGROUNDED system, you have, by definition, no ground referenced neutral. You cannot use this for single phase L-N loads. Any phase to ground voltage measurement is affected by the capacititance to ground, which will vary.

Please hire a QUALIFIED electrical engineer...

In my opinion, 0.3 sec for 10 V it is very low. The system may function at sqrt(3)*3.8=6.6 kV for 1 min- the cables, at least. 20 V and 1 sec it seems to me be more reasonable.

If the neutral isn't grounded anywhere, then why worry about the neutral to ground voltage?

As a second opinion:

why do you need a neutral since your transformer secondary windings are delta connected?

The 4th conductor could be a grounding wire[ungrounded?].

However, in your case, for 6.282 kV [phase_to_phase] it has to be only 3.627 kV [symmetric phase_to_ground]. In no case you cannot measure at the same time 4.096 kV maximum and 3.876 kV minimum.

In my opinion, the potential transformer ratio is different: for between phases [R-Y,Y-B,B-R] the PT ratio is 6.6kV/110 V and for phase_to_ground PT the ratio is 7.2kV/110 V.

Sorry. I meant[vice versa]:

for between phases [R-Y,Y-B,B-R] the PT ratio is 7.2kV/110 V and for phase_to_ground PT the ratio is 6.6kV/110 V. However, you have to check the PT ratio any way.

The PT ratio is 6kV/0.11kV. The Potential transformer connected is of star-star (grounded at both sides). The Phase voltage measured at the secondary of the potential transformer is constantly fluctuating but the line voltages are normal. Is it the phenomena of ferroresonance?