Functioning of AVR on Heavily Unbalanced System?

Operation of a generator with a phase open is not acceptable and utility grade generators have protective relaying to trip the unit within a relatively short period.

Basically the fact that a line conductor is open doesn't mean that no current is flowing in the windings, in fact there are very high negative sequence currents in all three windings, and these double frequency currents induce huge amounts of eddy currents in the stator laminations, rotor surface, and surrounding metal parts. These undesirable currents cause intense heating that rapidly breaks down insulating materials shortening the life of the generator.

A thorough discussion of the reasons require an understanding of the internals of a generator and symmetrical component analysis, so here's the short version from a NERC (North American Electric Reliability Council) slide on the subject, from slide 6:

http://www.nerc.com/docs/pc/spctf/Phoenix%20Prot%20Coord%20Workshop_Session%207_Device%2032%2046%20et%20al_20100317.pdf

"...The Need for Negative Phase Sequence or
Unbalanced Overcurrent Protection - Function 46


"There are a number of system conditions that may cause unbalanced three-phase currents in a generator. The most common causes are:

•system asymmetries (untransposed lines),
•unbalanced loads,
•unbalanced system faults, and
•open phases.

These system conditions produce negative-phase-sequence components of current that induce a double-frequency current in the surface of the rotor, the retaining rings, the slot wedges, and to a smaller degree, in the field winding.
These rotor currents may cause high and possibly dangerous temperatures in a very short time.

The ability of a generator to accommodate unbalanced currents is specified by IEEE Std C50.12, IEEE Std C50.13, and IEC 60034-1 in terms of negative- sequence current (I2 ). This guide specifies the continuous I2 capability of a generator and the short time capability of a generator, specified in terms I22t=K, as shown in Figure 4-39 (curve drawn using data from IEEE Std C50.13)."

IEEE C37.102-2006 - Guide for AC Generator Protection, Section 4.5.2 (emphasis added)..."

Sorry for the jargon but that's how utility engineers talk, so whenever you come across a phrase you've never seen before just Google until it starts to make sense. Enjoy the trip!