ref

restricted earth fault is a differrential scheme applied to a particular winding of a transformer or to a generator winding. on the phase side are three CT's connected to measure the resultant (residual) current and compare this against the CT in the neutral leg. the idea, simply put, is that if the phase side currents are balanced their resultant will be 0A and an imbalanced situation will result in the imbalance being the output of the connected phase CT's and this imbalance will also appear in the neutral CT. a fault within the winding will mean the two quantities (phase resultant & neutral) will not balance hence a relay trip but will be stable for external faults.

the scheme can be unstable i.e. trip for heavy external faults because of mismatch in CT's causing some to saturate earlier and this is minimised either by the use of a biased scheme or high impedance scheme as described in the manual mentioned above by Sparky... excellent reference material.