Instrument transformers can and do work above their rated values, but the problem is core saturation: putting more magnetic lines of flux through the CT core than it was designed for. This will cause all kinds of measurement errors if it occurs.
Bear in mind that you'll want the CT to be able to handle transients without saturating, which is why the additional "headroom" beyond 600 amps exists to begin with. Exceeding 600 amps RMS compromises this margin, which is why it should not be done.
The CT manufacturer should be able to give you specs describing the saturation limits of the transformer, but I think the best solution is to upgrade to a larger CT if you really need more measurement range.
__________________
They call me "lightning" when wielding a hammer, because I never strike twice in the same place