COP Improvment Meaning

Then how do you know what the COP improvement is?

There was a unit called a ClimaCheck installed that measures COP

Usually COP is understood as heat energy moved ÷ energy consumed, in matching units. Without measuring kw (or amps, assuming constant voltage), you don't get a direct measurement of energy consumed. I take it the ClimaCheck scheme measures temperatures and pressures, looks up or calculates what the matching energy usage should be, and then displays this imputed COP. How close this is to the real COP depends on how accurate the sensors and programming are.

A big YES, you are definetely seeing savings. Improvement in COP is direct indicater of improvement in energy efficiency, beacause COP is nothing but 'Refrigeration Effect' acheived per unit of power spent for compession.

If its a "big Yes", how should I calculate the .10 or 3% improvement to mean something in laymens terms?

While agreeing with Mr Aerodeo, I can suggest following simple way to calculate the improvement in laymans terms, considering there are no other changes:

COP = Refrigerating Effect (RE) / Power for Compression (P)

(both shall be in same units, either in kW or in heat units like KCal or kJ)

For previous COP, if I assume 352 kW is RE, then P shall be 100 kW to get COP=3.52

Later COP increased to 3.62 (2.84% increase). Considering RE is same i.e. 352 kW, now revised power requirement shall be 352/3.62 = 97.24 kW which is 2.76% lower than earlier. Hope it is clear.

In short, YES, you are seeing energy savings, though how much depends on environmental conditions and run time. ClimaCheck calculates COP based on a number of inputs, but, most relevantly to you, it does indeed measure the line voltage and current draw (to calculate power consumption) of the compressor.

If you can read the COP within ClimaCheck, then you can certainly access your kw consumption. Or you can contact ClimaCheck and they will be able to tell you exactly how your system is performing.

Assuming the cooling capacity is the same (not a trivial assumption) then you could use the formula for COP: COP=Qevap/Pcomp do derive the improvement in Pcomp

i.e. : Pcomp = XBtus/0.1

Please note that this is a gross oversimplification and I would still advocate for contacting ClimaCheck so they can tell you how much extra cooling you're getting or how much less energy you're using or (far more likely) some combination of the two.