Is this a college project, a bit of DIY, or what?

Given:

1 [V] (RMS or peak-to-peak?) sine wave at 10 [kHz]
10 [microseconds] between samples


Please supply the following parameters:

MOSFET Rdson value [ohms]
S&H capacitor C value [microfarads]
Buffer amplifier input impedance and/or bias current [ohms] [nanoamps]

I think your ill defined noisy harmonics comes from the charge injection that your solid state switches generate. Depending on the topology and component selection, with sufficient time settling after acquisition this error will either be proportional to the voltage to be measured or a nominal constant value. In either scenario though this should be a predictable offset that can be compensated for, once understood.

Obviously though this relies on the idea that your noisy harmonics are harmonics of your sampling frequency. If you mean something else then please clarify what these harmonic frequencies are multiples of what anticipated frequency. Also don't forget to identify where you measure these signals; after your input buffer, on your storage capacitor, after the storage capacitor buffer. Lastly identify what you used to measure these harmonics; analog oscilloscope, digital oscilloscope with FFT capabilities, dynamic signal analyzer, analysis of data set from A/D converter.