A specific o-ring requires a specifically sized and shaped groove or recess. This is to ensure that the correct amount of compression is applied to the o-ring on assembly and that it conforms to the groove correctly. The correct amount of compression ensures that the optimum sealing for the design is applied.
Somewhere, at home, I have a set of charts that delineate groove sizes for specific o-ring part numbers. We, in the machine shop I worked in at the time, used these charts for a variety of aerospace applications; face seals, shaft seals, etc. I'll see if I can find them, scan them in and make them available for this thread. The o-ring manufacturer should also be able to provide this info.
Any kind of irregularity in the groove is going to be detrimental to the sealing action that the o-ring is designed to perform. It may not be readily apparent and may not even cause an immediate leak but it does compromise the seal, particularly if the part is regularly disassembled. One small nick in the o-ring can render it useless. Twisting the o-ring on installation can also destroy its sealing action.
Also, depending on the o-ring material, vaseline (petroleum jelly) is a no-no. It can break down the material in short order. A (very) light coating of a silicone based grease is more often used. Anything else can also compromise the sealing action. The o-ring manufacturer should be consulted for proper lubrication.
Thread tape is counter-productive. O-rings are not interference threads. Putting a larger o-ring in than a design calls for will also be detrimental. O-rings are a precision sealing device and should be treated as such, not requiring "fiddling". This application almost sounds like it would be better served if teflon tape was applied to the threads and the o-ring left out all together.