Torsional vibration calculations

Very complex since there are variable masses and distributed compliances. Let it be done by a specialist.

I've done myself such analysis in a dynamometer some years ago. Not really complicated, well, at least, not more complicated than a mass-spring system expressed in a linear way. Just keep in mind that instead of a spring coeficient regarding a force or weight, you'll have torsional stiffness regarding torque.

A good FEA software can help you with this matter. Once solved, the system will show you among the several different vibration modes the free body one for the first order, the two first order transverse modes, and then, the rotation related one.

You could be surprised on how low the first order frequencies could be, even if shaft stiffness and structural elements perform just fine.

You are right if you limit at the two inertia system with a compliance in between, it is complex if you consider as well all what happens in the engine itself.