fin design

Increase their surface area.

I assume that you are talking about fins to reject heat from cylinder walls, etc. since you referred to "heat engines". If the engine is not moving, add a fan to increase airflow over the fins. It has the same effect as increasing fin area by about 500%.

If the fins are vertical (horizontal cylinder) you can also install a "chimney" by building a device similar to a funnel (but rectangular- with a "long" neck) over the fins to create a constant airflow with hot "exhaust" that will constantly pull in the cooling air and maintain adequate velocity to improve heat transfer.

Paint them black.

Blow air over them.

If increasing their surface area by increasing their number, there will come a trade-off of diminishing air circulation between closer-spaced fins.

Here's a fun applet intended for electronics heat sink, but you can learn a lot pretty easily.

Forced ventilation helps.