Hi Antti,
If the controller is overheating, you will need to improve the heatsink to enable it to dissipate the unwanted heat more efficiently.
A water cooled heat sink plate would appear to be the obvious solution in a boat:
Alternatively, a suitable air cooled heat sink
(with electric fan for improved airflow) should be able to cool the controller sufficiently to prevent overheating:
A bit more information could be beneficial to other members contemplating a similar conversion:
What size boat have you converted?
(Model, type, length, weight/displacement etc.)How many HP did the old diesel motor produce?
Are you using the 10kW motor?
What is the gear ratio between motor and propshaft?
Diameter and pitch of prop?
Top speed and average cruising speed etc.?
Hopefully you can find the time to answer these questions and post some pictures of the conversion too.
Alan