I've always wondered if potting the coils would be worthwhile (or thermally detrimental).
I would not recommend potting the stator. I have worked quite a lot with potting professional power electronics. If you plan on potting the entire stator assembly you need to be very careful. You will need to calculate thermal conductivity, heat capacity, thermal expansion. You will also need to consider other important aspects like surface treatment (priming, etching), curing temperature, metering ingredients, mixing and removal of air bubbles and moisture. You will also need to make a mould of some kind to keep the potting compound in place during curing.
If you do a mistake in the potting procedure you will likely get delamination between wire and compound due to lack of bonding and possibly thermal expansion differences. Delamination causes hotspots since you will have no air circulation and no thermal conductivity - only a wire that gets warmer and warmer.
Instead I would recommend coating with impregnant - a thin conformal coating specially made for coils of magnet wire. This type of treatment is very good, as can be read from this document from Toshiba (it is geared towards higher power than ebikes, but it contains relevant information):
http://www.toshont.com/ag/mtrldesign/AG38EM.pdfI have had two rotating machines fail due to lack of coating. One was a car alternator, the other was a fan in my clothes dryer. In both cases the insulation of the magnet wire was worn off causing short to ground. This is unlikely to happen when the stator is coated.
The optimal coating is a layer strong enough to stabilise the wire from breaking loose from the stator, but so thin that heat will not build up. Such coating is normally applied in special vacuum/pressure vessels (VPI) using epoxy or polyester impregnation and heat curing. Motor coatings are mechanically stronger than transformer coatings since they are used on wound rotors. BLDC stators can use either transformer or motor coating, transformer coating would probably give slightly better cooling.
I would recommend that you bring your stator to a professional motor and/or transformer rebuild shop. They would normally have at least a vacuum vessel, if not a full VPI system. That is sufficient for the rather thick magnet wire in ebike motors. With impregnant and the infrastruture to apply the varnish using vacuum pumps and other equipment they do an excellent job. You should check that the curing temperature is compatible with the hall sensors, if not you must remove the sensor temporarily.
You can do this coating yourself but it is messy, and requires vacuum and experience for good results, and you get only one try. Correctly done, your stator and winding will outlast everything else on your ebike.
Here are some more information about motor coil impregnation:
http://findarticles.com/p/articles/mi_qa3726/is_199902/ai_n8849899/http://www.insulatingvarnish.com/main201.htmThe best would be to have Golden Motor impregnate the stators when bulding motors. I hoped they were coated but found out it was not the case. Not on our 250W/36V minimotor.
