I don't how much difference this will make, but did you set the
Number of pole pairs to suit the pole pairs in the hubmotor or is it still set at
4 as per the 3kW motor?
I don't know how many magnet pairs your particular hubmotor has, but you can use a voltmeter to measure the pulses from a single hall sensor signal wire in one complete revolution of the wheel.
If the voltmeter alternates between high and low
(5V~0V) 24 times per revolution, your motor will have 24 pole pairs.
The GM hubmotors contain 10, 20, 23, 24 or 28 pole pairs
(20, 40, 46, 48 or 56 magnets) determined by the motor's diameter.
Have you checked that the alterations to your parameter settings have actually been saved correctly by the controller by reloading the settings from the controller to make sure they have changed?
Will the controller accept a lower starting phase current than 10A or is that as low as it will go?
Have you tried lowering the
Battery drawn current limit (A) to 30~40 Amps?
Unfortunately, it may simply be that the stator windings in the hubmotor are not compatible with the vector controller's sensing requirements.
I encountered similar problems with speed regulation on my Magic Pie III when I installed a Magic Pie 4 vector controller. I eventually gave up because I was unable to get the motor to run in an acceptable manner.
Alan