Hi and
to the forum.
It sounds as if it might be a hall sensor or wiring problem to me.
Presumably the wheel still turns smoothly without power?
Check that all the wires are correctly inserted into their connectors. If that doesn't help I would suggest unplugging the hall sensor connector from the controller and see if the motor then runs smoothly, but you will still need to have the wheel spinning before the throttle will work.
If it runs smoothly with the hall sensor plug disconnected then I would say it could be a failed hall sensor or broken wire.
If it is still vibrating badly in sensorless mode it is likely to be a poor connection on one of the three phase wires, a controller fault or possibly a fault with the motor windings.
Please let us know what you find.
Alan