I finally got around to replacing the bearings. I'm halfway done and I don't think I can go further. I can't remove the controller, as the screws are rusted in pretty good. I would strip the head trying to get it out, which isn't a big deal, as I have a screw extractor. However, if they won't budge, there is no point and trying that either. I was able to get one screw out, so I can take that and get a replacement. All I did for the freewheel side was remove the screws, and gently pry it open with a flat head. Then slide the spacers off, and then I was to simply slide the cover off. With a bit of fiddling around, I got the bearing out. I think I just stuck my finger in and it came out. The replacement one took little more effort. I don't have a rubber hammer, so I folded up some rubber and gently taped the bearing into place.
As for the otherside. It's not coming out. I whacked it pretty good with a piece of wood and hammer, and it moved outwards little bit, but not enough to get my flat in. I also had the side effect of it sliding down. I noticed when I tried to put the screws back in, they were not lined up. Thankfully, I knew which direction I needed to go, as I marked the wheel with a permanent marker. I grabbed a microfiber cloth and tried to turn the cover back, but went the wrong direction. So it ended up sliding in the wrong direction. I was able to turn it mostly back. I then took my flat head and gently tapped on the heatsink of the controller, until everything lined up.
I put everything back on the bike, gave it a quick test spin to make sure I didn't damaged the wires inside, and everything seems to be ok. From the looks of it, the large bearing is rubber sealed ? It might possible to access the seal and bearings, from the freewheel side, without needing to remove it.
EDIT: Couldn't I just tap out the cover from the freewheel side ? Gently tap at it with a piece of wood or something, so as to not damage the inside of the cover ??