Hmmm
Is this the one?
If so, its not too difficult. I had the same on my Cyclops bike, I just unscrewed the button allen bolt and it slid off the bar.
If your grips are holding you back, make a bucket of steaming hot soapy water and soak the bars in there until you can pull them off.
Either that or cut them off like I did
Also, the wire entry for the cruise/horn button does not travel through the O shape, so you could cut the O shape (that fits on the handlebar) into a U shape then attach a cable tie to secure it or even glue it there if you want.
Cheers
MM