Can I remove upvotes?

Upvotes can be removed directly from the idea page.

On the right sidebar, where you have a list of voters, each voter will have an "x" icon after its name.

You can click the icon to delete an upvote: