No, the poll worker would put the voter's smart card into their nifty little spyrus, and see that it wasn't voted. No problem. Kinda like they would do with, uh, paper ballots? How are you planning on dealing with the voter that finds and uses the hidden cancel button, and then comes back to the poll worker asking for another smart card?
To clarify, I don't have a problem with the pollworker being informed of the ballot being canceled, one way or another. Its the hidden button that is offensive.