band or no band

I worry about how the cigar will smoke if I take the band off, possibly damage the wrapper or get sweat and oil on the wrapper from my hands. I'm not sure it would make a difference either way, but I leave them on, but not to show off (I dont have many fancy cigars anyhow).