Merchants is a nice place for sure, though you may feel a bit out of place in jeans and T, even if they don't exactly ban that. For sure you could not get into Club Macanudo (which would likely be a bit closer to your hotel), as they wouldn't even let A-Rod entry in jeans and sneakers . . . and that was when he was still hitting!
Don't know when JRs on 5th Ave or Nat Sherman's off 5th close, but you can likely smoke a stick in those places too.