can we do a test where we only show it to new visitors?

Hi Ashley, yes. You'd need a way to determine which users are new, and then you can add a targeting rule to the experiment.