If I switch to a free plan will I lose my vanity domain url?

The feature to add a vanity URL is available only from the Blossom plan and therefore if the account is downgraded to the Sprout plan, you will not be able to make use of the Vanity URL.