Are videos ever removed from my Watchlist without my knowledge?

No. All videos you have added to your Watchlist remain in your Watchlist for as long as you want them to. It should be noted that expired videos will not play.