Keeping the pointer creates use-after-free opportunities, and we don't have much reason to keep it around anyway.