If we don't unset it, all the CMake checks will run with PortAudio in the library list. This didn't affect those who had PortAudio, but for select Debian systems without it, it breaks everything.