Only delete things that we expect to find, to avoid deleting other things that people might have lying around in their build tree. Explicitly skip .git to avoid e.g. accidentally matching a branch name.