I redid my sd card install, this time I used gparted to move the swap file to the end and expand the filesystem over the full 8gb. Then I changed the swappiness from 60 to 10.
I swear the sound quality has improved, and it was already great.
Should I have expected a change at all, or is this just completely subjective?