This is a real nice beatbox ! it sounds just like the Commodore 64 I owned in the past…
I found two possible causes regarding the unwanted buzzing sound.
First you could check your filter on the breadboard. I experimented a little with different capacitors, when they have a lower value (i tried 0.01 uF) then you can hear the buzzing sound from the PWM signal.
When you remove the GND connection from the filter, then you can really hear this buzzing sound.
Then I found something strange…
First I used the P1 power bit with a 12V wall wart. The beats sounded 100 % OK. No disturbances at all.
Then I removed the P1 power bit and replaced it by the P3 USB power bit. Then the buzzing sound appeared ! As you also noticed, it shortly stops at the end of each beat.
I used a USB wall wart and I used the USB outlet on my PC, and it made no difference at all.
See posts below, it was caused by the VCC voltage level, as discovered by @Frankje.
My P3 USB power bit has a slighter higher voltage than my P1 power bit.