冗談は顔だけにして (jyodan wa kao dake ni shite) Or The only joke you need to worry about is your face.
Japanese is super lacking in any kind of swears. It’s not uncommon to hear in a fight between two rather butch looking men
“no, you are a fool!”
“you are a STUPID fool!”
So when you find yourself around a wasted Japanese salaryman who is making jokes about the size of your stereotypically large penis (we’ll talk) or someone thinks they are being funny when really they are just annoying you ; you can use “jyodan wa kao dake ni shite”