I say you tell him straight to his face that you don't want to hang with him anymore.
Don't offer any explaination, it is not necessary.
If you DO get beat up, it's better to get beat up once, rather than having to deal with it constantly since now, you're his "new best friend".
-ralex