[Perndog]

The Egyptian god Amon had the head of a goat and was known as "the hidden one...the soul of all phenomena." His likeness, the Goat of Mendes (after a city in Egypt) has been used by magicians (particularly "black" magicians) since ancient Egyptian civilization; it was demonized by early Christians - sheep vs. goats is a well-known Christian symbolism, and they added quite a bit to it, which made the symbol all the more popular among Satanic types.

So now you know. It's basically a symbol of opposition from Egyptian times that has been reinforced by Christianity.