Well... the Gupta empire of India was very market oriented - while the concept of capitalism was certainly nonexistant, texts have been found detailing a system which was roughly capitalist. Only those who tilled the royal lands payed taxes, and local governments held the most power. It was more like a confederation of city-states than anything else.
More extreme versions of capitalist societies include
medieval Iceland, which was nearly anarcho-capitalist in nature. Some also like to site the "Wild" West as a very capitalist society.