If a person just says "call me sometime", I'd wait a few days, like you said. But if they wanted me to call to set up a date for a few days later, I'd call the day after. You'd have a day to figure things out, and you wouldn't leave the person waiting.
I can understand why you waited, but I don't think it seems desperate for someone to call the next day if it's to set up a meeting.
