But you would have to wait till the future to find out that you went to the past to do whatever you did. Unless you saw your future self do it or someone told you that they saw you do it. In either case, you wouldn't believe your own eyes or the person telling you the story.
So how again do you know that time travel to the past can't happen?
Threaded Mode