Time travel is now at the point where space travel was centuries ago. It is really no different!

Mankind has always moved in one direction at the same speed: the orbit of the planet. Any other direction was insignificant compared to the speed of the planet. Time is like that: we think we go forwards at a fixed speed, and we think we can't change that in any significant way. But then people invented the hot air balloon. And that let them imagine rockets! Now we know that rapid movement is possible in any direction! And it is the same with time. The only limit is our understanding.

My friend has an objection: "If time travel was possible then a time traveller would already be ruling the world. Because they would go to the past, and gamble on the horses of otherwise change history in their favour." Should I answer him? Or should I prove I am right by building a time machine? Or do something else?