Observations of the cosmic microwave background, leftover light from when the Universe was only 380,000 years old, reveal that our cosmos is not rotating. Infinitely long cylinders don’t exist. The interiors of black holes throw up singularities, telling us that the math of GR is breaking down and can’t be trusted. And wormholes? They’re frighteningly unstable. A single photon passing down the throat of a wormhole will cause it to collapse faster than the speed of light. Attempts to stabilize wormholes require exotic matter (as in, matter with negative mass, which isn’t a thing), and so their existence is just as debatable as time travel itself.
This is the point where physicists get antsy. General relativity is telling us exactly where time travel into the past can be allowed. But every single example runs into other issues that have nothing to do with the math of GR. There is no consistency, no coherence among all these smackdowns. It’s just one random rule over here, and another random fact over there, none of them related to either GR or each other.
If the inability to time travel were a fundamental part of our Universe, you’d expect equally fundamental physics behind that rule. Yet every time we discover a CTC in general relativity, we find some reason it’s im possible (or at the very least, implausible), and the reason seems ad hoc. There isn’t anything tying together any of the “no time travel for you” explanations.