"...but why is time travel into the past so difficult?"

It is impossible. We are 100% made of elementary particles and there is not enough energy in the universe to have one of them travels in the past, even one Planck time. Why? Because all elementary particles are "entangled' with the whole universe, having been in contact with all other elementary particles in the pre-BigBang singularity.

And if they weren't there, they were all "created" at the same instant at the Big Bang.

So, the travel back in time for one elementary particle entails travelling back in time of all the other elementary particles of the universe, and there is not enough energy in the universe to do that.