Yes, you're most likely targeting enemies when trying to get to a tower because you're accidentally or intentionally pressing R1, (trust me I've done this too, I thought warping to a tower involved pressing R1 and it does not)
You cannot be locked onto an enemy if you want to warp to a tower, you have to aim your camera AT the tower with your right stick (the one that controls the camera) and a blue warp circle will pop up and then you press X. Your not supposed to lock on when warping to a tower at all.
Now having to re-explain it, this is also over complicated to warp to a tower XD!!!
(also I understand you guys know how to use a controller and it's functions but I just wanted to be clear with my words!!!)
I keep seeing posts about the same problems and frustrations I experienced, but once I sorted out the controls, the battle experience was much better!