I have not started on P4 yet, I'm finishing up FES and getting ready to start The Answer. I have finished the original P3 and will add that P3 and P3:FES are quite different and both totally awesome. FES really improved the game more than most Special Editions do.
P4 also has a point where you get to visit a few places from P3 so you may want to play P3 first just so you can get the in-joke. I don't believe the games offer major connections to each other but can't say for sure. Also, don't worry, you don't need to have played Revelations: Persona ~ Be Your True Mind , Persona 2: Innocent Sin, or Persona 2: Eternal Punishment to understand Persona 3. You may catch a few more in-jokes but the stories don't really cross.
I also second The World Ends With You