Matsuno said VS was not going to be part of the Ivalice world originally and changed his mind afterwards. My reasoning for XII being a reboot is that starting with XII, Ivalice became more of consistent world, helped by being closer together in the timeline but the XII era of Ivalice begs a lot of questions about FFTactics world. I mean Moogles in FFTactics are largely extinct beyond the summon spell and the description of them in the game's lore describes them as being closer to the Moogles encountered in FFV and VI than the talking, mechanically inclined bunny creatures featured in FFTA and XII. VS has some subtle references to Tactics with a couple of name drops, but most of it's strong connections to Ivalice were created in XII with the Empire being placed in Valendia, the Kildian religion popping up and the concept of Pailings and the similarities between Mist and the Dark appearing, which were all absent in FFTactics. Hell, FFTA's Gran Grimoire is direct shout out to Vagrant Story where the concept originated from in the Ivalice titles, but it's nature and power are completely different in both games and only related by name.

I don't believe that Ivalice and Zetegenia are really connected. In my own interpretation, Ivalice is basically Matsuno trying to reboot the Ogre Battle franchise with Square's better finances and development. So Ivalice is basically a spiritual successor and most likely used ideas Matsuno had planned for Ogre Battle/Tactics Ogre but I feel it would be a stretch to say they are part of the same universe. Basically Ogre Battle/Tactics Ogre are to the Ivalice Alliance what Xenogears is to Xenosaga. Same concepts, design, themes, and even similar plot elements but they are too many little discrepancies to make them fit neatly and instead references to the previous work should be viewed as a fun shout out than some evidence of connection. Square didn't even own the rights to the franchise until after the merger and by then Matsuno had already created FFTactics, Vagrant Story and was working on Tactics Advance and FFXII, Ivalice was well on its way and Quest had also expanded the Ogre Battle universe itself.. I guess it's possible to retcon them into continuity but I doubt fans would be pleased by this. Most fans of the Quest series are still bitter about Matsuno jumping franchises and making FFTactics.

Basically my point is that I don't think Matsuno planned out Ivalice from the get go like he did with the Ogre Battle franchise, instead he's been backtracking with new games he makes to make them connected and frankly it is not quite as smooth as some franchises like say Suikoden, which even the games not made by the original creator are pretty meticulous about not altering established canon. Ivalice was sort of made into a franchise by force of will and some heavy handed reference dropping in new titles. I don't believe he really meant for Ivalice to expand into a giant world until sometime during XII's development which is when he actually started to talk about it like a franchise as opposed to just the individual games.