Most of the games explain where they are.

If they have leisure time, I'm sure that there's some ethreal dimension that they spend time at, that people cannot normally pass through without some kind of effect, like in FFIV (Rydia becomes an adult due to the expenditure of time).