SMT is actually one of the top five RPG series in Japan, the franchise dates back to NES days.

SMTIII was the first breakout hit for the franchise in the U.S. thanks to the extra PR from having Dante of Devil May Cry fame making a cameo in the game. The franchise saw some modest hits with Digital Devil Saga and Devil Summoner. Persona 3 was the big sleeper hit that caught people by surprise and then the buzz from that game helped make P4 become a hit as well. Atlus has gone on record saying Persona is their biggest franchise now, finally overtaking SMT, hence why P3/4 have remakes, directors cuts, ports, spin-offs, and so forth.

The series is still niche abroad, but if you really look at sales numbers then nothing is going to really compete with Final Fantasy and The Elder Scrolls at this point, so most RPGs are basically "niche".