Then I'll tell you.
Martine, who ran the Galbadian Garden that Irvine was at, confiscated all of the card in the Garden.
Then he lost the Garden, ended up in FH and lost the cards there.

There, you see, it was simple wasn't it?
And it makes perfect sence.