I actually don't mind mini-games as long as they are integrated into the story. Having a task turned into a mini-game doesn't bother me and generally breaks up the monotony of talking to NPCs or dungeon crawling, especially since the RPG genre has decided to abandon actual puzzles in dungeons. I feel Chrono Trigger, VII, IX, and a few others did it best. FFX is an example of how not to do it cause all the mini-games are not only tedious and lame, but feel out of place considering the story.

Yuna: Let's go save the world Tidus!

Tidus: Can't, I still have to dodge 50 more Lightning bolts then go to Macalania Woods for some Butterfly catching!

Yuna: But Sin's going to destroy Spira...

Tidus: Then it's for a few blitzball tournaments, and training Chocobos to dodge balls being thrown into their faces while they catch balloons! Also, remind me to pester everyone in Kilik who is busy restoring their ruined village, to see if they want to join a rival blitzball team, our goalies contract is almost up...

Yuna: But... Sin...

Granted, games like VII and IX have some non-story related sidequests but they also had some actual ones tied to the story. Hell, most of VII's good optional mini-games are just being able to replay the mini-games introduced earlier in the title.