I don't know anything about japanese, but when you said 'stay patient' I just thought of something. When I lived in Quebec, I made no real effort to learn French, and though I was bombarded by it whenever I left the house, at home it was all English all the time.

You are clearly making an active effort, but remember, children generally aren't proficient with a language until at least their fifth year. While I'm not comparing you to a five-year-old, it could mean you have a long way to go. Maybe it'll take you less time to become fluent because your brain is more developed than a child's, or maybe longer because not only is there one (or more) language already in there, but all kinds of other information.

As long as you're making a good effort, you should learn a lot though

Good luck!

GANBATTE (or somesuch...hope i didn't just swear)