Having fun is the important part. Whilst I can hack it on the higher difficulties I very rarely play on them, for precisely the same reason - I just want to derp around building my empire and teching.