Daryl has never met Morgan, guys. I am sure he knows of him but not enough to invite him back to camp. Remember, the last time Rick, Carl and Michonne saw Morgan he was nuts.