Granted. This works so well that, as time goes on, you wish for more and more clones to build your empire, and for years, that wish gives you prosperity and happiness. One fateful day, two happen to stumble across each other. "Forsooth! It appears that we are mere pawns in his cruel games, being exploited for no other reason than the simple fact of our existence. This cannot stand! We must band together with our bretheren, for the good of all and the pursuit of freedom!"
Through their combined efforts, they band together, gathering their fellow mini-me's, to stand strong in the face of their mutual adversity. With their great force assembled at last, they hit you with a class-action lawsuit, on the basis of your inherently exploitative labor practices. Wealth depleted, morale at rock bottom, you die alone in a rickety shack in a cornfield.
I wish I could learn to better manage my free time.