Perelman's writings on Smith's populism, and that famous metaphor, are worth quoting at length. From The Invention of Capitalism, page 208-
I suspect that Smith's work earned much of his popularity because he expressed so eloquently what others deeply felt. Unlike many of the less educated populists, Smith was usually able to sublimate his rage into his charming theory of the invisible hand, in which competition and even aggression is channeled into harmonious actions that better the world. Frequently, cracks appeared in this fantasy, and the harsh reality of the world around him intruded. At such times, we can catch a glimpse of Smith's theory of primitive accumulation.
Smith's vision of the bizarre heroism of the petit bourgeoisie seems to reflect his own rage at those who refused to adopt the values that were so dear to him. Even if Steuart's [Sir James Steuart] language was brutal, I suspect that society has more to fear from the repressed emotions of someone like Smith. His metaphor of the invisible hand may be relevant in this regard. We may equate friendship with an open, outstretched hand, but an invisible hand has something sinister about it. In this spirit, Macbeth requested that the darkness of night, "with thy bloody and invisible hand," cover up the crimes he was about to commit.