For as smart as he was, and as important as he was in framing the Constitution, should he have been a better president? Or perhaps those are different skill sets.
But at least he had Dolley.
It was a very different time and attitudes toward, say, hair, were very different as well. Men still wore powdered wigs in public, for example.
Federalists and [Democratic-]Republicans differed in dress as well as politics. In Washington, Federalist Congressmen wore long powdered queues and had them dressed every day. Jeffersonians tended to wear short hair, or at least small queues tied up carelessly and requiring little professional attention. One day, when a Federalist was having a shave, his barber suddenly began bemoaning Madison’s nomination for President.
“Dear me!” he sighed. “Surely this country is doomed to disgrace and shame. What Presidents we might have, sir! Just look at Daggett of Connecticut, and Stockton of New Jersey! What queues they have got, sir!—as big as your wrist, and powdered every day, sir, like real gentlemen, as they are. Such men, sir, would confer dignity upon the chief magistracy. But this little Jim Madison, with a queue no bigger than a pipe-stem! Sir, it is enough to make a man forswear his country!”
— Presidential Campaigns by Paul F. Boller Jr.