Dolley Madison’s popularity as a hostess in Washington added greatly to the recognition of her husband by those members of congress whose electoral votes then chose the winner of presidential races. During the 1808 election, however, there was an attempt by Federalist newspapers in Baltimore and Boston that implied Mrs. Madison had been intimate with President Jefferson as a way of attacking her character. Her popularity prevailed during the 1812 election.
As the invading British army neared Washington in 1814 and the White House staff hurriedly prepared to flee, Dolley Madison ordered the Stuart painting, a copy of the Lansdowne portrait, to be removed:
“Our kind friend Mr. Carroll has come to hasten my departure, and in a very bad humor with me, because I insist on waiting until the large picture of General Washington is secured, and it requires to be unscrewed from the wall. The process was found too tedious for these perilous moments; I have ordered the frame to be broken and the canvas taken out”….. “It is done, and the precious portrait placed in the hands of two gentlemen from New York for safe keeping. On handing the canvas to the gentlemen in question, Messrs. Barker and Depeyster, Mr. Sioussat cautioned them against rolling it up, saying that it would destroy the portrait. He was moved to this because Mr. Barker started to roll it up for greater convenience for carrying.” [15][16]
Popular accounts during and after the war years tended to portray Dolley Madison as the one who removed the painting, and she became a national heroine. Early twentieth-century historians noted that Jean Pierre Sioussat, a Frenchman, had directed the servants in the crisis.[17][18]
Dolley Madison hurried away in her waiting carriage, along with other families fleeing the city. They went to Georgetown and the next day crossed over the Potomac into Virginia. When the danger receded after the British left Washington a few days later, she returned to the capital to meet her husband.