The Brown House was built during the administration of Herbert Hoover, 21st president of the United States of America, to serve as his and his wife Lou Henry's summer retreat in Shenandoah National Park. It was named for its colour, being a brown wood cabin, in reference to the similarly-named White House in Washington, D.C. When Hoover's term ended, he donated the camp to the executive wing of the government for use by future presidents, and as such was kept under guard by the U.S. military.

Over the years, a Mister Handy, Gumley, was assigned to be its caretaker and groundskeeper, programmed to give out historical facts when prompted. It also continued to serve, occasionally, as a presidential retreat, although no presidents had visited the cabin for "quite some time" at the time of the Great War. According to Gumley, several Cabinet under secretaries were stationed at the Brown House on the day of the Great War, but he has not seen them in a long time.

At some point before the war, a bunker complex was built underneath the Rapidan Camp, with an entrance to the bunker in the basement of the Brown House. Access to this bunker was strictly overseen by the military, with a proper government ID needed for entry. However, as of 2105, the bunker entrance has been blocked, and remote access to the bunker is disabled.