The legality and intensity of slavery within the counties of the states are powerful clues for predicting which of the three main presidential candidates would win a county's vote - and perhaps even by what margin. (Compare the two maps at page bottom.)
Most significantly, Fremont never wins any county in a Slave State. Indeed, he only won only
The slave-heavy lower Mississippi valley and so-called Black Belt in the southeast are Fillmore country. In general, as one moves within a Slave State from counties with a lower slave fraction to those with a higher one, the vote shifts from Buchanan toward Fillmore. For example, victory passes from the former to the latter as one enters the slave belt of Missouri. But several Upper South states - Virginia, Tennessee and North Carolina - are not as inclined to follow this simple rule.
In the South, the issue of Catholic immigration shows up in the Buchanan-Fillmore divide as well. The only state whose electoral votes Fillmore took was Maryland - the state with the highest rate of church (i.e. Protestant) adherence. States west of the Mississippi are all biased away from Fillmore, albeit the Buchanan-Fillmore slave-density-dependent shifting pattern persists. These are "frontier" states in which the male/female ratio is higher and the church adherence rate is much lower. (See the religious adherence map here.)
Haralson county and its neighboring Georgia counties tend to follow the general trend seen throughout the Lower South. With very low slave fractions, Haralson and Paulding vote heavily for Buchanan. Carroll County, with a greater slave fraction, still votes for Buchanan, but with a thinner margin. And Polk County, with an even higher slave fraction, votes for Fillmore. In Georgia, with few exceptions, low-slave-fraction counties like those in the northern mountains go for Buchanan, while high-slave-fraction counties go for Fillmore.
It is not surprising that Slave State counties do not vote for a candidate, Fremont, whose party wants to contain slavery. But why do slave-richer counties shift preference from Buchanan to Fillmore? Remember that Fillmore's American, or "Know-Nothing" Party is intent on keeping immigrants - white people without slaves or the ability to own any soon - out of the country if they can. First, success of such an effort would obviate electoral strength growth in the House of Representatives by the Free States, where almost all immigrants would originally settle. It would also reduce the number of white people who might eventually settle and work the land in the new western territories with their own hands, leaving more space for slave-labor. Indeed, it would tend to reduce the upward pressure on land prices there for slave-owners. But more importantly yet, under the "popular sovereignty" doctrine layed out by Senator Stephen Douglas, it would also make it less likely an unsettled territory would vote for admission as a Free State, barring the practice of slavery within it forever.
Of course whites without slaves would be equally interested in keeping slave owners from settling in the new territories for parallel economic and political reasons. And such motivations required no sympathy for negro slaves on their part - but such claims might prove useful in seizing the moral high ground. Even for whites remaining within older Free States, the "aristocratic" threat of what was called the Slave Power would motivate them to prevent the spread of slavery to the new territories. A Free State with, say, 500,000 white people, including 100,000 voting adult males, would get a certain number of seats in the federal House of Representatives. But a Slave State with the same number of white people and same number of voters might get many more. If that Slave State had, say, 500,000 negro slaves, it would have roughly 60% more seats than the Free State, based on the so-called three-fifths compromise. (Even worse potential threats of Slave Power to a society of nominally free and equal white citizens were cautioned against, too.)
[Aside: Even at the end of the Civil War, a mere six northern states allowed negroes to vote. It would take passage of the Fifteenth Amendment in 1870 to even theoretically safeguard negro suffrage. Those opposed to negro equality, including voting, would sometimes point to the example of the negro republic in Haiti to support their arguments.]
In 1856, if you were a slave owner or someone else who profited from slavery, you wanted to keep the United States as black as possible; so Fillmore was your man. He won almost no counties in the northern Free States. For southerners, while not negligible, the religious angle was not always overwhelming: even if Catholics made you uneasy, there was little chance immigrant Catholics would settle in the South anyway. And in a state like Louisiana where Arcadian or "Cajun" (French) Catholics lived already, thousands of them actually backed Fillmore.
While Fillmore lost the election, and his American Party soon evaporated forever, its influence in the South may have been deeper. Often clandestine in its operation, it stoked anxieties about immigrants in general and Catholics in particular. These characteristics would be echoed when another organization was refounded in Georgia in 1915 - the Ku Klux Klan.
At the risk of gilding the (white?) lily, one might ask why virtually all southerners not voting for Fillmore voted for Buchanan; Fremont got only six hundred of their votes. One reason was probably logistical: The wealthiest and most powerful people in the South were slave owners and would use all their influence to prevent a candidate like Fremont from even appearing on a ballot. But surely racial politics played the essential role, aside from the question of slavery.
Perhaps it is fair to say a vast number of white Americans at that time would look upon someone else who benefited from ownership of a negro as an affront to their ideals concerning equality. But if they could be that someone else - or maybe just imagine they one day might be - their sense of indignation would almost always be quieted. In any event, very few white Americans wanted to see people of any complexion long held in bondage liberated to move about in their midst and face the challenges of freedom. And if such people didn't look like them, sympathy would be much harder to generate yet.
Thus white people in the North, where no new black people would ever again be manumitted, were perfectly willing to vote for a party and candidate hostile to slavery. But white people in the South, even those without any slaves, would not entertain such a possibility. Thus Fremont would win counties nowhere near where slaves were held, whereas Buchanan would win all other counties not going to Fillmore. A tiny idealistic minority, about 3,000 Americans, would vote for the Abolition Party, but fail to win a county, much less a state, anywhere. Their ideas would be ridiculed, such as in a song Stephen Foster wrote for the Buchanan campaign.
Surely many times considerations unrelated to slavery would effect for whom a white man would vote in the American presidential election of 1856: say, Fillmore's experience as president, Buchanan's long diplomatic service or Fremont's glory as an explorer. But if we stereotype voters according to their relationship to people held in slavery, Fillmore voters were more likely than average to be slave-owners, Buchanan voters more likely than average to be other slave-neighbors, and Fremont voters almost exclusively slave-strangers.
As Alexis de Tocqueville famously wrote a generation before 1856,
Whoever has inhabited the United States must have perceived that in those parts of the Union in which the Negroes are no longer slaves they have in no wise drawn nearer to the whites. On the contrary, the prejudice of race appears to be stronger in the states that have abolished slavery than in those where it still exists; and nowhere is it so intolerant as in those states where servitude has never been known... In the South... although legislation treats [those enslaved] more harshly, the habits of the people are more tolerant and compassionate.
Negro expulsion from railway car, Philadelphia (Illustrated London News, September 27, 1856)