The Myth of Measuring Democracy

On April 5, two weeks after Georgia passed its controversial voting law, Vox published a provocative headline: “Study: Republican control of state government is bad for democracy.”  The study, “Laboratories of Democratic Backsliding,” was conducted by the University of Washington’s Jake Grumbach and outlines a “comprehensive measure of democratic health” based on 61 different factors.

The best predictor of a low “democracy score,” Grumbach found, was Republican control of state legislatures, leading him to conclude that the GOP is the main driver of “democratic backsliding” in the United States. He’s hardly alone; plenty of political scientists reached the same conclusion, using similar methodology, throughout the Trump era.

But a closer look at Grumbach’s scoring system reveals a paradox: At least three of the factors that decrease a state’s democracy score—voter ID laws, high incarceration rates, and denying felons the right to vote—are things that large majorities of Americans support. According to Grumbach, maximizing democracy means defying the popular will.

Grumbach’s study and others like it have reverberated across the media in recent months. “[T]he Republican War on Democracy is Moving to the States,” New York Magazine said of Grumbach’s findings. “US Republicans are starting to look a lot like authoritarian parties in Hungary and Turkey, study finds,” read an October CNN headline. “Historic data” show that “the United States is backsliding into autocracy under Trump,” the Washington Post’s Christopher Ingraham said in September.

The studies are all part of a political science cottage industry that purports to “measure” democracy, often with support from government-funded NGOs. Freedom House and the Polity Project—funded by Congress and the CIA respectively—have both devised widely used indexes for quantifying democratic performance, as has the Sweden-based V-Dem Institute. Linking these indexes together is the idea that democracy is not just in the eye of the beholder; it’s something scholars can measure empirically.

But in practice, most political scientists just seem to be measuring their own biases, which are then passed off as impartial, statistical truths. Beneath each metric of democracy is an implicit definition of it, an account of what democracy is or should be. And when the media report on these metrics, they often omit the assumptions and values underpinning them—reinforcing the veneer of objectivity the metrologists project.

Bias enters the measurements in two ways: the choice of variables used as proxies for democracy and the process by which those proxies are assessed.

Grumbach’s paper exemplifies the first sort of bias. Many of its “democracy indicators”—absentee voting, same-day voter registration, felon enfranchisement, decarceration, no voter ID laws—parallel the policy preferences of progressive activists. Not all of those policies are popular. Most Republicans and over half of Democrats back voter ID laws, recent polling suggests, and one 2019 survey found that 69 percent of Americans oppose letting felons vote in national elections. Furthermore, 76 percent of Americans say the criminal justice system is either “too lenient” or “about right.”

Reached for comment, Grumbach acknowledged the paradox, conceding it was a “fundamental” problem that he hadn’t solved.

The paper also gives extra weight to metrics on which the GOP underperforms. Sixteen of its 61 indicators—over 25 percent—are different measurements of gerrymandering, a practice Republicans have recently engaged in more than Democrats. No other factor comprises such a large percentage of the index. So even if all 61 indicators mattered equally, gerrymandering would still have an outsized effect on states’ democracy scores, since an outsized portion of the indicators concerns gerrymandering.

If Grumbach’s scale has a leftwing bias, others have a reactionary one. According to the Polity Project, the United States was a full democracy—10 out of 10—as early as 1850, when slavery was legal and women could not vote. That’s because the Polity proxies give a great deal of weight to constraints on executive power, but almost none to constraints on universal suffrage—implying that democracy is more about limited government than the consent of the governed.

In 2017, Vox cited the Polity Project to argue that democracy was on the rise worldwide.

The most balanced framework is arguably V-Dem, which uses hundreds of indicators to measure democracy along multiple dimensions. Some track values emphasized by conservatives, while others track values emphasized by progressives, making the index appear neutral and nonpartisan.

But it too reflects the liberal consensus—because the indicators are all scored by liberal academics. For each indicator, V-Dem asks five experts to rate a country’s performance on a numerical scale, then aggregates their ratings into a single number. That number doesn’t reflect anything objective; it’s just a mathematical representation of expert opinion. And since most academics are liberal, so are V-Dem’s ratings—even on dimensions of democracy that are ostensibly objective.

That helps explain why “executive respect for the Constitution” took a nosedive only after Trump’s election, even though most conservatives—including those who are anti-Trump—argue the decline began earlier and accelerated with Obama’s executive orders. Respect for the Constitution is an abstract concept; it can’t be counted the way elections can. But by framing both quantitatively, V-Dem hides the bias inherent in its indicators.

This sort of bait-and-switch lets experts launder partisanship as political science. According to Jason Blakely, a political theorist at Pepperdine University, it also lets them usurp democracy.

“What democracy ‘ought’ to be is too often dressed up in the guise of what it simply ‘is,’” Blakely told the Washington Free Beacon. This gives expert opinion “an aura of scientific inevitability” that is then “used to override” the popular will. Ironically, Blakely went on, the very act of measuring democracy can become “anti-democratic by certain definitions.”

Grumbach said that he was “sympathetic” to this argument.

Scientific concepts of politics seemed due for a reckoning after the 2016 election. Thinkers from across the political spectrum saw Trump’s victory as a revolt against technocrats, who stood accused of taking too many matters out of the hands of the people. The technocratic tendency, these thinkers stressed, was bipartisan; those dissatisfied with it had rallied around the one candidate promising to “drain the swamp.”

Yet for all that, the desire to displace democracy persists. Neither left nor right is “immune [to] the temptation to present their ideology as the only … scientifically supported way to do politics,” Blakely cautioned. It’s a temptation to which both sides can succumb.

*As an Amazon Associate I earn from qualifying purchases