Munck, Gerardo L. and Jay Verkuilen. "Conceptualizing and Measuring Democracy: Evaluating Alternative Indices". Comparative Political Studies, Vol.35, No.1 (2002): 5-34.

Munck, Gerardo L. and Jay Verkuilen. "Conceptualizing and Measuring Democracy: Evaluating Alternative Indices". Comparative Political Studies, Vol.35, No.1 (2002): 5-34.

• Definitions of terms like democracy should not be overburdened with other concepts and requirements of marginal importance to the core definition, as it limits the experimental value of the term. This is shown in the inclusion of market economy as a requirement for democracy, which only serves to limit the ability to explore correlations between democracy and free markets (9).
• Most of the indices of democracy currently used stick a procedural definition of democracy, keeping them analytically lean, with the notable exception of Freedom House, which includes several variables related to market economies in its definition of democracy (9, 11).
• Many of the indices of democracy omit participation as a variable in determining democracy, despite its important in Dr. Robert Dahl's definition of polyarchy. This is especially crippling for indices like the Polity Project, which extend back 200 years, and thus fail to account for the major expansion of suffrage over time (11).
• Another common omission in most datasets is the number of elected offices open for election, which is included in the ACLP index developed by Dr. Michael Alvarez and others. This is an important aspect of democracy and should be measured (11-12).
• To be more analytically useful, the variables for measuring democracy used by those indices should be organized into defined categories underneath different qualities of democracy, like competition, participation, or the impact of elections. Data should be categorized, not serving only as a checklist (12-14).
• Instead of beginning with variables that seem useful to the measurement of democracy, researchers should begin with concepts that underpin a healthy democracy and try to determine possible indicators for those concepts. The variables should be chosen afterwards, not before (15).
• Scholars do a notoriously bad job confirming both the validity and reliability of their data: not bothering to check if measurements taking really correspond to levels of democracy either in the abstract or in tests. Invalid measurements are even more likely, since most scholars may suffer from the same intellectual biases (18).
• A number of indices depend on subjective judgement to generate their quantitative data. At a minimum, the checklists used to determine the values of quantitative variables should be available to the public, which is often left entirely at the mercy of an 'expert' for the assessment of appropriate values for a country (18-21).
• Despite collecting large amounts of extremely useful disaggregate data, composed of all the individual variables used to create the composite score, most indices do not publish this information, causing the public to miss out on large amounts of more nuanced country-specific information (22-23).
• The unavailability of disaggregate data also hides the weakness of many indices, especially Freedom House. Decisions to give the actual power of representatives the same weight as the strength of political parties in calculations of democracy are hidden if only the final score is available (25).
• A final table showing the strength and weaknesses of the available indices of democracy is available on page 28.

Basic lesson of this piece is that constructing a index for the measurement of democracy or any other abstract concept is that you should first establish the constituent concepts needed for the measured concept to exist, then look at what concrete, measurable variables could be used to assess those constituent concepts.