I was at the Right to Information and Transparency in the Digital Age conference at CDDRL earlier this week. The final wrap up session “What does this all amount to? Assessing the impact of transparency”, presented three rigorous impact evaluations conducted by political scientists. With some interesting nuances, all three papers basically found the information initiatives they evaluated had little to no impact.
Dan Posner of UCLA said the null findings had prompted his team to return to their priors, that better information leads to more active citizenship. Given the null result, they backed up and ask what the conditions actually were under which it might be reasonable to expect that better information might actually lead to greater demand from citizens for accountability.
This is absolutely the right question, and the direction the field should go in.
In assessing the impact of interventions in the field of political development and governance, the devil resides squarely in the details. This is (one reason) why so many rigorous studies in this field find null results.
The way forward is to link very careful and much more specific thinking about the conditions under which we should expect interventions to work, together with our rigorous evaluations of whether they actually work or not. Specific criteria used for assessing likelihood of success can be very helpful for creating benchmarks against which to assess impact. It also may be an important way to think about aggregating up very micro-level research into a broader set of findings about the effectiveness of various forms of intervention.
An article called “Assessing Advocacy” in the most recent edition of the SSIR takes up this question with respect to evaluating advocacy work. The authors provide a framework for evaluating the prospective success of advocacy campaigns that has 9 factors, or conditions, for success. Perhaps further iterations of such a framework might include ways to assess the relative weight of each factor in a given advocacy campaign, or the relative (as opposed to simply the absolute) strength of the campaign on a given factor versus opposing forces.