Increasing effectiveness with high-quality internal research

Company blogs
Abstract: Kinder wants to encourage organisations to conduct internal research on their programs and produce data, but currently has limited measures that can make sure the research is not useless. We don't want organizations to waste time and resources by doing low-quality research just to check a box, and we don't want them to produce misleading information. Here are some ways in which we are already well-adapted to the problem, and some ways we could improve our framework to take these issues into account.

Kinder believes that there are still too many charities carrying out programs without sufficient evidence that these programs are effective or work as intended. This state of affairs is reflected in public perception and may be causing harm to the charitable sector.

Kinder wants to shift the culture so that charities invest in interventions that work and regain public trust by providing evidence and reasoning. That’s why stage two of the Kinder Vetting Framework, Organisational Competence, includes measures of charities’ commitment to conducting research on their programs and producing data.

Lack of sufficient evidence in the sector is by no means a new problem, and Kinder is certainly not the first organization to try and rectify it. Back in 2006, the Stanford Social Innovation Review (SSIR) was already warning us that charities were “drowning in data”. According to SSIR, charities were conducting low-quality evaluations with low-quality results at the request of large donors who underestimate the difficulty and expense of impactful evaluations.

We want to avoid this type of failure and failures that are related to it. Goodhart’s Law accurately encapsulates our view:  “when a measure becomes a target, it ceases to be a good measure”.

A close-to-home example of a Goodhart type failure is the current hyperfocus on reducing overhead (operational) costs in philanthropy. This failure can, and usually does, cause underinvestment in hiring skilled staff. The president of the charity consultancy Grey Matter, Ron Sellers, claims one of the reasons calculating overhead ratio became so popular among nonprofits is that it’s so much easier than quantifying impact. But as a guide to impact per unit donated, it’s almost useless.

To sum up, rewarding charities for doing internal research could, if not done with care, result in organizations wasting resources, publishing low-quality research, or attempting to “prove” their effectiveness instead of testing it.

What we are already doing right

In some ways, Kinder is already combating this issue. In step two of the vetting framework, we look for several signifiers of quality that are relatively hard to fake. We check for:

  • Data transparency (e.g. the frequency of updates)

  • Adjustment of programs in sync with data collection (a test of the intentions behind the data collection)

  • Making the research available online and inviting peer review

  • The use of external references and external assessments in impact analysis, among other things.

 We also have a section which assesses the logic of the strategy, and a section that looks at whether the charity checks their programs are functioning as intended or not. Both are prerequisites for a valuable impact analysis.

Instead of focusing only on evidence of effectiveness, rewarding organizations for producing high-quality research counters the temptation of publication bias — a type of bias that occurs when the results of a research project influence the decision to publicly share it.

With regards to the authenticity of internal research, the third step, Intervention Effectiveness, will be the proof of the pudding since organisations which successfully improve will automatically score higher in effectiveness over time.

Ways we could improve

 In future upgrades of our framework’s second step, we should focus on checking the authenticity of the intention to improve. This can be done by adding more questions to the vetting framework. We have to balance the ease of comparing charities with the security of hard-to-game human discretion. Both human discretion and having more questions make the framework harder to ‘game’.

To set an example, and to better understand the difficulties charities face in gathering data, Kinder should swallow its own medicine. First of all, we should conduct some external research on the effects Kinder could have on charities’ behaviour, and later, research on the effects it is actually having. Then we will be able to track with evidence whether our intervention is functioning as intended and adapt, or abandon it as needed.

Further reading:

Goodhart’s Law and Why Measurement is Hard; “why the problem with metrics — and algorithms that rely on them — isn’t something that can be avoided.”

More Stories

  • Climate crisis has arrived, stop feeling guilty and start imagining your future

    Obstacles

    Evidence of the devastating impacts of anthropogenic climate change are stacking up, and it is becoming horrifyingly real. There can be no doubt that the climate crisis has arrived. Yet another “shocking new study” led The Guardian and various other news media this week. One-third of Himalayan ice cap, they report, is doomed.

    Meanwhile in Australia, record summer temperatures have wrought unprecedented devastation of biblical proportions – mass deaths of horses, bats and fish are reported across the country, while the island state of Tasmania burns. In some places this version of summer is a terrifying new normal.

    The climate disaster future is increasingly becoming the present – and, as the evidence piles up, it is tempting to ask questions about its likely public reception. Numerous psychological perspectives suggest that if we have already invested energy in denying the reality of a situation we experience as profoundly troubling, the closer it gets, the more effort we put into denying it.

    While originally considered as a psychological response, denial and other defence mechanisms we engage in to keep this reality at bay and maintain some sense of “normality” can also be thought of as interpersonal, social and cultural. Because our relationships, groups and wider cultures are where we find support in not thinking, talking and feeling about that crisis. There are countless strategies for maintaining this state of knowing and not-knowing – we are very inventive.

    The key point is that it prevents us from responding meaningfully. We “succeed” in holding the problem of what to do about the climate crisis at a “safe” distance. As the crisis becomes harder to ignore – just consider the current batch of shocking reports – individually and culturally we will dig deeper to find ways to strategically direct our inattention.

    How do you feel?

    The standard narrative for a piece like the one I’m writing here, as a social scientist, is to now say something about how the crisis could be better communicated. The billion-dollar question, of course, is whether this most recent disaster can be used to motivate real change. No doubt it is important to keep this kind of commentary up. It is key that we consider how to give the climate crisis traction in a culture so accomplished at distancing us from uncomfortable realities.

    But let’s be honest. No one really knows what works. We have never been here before. And I’m starting to think that more of this kind of analysis is, perversely, another example of distancing us from that crisis. Intellectualising terrifying climate crisis stories as an issue for “communicators” and “the public” is another way of detaching ourselves from their reality, from the relevance to me and you.

    So let’s cut through all that and stop invoking an imaginary audience. Many terrible things are happening as a result of climate change – their happening is being reported. How are you receiving it? How does it feel? Are you shocked, horrified, scared, bored, tired? What do you do with the terror? Do you compartmentalise it somewhere “safe”? Perhaps like me, you know you care. You attach importance to climate change, you want to act correctly, avoid risking other lives, damaging homes and habitats. Perhaps you know you are scared too – scared of contemplating what we have already lost or of what will happen as the crisis gets closer still. Scared of what you are being asked to give up.

    Add in some residual guilt and you might then engage in a defence of some kind, consciously or otherwise – telling yourself that others are more responsible, there is nothing we can do, everybody else seems to be carrying on as normal. As the crisis deepens, the walls close in, you might double down on those defences.

    Imagining a future

    So where do we go from here? How might this knowledge help us – you and me? We must make a commitment, but not of the kind you might imagine. The shocking reality of the climate crisis is making its way into the webs of everyday life, emotions, thought processes, relationships, hopes, dreams and fears. Perhaps we should commit to letting it, as an alternative to doubling down on our denial.

    We can do this individually, but more important is collectively acknowledging our fears about actual and anticipated losses. Fears about the loss of species and habitats, but also our established ways of life. This leads to more constructive questions, about what we want to hang on to, what are our obligations? I don’t have ready answers to these questions, but I am still confident we can find ways to keep doing the things we really care about – for ourselves, each other, the places we live in. But we need to talk about these choices.

    Such a process is still miles apart from many “sustainability” agendas. Halting the climate crisis is still predominately framed as a matter for individual choice and change – use less plastic, cycle to work, fly less. But the behavioural response required is way more complicated than that.

    When it comes to the climate crisis, the personal is political. I am talking about a politics that grows from opposition and critique of our current systems. This is evident in young people organising school strikes and protesters willing to get arrested for their direct action. But we also need to pay more attention to what is lost, to who and what we care for, to other possible ways of being.

    Some conservation scientists, at least, see recent cultural change as a hopeful sign of a growing sense of care and responsibility. So stop feeling guilty, it’s not your fault. Be attentive to what’s going on, so that you might notice what you care about and why. What are you capable of, and what might we be capable of together, when we aren’t caught between knowing and not knowing, denial and distress?

    See what obligations emerge. There are no guarantees. But what else do we do?

    This article is republished from The Conversation under a Creative Commons license. It's written by Olu Jenzen, Principal Lecturer in Media Studies at the University of Brighton. Read the original article here.

    This story features:
    Read more
  • Grassroots not grass-fed: the US might be getting its first vegan president

    Solutions

    The newest addition to a generous list of 520 (!) candidates for the upcoming 2020 US presidential election, was senator Cory Booker. On February 1st, Booker announced he is running for the Democratic nomination for President of the United States.

    Booker, a Rhodes scholar and Yale Law School graduate started his career in politics in 1998 as a Member of the Municipal Council of Newark with an upset victory. Between 2006 and 2012 he served as the 36th Mayor of Newark and assumed office as a US Senator in 2013, making him the first Black senator from New Jersey.

    As Senator Booker joins the most diverse Democratic presidential candidate pool in history with five women, one LGBTQ+, one Latino and two Black candidates (as of February 4th 2019) he’s also making history as the party’s first vegan candidate.

    A vegetarian since 1998 and vegan since 2014, Senator Booker often speaks about how switching to a plant-based diet has improved his life both mentally and physically. Alongside advocating for a plant-based diet for personal health reasons he is also very outspoken about the negative effects of the animal farming industry on the planet and on the lives of fellow humans.

    In an interview with plant-based magazine VegNews Booker said:

    “You see the planet earth moving towards what is the Standard American Diet. We’ve seen this massive increase in consumption of meat produced by the industrial animal agriculture industry. The tragic reality is this planet simply can’t sustain billions of people consuming industrially produced animal agriculture because of environmental impact… We will destroy our planet unless we start figuring out a better way forward when it comes to our climate change and our environment. ”

    Booker’s dietary preferences stand as a stark contrast with the current president ’s notorious diet of steak with ketchup and twelve diet cokes a day, as does his central message of unity and grassroots action.

    Cory Booker is not the only vegan US politician who’s running for future elections. Brooklyn Borough President Eric L. Adams, a vegan advocate who has several initiatives such as promoting vegan lunches in local school districts, will be running for mayor of New York City during the 2021 elections. A fellow New Yorker and vegan, Council Member Helen Rosenthal will also be running in the 2021 elections but for New York City comptroller.

    Many people have declared 2019 the year of the vegan and the plant-based diet the future of nutrition. A new way of eating with the premise of a flourishing planet and better lives for its inhabitants is making its way through our society from music to technology, and now outspoken vegan politicians with a real shot at the office in the country that is the world’s largest meat consumer.

    All signs point to a plant-based future.

    Speaking of a plant-based future, our first Kinder Conversation on the Future of Meat is fast approaching. We'll talk about the 'new meat' and how as a society we can (and should) reduce our dependency on animal farming. Get your tickets before we sell out: http://bit.ly/KinderMeat

    Header image is by Sean Davis via Flickr.

    This story features:
    Read more
  • Capitalism is killing the world’s wildlife populations, not humanity

    Obstacles

    The latest Living Planet report from the WWF makes for grim reading: a 60% decline in wild animal populations since 1970, collapsing ecosystems, and a distinct possibility that the human species will not be far behind. The report repeatedly stresses that humanity’s consumption is to blame for this mass extinction, and journalists have been quick to amplify the message. The Guardian headline reads “Humanity has wiped out 60% of animal populations”, while the BBC runs with “Mass wildlife loss caused by human consumption”. No wonder: in the 148-page report, the word “humanity” appears 14 times, and “consumption” an impressive 54 times.

    There is one word, however, that fails to make a single appearance: capitalism. It might seem, when 83% of the world’s freshwater ecosystems are collapsing (another horrifying statistic from the report), that this is no time to quibble over semantics. And yet, as the ecologist Robin Wall Kimmerer has written, “finding the words is another step in learning to see”.

    Although the WWF report comes close to finding the words by identifying culture, economics, and unsustainable production models as the key problems, it fails to name capitalism as the crucial (and often causal) link between these things. It therefore prevents us from seeing the true nature of the problem. If we don’t name it, we can’t tackle it: it’s like aiming at an invisible target.

    Why capitalism?

    The WWF report is right to highlight “exploding human consumption”, not population growth, as the main cause of mass extinction, and it goes to great lengths to illustrate the link between levels of consumption and biodiversity loss. But it stops short of pointing out that capitalism is what compels such reckless consumption. Capitalism – particularly in its neoliberal form – is an ideology founded on a principle of endless economic growth driven by consumption, a proposition that is simply impossible.

    Industrial agriculture, an activity that the report identifies as the biggest single contributor to species loss, is profoundly shaped by capitalism, not least because only a handful of “commodity” species are deemed to have any value, and because, in the sole pursuit of profit and growth, “externalities” such as pollution and biodiversity loss are ignored. And yet instead of calling the irrationality of capitalism out for the ways in which it renders most of life worthless, the WWF report actually extends a capitalist logic by using terms such as “natural assets” and “ecosystem services” to refer to the living world.

    By obscuring capitalism with a term that is merely one of its symptoms – “consumption” – there is also a risk that blame and responsibility for species loss is disproportionately shifted onto individual lifestyle choices, while the larger and more powerful systems and institutions that are compelling individuals to consume are, worryingly, let off the hook.

    Who is ‘humanity’, anyway?

    The WWF report chooses “humanity” as its unit of analysis, and this totalising language is eagerly picked up by the press. The Guardian, for example, reports that “the global population is destroying the web of life”. This is grossly misleading. The WWF report itself illustrates that it is far from all of humanity doing the consuming, but it does not go as far as revealing that only a small minority of the human population are causing the vast majority of the damage.

    From carbon emissions to ecological footprints, the richest 10% of people are having the greatest impact. Furthermore, there is no recognition that the effects of climate and biodiversity collapse are overwhelming felt by the poorest people first – the very people who are contributing least to the problem. Identifying these inequalities matters because it is this – not “humanity” per se – that is the problem, and because inequality is endemic to, you guessed it, capitalist systems (and particularly their racist and colonial legacies).

    The catch-all word “humanity” papers over all of these cracks, preventing us from seeing the situation as it is. It also perpetuates a sense that humans are inherently “bad”, and that it is somehow “in our nature” to consume until there is nothing left. One tweet, posted in response to the WWF publication, retorted that “we are a virus with shoes”, an attitude that hints at growing public apathy.

    But what would it mean to redirect such self-loathing towards capitalism? Not only would this be a more accurate target, but it might also empower us to see our humanity as a force for good.

    Breaking the story

    Words do so much more than simply assign blame to different causes. Words are makers and breakers of the deep stories that we construct about the world, and these stories are especially important for helping us to navigate environmental crises. Using generalised references to “humanity” and “consumption” as drivers of ecological loss is not only inaccurate, it also perpetuates a distorted view of who we are and what we are capable of becoming.

    By naming capitalism as a root cause, on the other hand, we identify a particular set of practices and ideas that are by no means permanent nor inherent to the condition of being human. In doing so, we learn to see that things could be otherwise. There is a power to naming something in order to expose it. As the writer and environmentalist Rebecca Solnit puts it:

    "Calling things by their true names cuts through the lies that excuse, buffer, muddle, disguise, avoid, or encourage inaction, indifference, obliviousness. It’s not all there is to changing the world, but it’s a key step."

    The WWF report urges that a “collective voice is crucial if we are to reverse the trend of biodiversity loss”, but a collective voice is useless if it cannot find the right words. As long as we – and influential organisations such as the WWF, in particular – fail to name capitalism as a key cause of mass extinction, we will remain powerless to break its tragic story.

    This article is republished from The Conversation under a Creative Commons license. It's written by Anna Pigott, Postdoctoral Research Fellow in Environmental Humanities at Swansea University. Read the original article here.

    This story features:
    Read more