Bringing evaluation home

By Lisa Jordan, Executive Director, 9 June 2010

Last week to a packed audience in Brussels at the European Foundation Week (blog), I had an opportunity to share some thoughts with a hundred or so other foundations on evaluation.

In an era of increasing accountability, more scrutiny and shrinking resources, the public expects that foundations will do more with less. How are we evaluating ourselves and our work in light of this new public demand?

Well, to begin with, here at the Bernard van Leer Foundation we did the unthinkable. We abolished the evaluation department.

We have always been diligent about monitoring and evaluation. From the early 1960’s Oscar van Leer insisted upon evaluation, long before it was fashionable. We required grantees to undertake evaluations regularly, to determine whether the projects we fund deliver the results we expect.

Every project was scrutinized by an evaluation department that helped programme staff apply log-frames. Outcomes were predicted, measurable results sought.

These techniques made it relatively easy to know how many children we were serving; how many centres we were supporting and how many families we could reach. We could count these results. Collecting data correctly allowed us to determine how efficiently our euros were applied to children’s needs across a wide range of contexts.

Log-frames are useful instruments for time-bound best practice projects with few moving parts. The Bernard van Leer Foundation aimed to replicate good quality child care across a wide range of countries and for the most part, we have  been very successful in providing good quality care for very disadvantaged kids.

As the Foundation is moving into programmes instead of projects and aims for systemic change and greater impact, the utility of our evaluation arrangements is  increasingly called into question.

Log-frames only measure predicted results. How can a small foundation embrace and respond to the true complexity that impacts kid’s options?

Evaluating individual projects against a predetermined set of outcomes does not help us understand how to create the conditions for sustainable change in young children’s lives. It is almost impossible to determine patterns, to monitor multiple factors of change, to capture the unpredicted outcome, or even to isolate and then replicate the most meaningful factors that made a difference.

Linear models cannot begin to monitor the complex social systems that define children’s opportunities. If there were a blueprint for creating equal opportunities for children, we would all know it by now.

By putting an end to the evaluations department we are shifting our culture of evaluation and monitoring away from a singular focus on the outcomes of a particular project toward the impact of our own strategies.

We are building a new learning culture, one in which evaluation feeds our goals, and provides an opportunity to learn together with partners about the success or failure of our own strategic choices.

Evaluation is now an integral part of the programme plans that are currently under development. It begins in the planning stages we are now in, with all of our new goals.

That means bringing evaluation home to assess our very own theories about how change may occur; to document the conditions of children when we arrive and the conditions of children when we leave, and then again long after we have left.

It means that we will need to develop a thorough understanding of what limits kids from development to their fullest potential; what creates unacceptable levels of violence in their lives; what leads to a lack of services for marginalized children; why kids drop out of school; the impact of their physical environment on them; and then experiment with a wide range of strategies, of interventions and approaches to see what works, what can be built that will change the conditions in which young children live, such that Bernard van Leer Foundation no longer needs to act as a service provider.

It means operating not only on the supply side, but also in building demand, shifting attitudes and practices, and engaging beyond the usual suspects. Programmes require a multitude of moving parts, all of which must be monitored and assessed in real time. No log frame can do all that.

We are very lucky to have a wide array of new tools available that can help us gather baseline data, define the types of outcomes we would like to see, assess everything from safety in and around the home, to attitudes toward violence in children’s lives. Programme officers can use many methodologies to assess the impact of a network, to the effectiveness of income-generating activities, to the political will of politicians to work on behalf of children.

The programme officers are researching a database of methodologies available through the Foundation Centre in New York called TRASI. They also have available research gathered by Philanthropy for Social Justice and Peace to measure long term systemic change and impacts on root causes.

There is no one size fits all model, no one methodology is going to work across all circumstances. The important thing to keep in mind is what Barry Knight at Centris always says: Evaluation has to be owned by the programme officer; with useable results that are robust enough to be reliable and can be read by someone who is not a member of the Royal Statistical Society (like he is). It should be as simple as it can be, but no simpler.

In other words, effective evaluation has to be OURS, not something that we impose on others.

As we move forward implementing our new strategies and using a wide array of  evaluation methodologies you are invited to follow our progress through this website.