- Even though much has been done in New York over the last decade, improvements in the most basic policy outcomes are at best modest.
- Cities should adopt cumulative evaluation to capture the impact of all their food-related action
- Urban food policy evaluation should pay attention to the impacts of “hidden food policies”
Over the last decade, New York has developed a wide array of food policies both at the city and state level. It is now regarded as one of the pioneering cities for urban food policies. However, to date, there had been no systematic effort to take a step back and look at the full picture of what these policies had achieved.
This is why researchers from CUNY Urban Food Policy Institute carried out an analysis of the cumulative body of evidence that had been produced – but never synthetized – over the years. Their work shows the limits of existing evaluation practices in the city, and calls for evaluation to be more fully embedded into urban food policy.
The sobering effect of cumulative evaluation
“We’ve done a lot, but what have we accomplished?” This question was the starting point for this work. Researchers investigated key questions such as:
- Are New Yorkers less food insecure, less vulnerable to diet-related diseases, better off-food wise… than they were 10 years ago?
- And how much of this change is attributable to the policies that have been implemented? Or to other factors such as changes in the socio-economic context?
They looked at 5 key health and social outcomes (such as daily fruit and vegetable consumption or rates of obesity and overweight…) as proxies to capture the intended impacts of food policy. What they found is crucial for all cities around the world. Namely, that even though much has been done in New York, improvements in the most basic policy outcomes, such as the ones regarding equity, are at best modest.
Is this due to policies being too small-scale? Indeed, a lot of programs were set up as experiments, with very little reach, and no clear upscaling strategy.
Or were these policies irrelevant altogether? Interestingly, the significant decrease (-14%) in food insecurity among New Yorkers between 2012 and 2015 could perhaps be better explained by the post-2008 crisis economic revival rather than food policies.
As Nicholas Freudenberg, who co-authored the study, puts in: “after a decade of food policy, we don’t know whether we haven’t implemented at scale, whether we aren’t doing the right things”, or whether we simply need to do more and longer of what we are doing.
Scattered evaluation leads to confusion
How can we account for such a situation? According to the authors, this points to the lack of a proper evaluation framework that would guide actors towards the most effective actions to take.
First, in New York, as in many other cities around the world, policies and programs are evaluated on a one-by-one basis. Furthermore, these studies are often short-term and focus more on implementation than on impact. So it is always challenging to get an idea about the full picture.
Second, the existing monitoring system is not well designed enough. In New York City, an overall evaluation report, called the Food Metrics Report, has been published every year since 2011. It covers 19 indicators and five goals. The city’s food metrics process is an important first step but researchers found that it had many shortcomings, for instance:
- It only uses data mandated by the City Council’s 2011 food metric law, but does not include other sources of data available in other departments or agencies, metrics that could give a more complete picture of progress. .
- It follows indicators because they are available, not because they are the right indicator to follow. For instance, when it comes to understanding whether policies actually creates food systems that support community and economic development, only having indicators about the number of community gardens on city-owned property, or funding awarded to food manufacturers, can prove extremely frustrating. Such an approach can even lead to misleading interpretations. For instance, reporting only the number of stores opened thanks to a specific policy (e.g., the FRESH program) prevents New York City from recognizing that this is smaller than the number of food stores that have closed in targeted neighbourhoods in the same period.
- It only provides citywide results. District level data is not available, making it impossible to track progress in the most deprived neighbourhoods and to go beyond average figures.
All in all, even if analysis of the Food Metrics Report indicators shows progress for half of them, not much can be said from this Report about whether New York inhabitants have healthier food environments now than ten years ago.
Taking evaluation seriously
Where can we go from here? The researchers call for an evaluation framework that focuses on impacts, and on explaining them, rather than on outputs. Such a framework should gather analysis to answer these three questions:
- “What is the cumulative impact of policies?
- What components of these policy portfolios contribute most and least to observed changes?
- What has been the differential impact of these policy portfolios on different populations groups?”
Such an approach would mean:
- Carefully choosing which indicators to follow, and focusing more on outcomes than on output indicators. This will most certainly mean creating new indicators, as the ones that already exist may be too program-specific.
- Involving a wide array of stakeholders in evaluation, as interpreting impacts indicators calls for a dialogue between all. Here, stakeholders should include residents from less well-off neighbourhoods that have the least the voice in the food system – and yet are the most harmed by it -, as well as academics who can usefully help to take a step back and advise on methodologies.
- Integrating the impacts of “hidden food policies”, namely other policies that are not explicitly targeting food issues but that do have an impact on food security or nutritional health. One could think, for instance, of urban planning, gender equality, housing, education or economic development policies. These will help understand if the impact of a policy is due to the policy itself, or to the wider context.
Such a framework will help cities identify and act upon the right policy drivers and structural barriers.
Albane GASPARD – March 2019
Urban Food Futures would like to thank Nicholas Freudenberg for his inputs and comments.
Picture credits: Image by Tanja Schwarz from Pixabay, Icones : The Noun Project
One Reply to ““We’ve done a lot, but what have we accomplished?” Lessons from New York urban food policy”