- One of the more interesting points to come out of discussions at a recent ODI hosted round table on public finance management (PFM) reforms was the weak relationship established between donor support for PFM reforms and the success of reforms. But the keyword here is established. Researchers are facing a major data constraint in doing cross country comparisons of PFM reforms and thus there is not much definitive evidence at the global level of whether a causal link exists – or its direction – between donor support and successful PFM reforms.Public Expenditure and Financial Accountability Assessments (PEFA) and Country Policy and Institutional Assessments (CPIA) are the main data sources for monitoring PFM performance. However, the data is not always comparable across countries and time periods. For example, PEFAs are not carried out annually and few repeat assessments have been done; while PFM is only part of the focus of a CPIA which also looks at everything from [economic management to policies for social inclusion and equity.The data limitation is surprising for several reasons.
Firstly, the central role that PFM plays in translating public resources into development results is now accepted (how states mobilise, allocate and account for public finances is crucial to the relationship between citizens and the sovereign). Secondly, in light of this importance, funding for PFM reforms has increased significantly in the past decade. According to the OECD Creditor Reporting System (CRS), funding for public sector financial management (which includes PFM reforms) has increased almost tenfold in recent years – from roughly US$52 million in 2004 to US$411 million in 2010. While the CRS micro data has inconsistencies and there are issues with reliability, quality and comprehensiveness (see De Renzio, Andrews and Mills) it’s fairly safe to say funding for PFM reform has increased rapidly in the last decade.
The absence of studies looking into whether increased funding has borne results can probably be explained by the weakness of available data. However, one study by Paolo De Renzio, Matt Andrews and Zac Mills, which looked at 100 low and middle income countries, did attempt to examine this whilst also grappling with the question of what determines successful PFM reforms. Somewhat unsurprisingly, they found that domestic economic and political factors are most important in explaining the differences in the quality of PFM systems. This is generally accepted. What is more interesting is thatthey found only a weak positive relationship between donor support and better quality PFM.
Delving further, De Renzio, et al found a significant positive association between the share of total aid provided as budget support to low income countries and the quality of PFM systems –good news to budget support enthusiasts, on the surface at least. However, due to the limited data set and the inconsistency in donor reporting, it wasn’t possible to discern the direction of causality between donor PFM support and the quality of PFM systems. In other words, it is difficult to say whether improvements in PFM systems just reflected the tendency for donors to invest in countries that have already shown capacity to reform, or if donor funding actually caused the improvements.
In the current climate of ‘value for money’, where aid budgets have come under greater scrutiny and there is greater demand to show a link between inputs and outcomes and to show impact quickly, these data weaknesses are particularly problematic. Developing sound PFM systems takes time – over 200 years in the UK, France and US – and practitioners and donors alike need to exercise more patience in achieving successful outcomes. However, the value for money climate does increase the importance of being able to measure subtle changes in PFM systems over time (particularly for cross country analysis) so that progress, however slow, can be shown.
To do this, researchers need systematic, reliable, long term data from both PEFA assessments and donor PFM funding. Currently the PEFA framework has a baseline and at least one repeat assessment for 32 countries but to do detailed cross country analysis we need systematic data for a longer period that covers more countries.
One option would be to encourage repeat assessments, which include an aggregated measure of PFM performance, and make these available in a user friendly format as a public good. This would enable the monitoring of progress in a given country and make cross country comparisons and analysis easier. Aggregate scores that are easy to understand and compare would allow researchers and decision makers in developing countries to make better use of these assessments. However, producing aggregate PEFA scores that are comparable across countries requires standardising PEFA assessment methodologies. The current situation has produced many good quality assessments, but also many bad ones. Streamlining and setting quality standards for aggregated scores will standardise reports and improve their quality and comparability. This will also make monitoring both cross country comparisons and changes of PFM quality within a given country easier and improve the consistency of PFM reports.
More accurate data on donor funding for PFM reforms is also needed, to answer questions around causality and donor impact, and help to show value for money.
Despite the fact that one of PEFA’s goals is to monitor progress on PFM reforms, it does not currently collect data on donor funding. Beyond the problems with data collection conducted by CRS mentioned above, their mandate is to collect data on all aid – not just that used to fund PFM reforms. A proactive approach to data collection, as done by Aiddata, is needed in order to get an accurate picture of donor PFM funding. PEFA is strategically situated to fill this gap
If, after another decade of PFM support, we still don’t know what impact donors support has on PFM reforms because of data weaknesses, justifying these reforms will be more difficult. Not knowing if donor investments represent value for money is not only problematic; it’s unsustainable in the long-term, given the short time horizon of aid. And, despite all the criticisms, PEFA assessments are the best we have in monitoring progress in PFM. It’s crucial they are strengthened, especially if aid for PFM reforms is to be increased or even sustained.
Let’s start by adding to PEFA’s list of responsibilities.
Pfm reforms: still clutching at straws when measuring impact