A new NBER working paper from Amy Finkelstein, Rema Hanna, Ben Olken, Arianna Ornaghi, Sudarno Sumarto and newly minted Nobel Laureate Abhijit Banerjee has been released which uses the much ballyhooed Randomized Control Trial method (or RCT, which I discussed in this post last month) to take a look under the hood of Indonesia’s national health insurance program.

The theory and design of the research is simple. Indonesia has a mandatory national health insurance program. Low-income participants are subsidized with public funds, and the cost of this is offset by payroll taxes on formal sector workers and a mandatory contribution levied on workers in the informal sector. Basically, the system will not work as designed unless lots of relatively healthy people (many of whom are in the large, nebulous and difficult to quantify informal sector) enroll and pay premiums. So getting as many people as possible enrolled is crucial.

The project used randomized control trials to test the effect of subsidies on enrollment. Three randomly selected groups were given 12 months of full subsidies, half subsidies or no subsidies. They also tested whether assistance with registration and better access to information had any effect. The group provided with 12 months of fully subsidized premiums had the highest enrollment rate - about 30%. However, after the subsidies ended, this plunged back down to nearly 10%, or only slightly higher than enrollment for the control group that received no subsidy at all. The researchers calculated that the per unit cost for the government was about the same whether there was a full subsidy or no subsidy, and therefore concluded that subsidies can help but are no “silver bullet.” Assistance with registration boosted enrollment, but the online registration system was so buggy that many people who wanted to register couldn’t. Informational treatment had no effect.

This is an interesting study, and is obviously rigorously structured. But the findings are also somewhat obvious - enrollment went up when the premiums were 100% paid for by someone else. But after the subsidies ended, they plunged back down to slightly higher but around the same level (10%) of the control group, which is nowhere near high enough to fund this program at scale. It also revealed that due to lack of state capacity, on-boarding new participants poses a serious problem. People who wanted to register couldn’t, because the system was buggy. This is the first clue that such a system as this might be too complex for the Indonesian state to execute efficiently.

The paper states: “Taken together, the most important take-away from our results is that large, temporary subsidies can work.” They are careful to qualify this statement, noting that this doesn’t get the program anywhere close to universal coverage. But to me, THAT is actually the most important take-away from this study. It’s that the program as conceived by the Indonesian government is not going to work. It’s badly designed. The best they can hope for, even with large temporary subsidies, is total enrollment of slightly above 10%. This is higher than the non-subsidy group, yes. But not anywhere near high enough to fund this program sustainably.

And ultimately, that is where the criticism of RCTs comes from. This project is great at narrowly evaluating the impact of this specific policy intervention. It convincingly shows that large, temporary subsidies do result in higher enrollment than in the untreated group. But not by much. And it (understandably) pushes this finding as the primary take-away rather than, say, delving into how impractical the design of this program, which relies heavily on an unenforceable individual mandate on informal sector workers, is in the first place. Given the structural constraints in the design of the plan, the government can subsidize initial enrollment all it wants - in my opinion, the only real policy outcome that will produce will be to burn a hole in the public balance sheet. The system needs a top-to-bottom rethinking, rather than narrowly probing the effect of subsidies in the context of a very impractical plan.