How Donor-Funded Programs Measure Health Technology Impact

Donor-funded health technology programs produce a lot of data. They also produce a lot of reports about that data. What they produce far less reliably is evidence that the technology actually changed health outcomes. The gap between deploying a digital health tool and proving it made people healthier is wider than most program documents acknowledge, and the measurement methods used to bridge that gap vary enormously depending on who is asking and who is paying.

Global digital health spending in low- and middle-income countries now runs into the billions annually. USAID alone allocated over $200 million to digital health activities across its portfolio in fiscal year 2024, according to its Digital Health Position Paper update. The Global Fund, Gavi, the Bill & Melinda Gates Foundation, and bilateral donors from Europe and Asia add substantially more. All of them want to know what their money accomplished. None of them measure it the same way.

"We have more digital health pilots than we have rigorous evaluations of digital health pilots. The evidence base has not kept pace with the enthusiasm." — Dr. Alain Labrique, Director of Digital Health, WHO, at the 2024 Global Digital Health Forum

How donors currently evaluate health technology investments

The measurement frameworks donors use depend on what they are trying to prove and to whom. A bilateral donor reporting to its parliament needs different evidence than a private foundation reporting to its board. A multilateral institution like the WHO needs evidence that generalizes across countries. These different accountability structures produce different measurement approaches, and sometimes contradictory conclusions about the same technology.

USAID's evaluation approach leans heavily on implementation research — studies that examine whether a technology works in real operational settings, not just controlled trials. Their Digital Health Vision 2020-2024, updated in 2024, specifically called for "pragmatic evidence" over randomized controlled trials, recognizing that RCTs in digital health often test conditions that bear little resemblance to actual deployment. When a research team evaluates an mHealth tool in a well-funded pilot site with trained staff and reliable connectivity, the results may say more about the pilot conditions than about the technology.

The Global Fund takes a different approach. Its results framework tracks coverage indicators — how many people were reached, how many tests were administered, how many treatments were initiated. Technology is measured by whether it increased throughput in existing health programs. If a digital tool helped a country screen 15% more TB suspects in a quarter, that counts. Whether the tool itself was the cause, or whether the additional funding and attention that came with it were the real drivers, is harder to isolate.

The Gates Foundation has invested heavily in building measurement capacity at the country level. Its Primary Health Care programs in Nigeria, Ethiopia, and India include embedded evaluation components that attempt to attribute health gains specifically to digital interventions. Dr. Orin Levine, who led the foundation's vaccine delivery strategy before transitioning to broader health systems work, described the challenge bluntly in a 2023 interview with Devex: "Attribution is the hardest problem in global health evaluation. Everything is confounded by everything else."

The DALY question

Disability-adjusted life years remain the standard currency for comparing health interventions across categories. One DALY represents one year of healthy life lost. Donors use cost-per-DALY-averted to rank interventions and allocate resources. A 2023 systematic review published in JMIR by researchers at the University of Cape Town examined cost-effectiveness analyses of digital health interventions and found that most fell below the WHO threshold of one to three times GDP per capita per DALY averted, making them technically cost-effective.

But the DALY calculations for technology interventions carry assumptions that researchers themselves flag as problematic. A smartphone-based screening tool does not directly avert DALYs. It identifies people who need treatment. Whether those people actually receive treatment, adhere to it, and experience better outcomes depends on the health system surrounding the tool. The DALY attribution chain from "app detected a condition" to "patient lived longer" passes through referral systems, drug supply chains, clinical capacity, and patient behavior. Each link introduces uncertainty.

A 2024 analysis in the Bulletin of the WHO examined digitalization of health care in low- and middle-income countries and found that while fiscal benefits were demonstrable in some settings, the evidence for direct health impact remained "fragmented and context-dependent." The authors, including researchers from the Barcelona Institute for Global Health, recommended that donors adopt a tiered evaluation approach that separates process outcomes (did the technology work?) from health outcomes (did patients get better?).

Measurement frameworks in practice

Several structured frameworks have emerged to standardize how donors evaluate health technology. None has achieved universal adoption, but each reflects a different theory about what matters.

Framework	Developed by	Focus	Strengths	Limitations
WHO Monitoring & Evaluation of Digital Health Interventions (2016)	WHO	Process and outcome indicators for digital health	Standardized indicator set, widely referenced	Focused on individual interventions, not system-level effects
DHIS2 Integration Assessment	University of Oslo / HISP	Data flow and system interoperability	Measures real integration with national health information systems	Does not measure health outcomes directly
RE-AIM Framework	Virginia Tech (adapted for digital health)	Reach, Effectiveness, Adoption, Implementation, Maintenance	Captures sustainability and scale dimensions	Requires longitudinal data most programs do not collect
mHealth Evidence Reporting and Assessment (mERA)	WHO mHealth Technical Evidence Review Group	Reporting standards for mHealth evaluations	Improves comparability across studies	Checklist approach, does not assess quality of evidence
Digital Health Impact Framework	USAID / PATH	End-to-end impact pathway from deployment to outcomes	Traces attribution chain explicitly	New framework, limited track record
Results-Based Financing indicators	World Bank	Payment tied to verified outcomes	Aligns incentives with measurement	Can distort reporting if payments depend on results

WHO's approach

WHO published its monitoring and evaluation guidance for digital health interventions in 2016, with updates through the classification of digital health interventions in 2018. The framework identifies indicator categories spanning inputs (funding, infrastructure), processes (adoption rates, data quality), outputs (screenings completed, messages sent), and outcomes (changes in health behaviors or clinical indicators). The practical challenge is that most implementing organizations report on inputs and outputs because those are easiest to count. Outcome data requires follow-up periods and comparison groups that short funding cycles rarely support.

Results-based financing

The World Bank has promoted results-based financing as a way to tie donor payments to verified health outcomes. In RBF programs, a government or implementing partner receives payment after an independent verification agent confirms that services were delivered or outcomes were achieved. A 2020 Cochrane review led by Soeters and colleagues found mixed evidence on RBF effectiveness, with some programs improving service utilization and others showing no significant impact compared to traditional input-based financing.

When technology is layered onto RBF programs, measurement gets more specific. The technology becomes a data collection instrument as much as a health intervention. In Rwanda, where RBF has been national policy since 2008, community health workers use digital tools to register services that trigger facility-level payments. The technology does not just deliver health services — it verifies them. This dual role creates interesting incentive dynamics. If the measurement tool is also the payment trigger, the reliability of the measurement matters financially, not just scientifically.

Where measurement breaks down

Three recurring problems undermine health technology impact measurement in donor-funded programs.

Short evaluation windows. Most donor-funded programs operate on three-to-five-year cycles. Technology deployment typically consumes the first 12 to 18 months (procurement, customization, training, rollout). That leaves limited time for the kind of longitudinal outcome data that would demonstrate health impact. By the time the technology is working well, the evaluation period is nearly over.

Dr. Patricia Mechael, co-founder of HealthEnabled, has written about this mismatch repeatedly. Her 2019 analysis for the Broadband Commission's Working Group on Digital Health noted that "the evidence cycle and the funding cycle are fundamentally out of sync," a problem that no amount of methodological improvement can solve without structural changes to how donors fund evaluations.

Attribution in complex systems. Health technology rarely operates in isolation. A new screening app gets deployed alongside new training, new supervision structures, new supply chain processes, and often new funding for the underlying health services. Isolating the technology's contribution from the broader program investment is methodologically difficult and sometimes impossible.

The 2023 RTI International ranking of 93 health interventions for cost-effectiveness in low- and middle-income countries found that digital health interventions were difficult to compare with traditional interventions because "the active ingredient in a digital health intervention is often the behavior change or service delivery improvement it enables, not the technology itself." This is not a flaw in the technology. It is a fundamental challenge in measuring tools that work through other interventions rather than directly on patients.

Data quality at the point of collection. Community health workers collecting data on smartphones in rural settings face connectivity gaps, device problems, time pressure, and sometimes insufficient training. A 2020 systematic review in Human Resources for Health by Maryse Kok and colleagues at the Royal Tropical Institute documented CHW attrition rates ranging from 3% to 77% annually across Sub-Saharan Africa. High turnover means the people collecting measurement data are frequently the least experienced members of the health workforce.

What emerging approaches look like

Several donor-funded programs are testing newer measurement strategies that attempt to address these weaknesses.

Embedded evaluation designs

Rather than conducting evaluations as separate research projects that run alongside implementation, some programs are building evaluation into the technology itself. Medic's Community Health Toolkit, used in over 14,000 health facilities across dozens of countries, generates continuous data streams that can be analyzed for trends without mounting a separate evaluation study. The data already exists because the health workers are using it for clinical decisions. The evaluation layer sits on top of operational data rather than creating parallel data collection.

Stepped-wedge designs

Randomized stepped-wedge trials, where different sites receive the technology at different times, allow programs to generate comparison data without withholding the intervention from anyone permanently. The D-tree International program in Tanzania used a stepped-wedge approach to evaluate its digital health tools for community health workers, comparing health indicators in districts that had received the technology with districts still waiting for rollout.

Smartphone-based screening as a measurement tool

Contactless health screening technologies add another dimension to this picture. When a smartphone can capture vital signs — heart rate, respiratory rate, blood oxygen levels — without additional equipment, the screening process itself generates structured health data at population scale. Programs deploying camera-based vital signs tools like those developed by Circadify can collect population-level biometric data as a byproduct of routine screening, potentially transforming measurement from a separate research activity into an embedded function of service delivery. More on this approach at circadify.com/blog.

Linking to national health information systems

The most promising measurement approaches connect technology-generated data to national DHIS2 platforms, allowing donor-funded program data to be validated against routine health system data. When a digital screening tool's data flows into the same system the ministry of health uses for planning, the evaluation question shifts from "did the technology work?" to "did district health indicators improve after deployment?" This is a harder question to answer definitively, but it is the right question.

Current Research and Evidence

The evidence base for health technology impact measurement is growing but remains uneven. A 2024 article in the Bulletin of the WHO by researchers including those from the Barcelona Institute for Global Health examined fiscal and health impacts of digitalization across LMICs, concluding that economic benefits often preceded measurable health benefits. Digital tools reduced costs and improved efficiency before they demonstrably improved patient outcomes, which makes sense given the attribution chain described above.

The Global Health Technologies Coalition published a 2024 analysis finding that $46 billion in U.S. public spending on global health R&D between 2007 and 2022 generated an estimated 200,000 jobs in the U.S. alone, alongside its global health impact. This kind of dual-benefit analysis is increasingly common as donors face domestic pressure to justify international health spending.

PATH's 2020 report "The Journey to Scale" documented that fewer than 5% of digital health pilots in LMICs successfully transitioned to national-scale deployment, a statistic that raises obvious questions about whether impact measurement at the pilot stage tells us much about what happens at scale. A technology that produces excellent results in a well-supported pilot may produce very different results when deployed across an entire country's health system.

The Future of Donor-Funded Health Technology Evaluation

The direction is toward integration — measurement built into operations rather than bolted on as a separate workstream. The WHO's 2023 global strategy on digital health explicitly calls for countries to develop national monitoring frameworks that include digital health indicators alongside traditional health system metrics. USAID's updated digital health strategy pushes implementing partners to design programs with measurement as a core function, not an afterthought.

Results-based financing will likely expand, creating stronger incentives for accurate measurement. The World Bank's health results innovation trust fund is supporting experiments in outcome-based payment for digital health programs in several countries. If payments depend on verified outcomes, the quality of measurement becomes a financial imperative.

And the tools themselves are getting better at generating evaluable data. Smartphone-based health screening, AI-assisted diagnostics, and integrated data platforms are producing richer datasets than previous generations of mHealth tools. The question is whether donor evaluation frameworks will evolve fast enough to use this data well. Right now, many are still counting outputs — number of devices distributed, number of health workers trained, number of screenings completed. The shift to measuring outcomes and impact requires not just better data, but better analytical capacity at the country level and longer evaluation timelines than most funding cycles permit.

Frequently Asked Questions

What metrics do donors use to evaluate health technology programs?

Donors use a mix of process metrics (adoption rates, data quality, system uptime), output metrics (screenings completed, patients referred), and outcome metrics (changes in health indicators, cost per DALY averted). The specific mix depends on the donor's evaluation framework. USAID emphasizes implementation research and pragmatic evidence. The Global Fund tracks coverage indicators. The World Bank increasingly ties payments to verified outcomes through results-based financing.

Why is it hard to measure the impact of health technology specifically?

Health technology rarely operates alone. A digital screening tool gets deployed alongside new training, supervision, supply chain improvements, and often additional funding for underlying health services. Isolating the technology's specific contribution from the broader program investment requires sophisticated research designs and longitudinal data that most program timelines do not support.

How long does it take to see measurable health impact from technology deployment?

Most experts estimate 18 to 36 months from full deployment to measurable health outcome changes, depending on the condition being addressed and the strength of the referral system. Technology deployment itself typically takes 12 to 18 months, meaning a five-year program may have only one to two years of outcome data. This timing mismatch between funding cycles and evidence generation is one of the most persistent problems in the field.

What is changing about health technology evaluation?

The field is moving toward embedded evaluation — using operational data from the technology itself rather than conducting separate research studies. Stepped-wedge trial designs, integration with national DHIS2 platforms, and results-based financing are all pushing measurement closer to real-world operations and further from controlled research conditions. Smartphone-based screening tools that generate structured health data as a byproduct of service delivery represent a promising direction.