Data sources are the resources used to obtain data for M&E activities. There are several levels from which data can come, including client, program, service environment, population, and geographic levels. Regardless of level, data are commonly divided into two general categories: routine and non-routine.
Routine data sources provide data that are collected on a continuous basis, such as information that clinics collect on the patients utilizing their services. Although these data are collected continuously, processing them and reporting on them usually occur only periodically, for instance, aggregated monthly and reported quarterly.
Data collection from routine sources is useful because it can provide information on a timely basis. For instance, it can be used effectively to detect and correct problems in service delivery.
However, it can be difficult to obtain accurate estimates of catchment areas or target populations through this method, and the quality of the data may be poor because of inaccurate record keeping or incomplete reporting.
Non-routine data sources provide data that are collected on a periodic basis, usually annually or less frequently.
Using non-routine data avoids the problem of incorrectly estimating the target population when calculating coverage indicators. Another advantage is that both those using and those not using health facilities are included in the data.
Non-routine data have two main limitations: collecting them is often expensive, and this collection is done on an irregular basis. In order to make informed program decisions, program managers usually need to receive data at more frequent intervals than non-routine data can accommodate.
Different Sources, Same Indicator
Data from different sources can be used to calculate the same indicator, although changes to the metric may be necessary. This illustration depicts one way that routine and non-routine data can be used together to provide for an effective M&E system.
For example, when calculating the coverage rate for the first dose of a Diphtheria-Tetanus-Pertussis (DTP) vaccine:
If population-based survey data are used, the definition could be proportion of children age 12-23 months who were immunized with the first dose of DTP vaccine before age 12 months.
Numerator: Number of children age 12-23 months who were immunized with the first dose of DTP vaccine before age 12 months
Denominator: Total number of children age 12-23 months surveyed
If a routine data source is used, such as clinic records, the definition could be proportion of infants 0-11 months of age in a specified calendar year who were immunized with the first dose of DTP vaccine in that calendar year.
Numerator: Number immunized by age 12 months with the first dose of DTP vaccine in a given year
Denominator: Total number of surviving infants less than 12 months of age in the same year
Data Collection
The M&E plan should include a data collection plan that summarizes information about the data sources needed to monitor and/or evaluate the program.
The plan should include information for each data source such as:
The timing and frequency of collection
The person/agency responsible for the collection
The information needed for the indicators
Any additional information that will be obtained from the source
Data Quality
Throughout the data collection process it is essential that data quality be monitored and maintained. Data quality is important to consider when determining the usefulness of various data sources; the data collected are most useful when they are of the highest quality.
It is important to use the highest quality data that are obtainable, but this often requires a trade off with what it is feasible to obtain. The highest quality data are usually obtained through the triangulation of data from several sources. It is also important to remember that behavioral and motivational factors on the part of the people collecting and analysing the data can also affect its quality.
Some types of errors or biases common in data collection include:
Sampling bias: occurs when the sample taken to represent population values is not a representative sample
Non-sampling error: all other kinds of mismeasurement, such as courtesy bias, incomplete records, or non-response rates
Subjective measurement: occurs when the data are influenced by the measurer
Here are some data quality issues to consider:
Coverage: Will the data cover all of the elements of interest?
Completeness: Is there a complete set of data for each element of interest?
Accuracy: Have the instruments been tested to ensure validity and reliability of the data?
Frequency: Are the data collected as frequently as needed?
Reporting Schedule: Do the available data reflect the time periods of interest?
Accessibility: Are the data needed collectable/retrievable?
Power: Is the sample size big enough to provide a stable estimate or detect change?
Data Use
The term data refers to raw, unprocessed information while information, or strategic information, usually refers to processed data or data presented in some sort of context.
Collecting data is only meaningful and worthwhile if it is subsequently used for evidence-based decision-making. To be useful, information must be based on quality data, and it also must be communicated effectively to policy makers and other interested stakeholders.
M&E data need to be manageable and timely, reliable, specific to the activities in question, and the results need to be well understood.
The key to effective data use involves linking the data to the decisions that need to be made and to those making these decisions.
The decision-maker needs to be aware of relevant information in order to make informed decisions. For example, if sales data from a program to provide insecticide-treated bednets show that the program is successfully increasing bednet distribution, the decision-maker may decide to maintain the program as is. Alternatively, the data may prompt the implementation of a new distribution system and could spur additional research to test the effectiveness of this new strategy compared to the existing one.
When decision-makers understand the kinds of information that can be used to inform decisions and improve results, they are more likely to seek out and use this information.
An indicator is a variable that measures one aspect of a program or project that is directly related to the program’s objectives.
Let’s take a moment to go over each piece of this definition.
An indicator is a variable whose value changes from the baseline level at the time the program began to a new value after the program and its activities have made their impact felt. At that point, the variable, or indicator, is calculated again.
Secondly, an indicator is a measurement. It measures the value of the change in meaningful units that can be compared to past and future units. This is usually expressed as a percentage or a number.
Finally, an indicator focuses on a single aspect of a program or project. This aspect may be an input, an output, or an overarching objective, but it should be narrowly defined in a way that captures this one aspect as precisely as possible.
A reasonable guideline recommends one or two indicators per result, at least one indicator for each activity, but no more than 10-15 indicators per area of significant program focus.
Quantitative and Qualitative Indicators
Indicators can be either be quantitative or qualitative.
Quantitative indicators are numeric and are presented as numbers or percentages.
Qualitative indicators are descriptive observations and can be used to supplement the numbers and percentages provided by quantitative indicators. They complement quantitative indicators by adding a richness of information about the context in which the program has been operating. Examples include "availability of a clear, strategic organizational mission statement" and "existence of a multi-year procurement plan for each product offered."
Why Are Indicators Important?
Indicators provide M&E information crucial for decision-making at every level and stage of program implementation.
Indicators of program inputs measure the specific resources that go into carrying out a project or program (for example, amount of funds allocated to the health sector annually).
Indicators of outputs measure the immediate results obtained by the program (for example, number of multivitamins distributed or number of staff trained).
Indicators of outcomes measure whether the outcome changed in the desired direction and whether this change signifies program “success” (for example, contraceptive prevalence rate or percentage of children 12-23 months who received DTP3 immunization by 12 months of age).
What Is a Metric?
An important part of what comprises an indicator is the metric, the precise calculation or formula on which the indicator is based. Calculation of the metric establishes the indicator’s objective value at a point in time. Even if the factor itself is subjective or qualitative, like the attitudes of a target population, the indicator metric calculates its value at a given time objectively.
Here is an example:
Indicator: Percentage of urban facilities scoring 85-100% on a Quality of Care Checklist
Note that because this indicator calls for a percentage, a fraction is required to calculate it.
Possible metrics:
Numerator, or top number of the fraction: number of urban facilities scoring 85-100% on a Quality of Care Checklist
Denominator, or bottom number of the fraction: total number of urban facilities checked and scored
Clarifying Indicators
In many cases, indicators need to be accompanied by clarifications of the terms used. For instance, let's look at the indicator: number of antenatal care (ANC) providers trained.
If such an indicator were used by a program, definitions would need to be included. For example, providers would need to be defined, perhaps as any clinician providing direct clinical services to clients seeking ANC at a public health facility. For the purposes of this indicator then, providers would not include clinicians working in private facilities.
Trained would also need to be defined, perhaps as those staff who attended every day of a five-day training course and passed the final exam with a score of at least 85%.
Another indicator for this program could be percentage of facilities with a provider trained in ANC.
In this example, because the indicator is a proportion or fraction, a numerator and a denominator are needed to calculate it.
The numerator would be the number of public facilities with a provider who attended the full five days of the ANC training and scored at least 85% on the final exam. Note that the numerator must still specify that the facilities are public and that the providers must have attended all five days and passed the exam in order to be counted. This information need not be included in the indicator itself as long as it is in the definitions that accompany it.
The denominator would be the total number of public facilities offering ANC services. This requires that this number be obtainable. If it is not known and it is not possible to gather such information, this percentage cannot be calculated.
In this example, it is also necessary to know at which facility each trained provider works. This information could be obtained at the time of the training. If it is not, all facilities would have to be asked if they have any providers who attended the training.
Characteristics of Indicators
A good indicator should:
Produce the same results each time it is used to measure the same condition or event
Measure only the condition or event it is intended to measure
Reflect changes in the state or condition over time
Represent reasonable measurement costs
Be defined in clear and unambiguous terms
Indicators should be consistent with international standards and other reporting requirements. Examples of internationally recognized standardized indicators include those developed by UNAIDS and those included in the UNDP Millennium Development Goals.
Indicators should be independent, meaning that they are non-directional and can vary in any direction. For instance, an indicator should measure the number of clients receiving counseling rather than an increase in the number of clients receiving counseling. Similarly, the contraceptive prevalence rate should be measured, rather than the decrease in contraceptive prevalence.
Indicator values should be easy to interpret and explain, timely, precise, valid, and reliable. They should also be comparable across relevant population groups, geography, and other program factors.
Linking Indicators to Frameworks
Let’s use this generic results framework for a family planning program to demonstrate how indicators are linked to frameworks.
For this program, the strategic objective (SO) is to increase the use of family planning services. There are two intermediate results (IRs) feeding into this objective.
Under the IR of increasing availability of quality services, there are three sub-intermediate results (sub-IRs): services increased, practitioners’ skills and knowledge increased, and sustainable effective management.
Under the other IR (increasing demand for services), the only sub-IR listed is to improve customer knowledge of family planning.
In order to develop indicators for this framework, the activities to be undertaken by the program must first be recognized.
This portion of the results framework shows what activities are planned in order for the program to achieve IR1 and its sub-IRs. These activities are:
A. Provision of support and supplies to community-based distributors
B. Expanding family planning services to additional clinics
C. Clinical training for providers
D. The development of a checklist to monitor the quality of care
E. Management training for supervisors
Note that some of these activities can affect several of the sub-IRs.
Next, indicators that measure these activities would be identified. Here you can see the indicators that are linked to the IR and sub-IR1. Other indicators would be linked to the other sub-IRs.
Although it is important to avoid assigning so many indicators that their measurement becomes unachievable, it is risky to rely on a single indicator to measure the significant effects of a project. If the data for that one indicator became unavailable for some reason, it would be difficult to document a significant impact on that result. Therefore, some diversification of indicators tends to strengthen M&E plans.
Linking Indicators to Logic Models
This example depicts how indicators are related to logic models. Here is a logic model for the same activity that was just depicted in the results framework.
Three indicators are linked with this activity:
Number of providers who have completed clinical training is linked to the output of having trained providers. This indicator can provide information about whether the program is meeting its targets for training providers.
Percentage of providers scoring 85-100 on the practitioners’ skills and knowledge checklist relates to the intended outcome of improving the knowledge and skills of practitioners.
Number of facilities providing family planning services links to the intended outcome of increasing the availability of services. The assumption is that increasing the skills and knowledge of more providers will result in more facilities being able to offer services.
Challenges to Selecting Indicators
We will now look at some common challenges to selecting indicators.
Choosing an indicator that the program activities cannot affect
For instance, imagine a program that planned to train health care providers in AIDS prevention and treatment services in an effort to expand access to these services.
The authors of the M&E plan selected the UNAIDS indicator the proportion of health care facilities with adequate conditions to provide care. However, many elements can affect this indicator, such as supervision, availability of supplies and equipment, and the drafting of appropriate treatment protocols. None of these factors would be addressed by the planned training program. In using this global indicator, the planners overlooked the fact that it did not accurately reflect their program activities.
Better indicators would be the number of clinicians trained or the number of facilities with a trained provider.
Choosing an indicator that is too vague
For example, imagine a radio campaign aimed at dispelling specific myths about HIV/AIDS transmission. Although the goal of the campaign is ultimately to increase knowledge about HIV/AIDS, the indicator percentage of the population with knowledge about HIV/AIDS does not specify the exact area of knowledge in question.
A better indicator would be one that measured precisely the objective of the campaign: percentage of the population not believing myths X and Y about HIV/AIDS transmission.
Selecting an indicator that relies on unavailable data
For instance, a program working on drug supply issues selected an indicator that stated percentage of days per quarter that service delivery points have stock-outs of drugs. However, information on stock-outs may not be collected often enough to provide this information.
A better indicator would be percentage of service delivery points that experienced a stock-out of drugs at some time during the last quarter.
Population-level data may also be unavailable or difficult to collect. For example, baseline numbers for immunization coverage in a certain population may be unknown.
Selecting an indicator that does not accurately represent the desired outcome
For instance, if an IR states expanded access to antiretroviral (ARV) treatment for pregnant women to prevent mother-to-child transmission (PMTCT) of HIV, what would an appropriate indicator be?
Would the indicator percentage of women on ARVs who are pregnant be appropriate?
Answer:
No, this would not be an appropriate indicator because it tells us how many women are pregnant out of all women on ARVs, rather than how many HIV-positive pregnant women are on ARVs.
In other words, the numerator for this indicator is the number of women on ARVs who are pregnant and the denominator is the number of women who are on ARVs. Let's say that there were 100 pregnant women on ARVs and a total of 400 women on ARVs. The percentage would be 100/400 or 1/4 or 25%.
If the denominator increased, that is, if more non-pregnant women received treatment for HIV but the number of pregnant women receiving treatment stayed the same, the indicator would decrease. For instance, if 1000 women were on ARVs, the percentage would become 100/1000 or 1/10 or 10%. The indicator would reflect this change, but this change is irrelevant to the desired outcome of the program,which is increasing the number of pregnant women on ARVs.
Similarly, if the indicator increased, for instance if the percentage of women on ARVs who were pregnant out of all women on ARVs went from 25% to 50%, this may be because more pregnant women received ARV treatment (the desired outcome) but it also could be because fewer non-pregnant women were on ARVs, which would not be related to the desired outcome of the program. Because it is not clear which change occurred, this would not be a good indicator to use.
Let's try another example. Would the indicator percentage of people on ARVs who are pregnant women be appropriate?
Answer: No, this also would not be an appropriate indicator.
Here the numerator is the number of pregnant women on ARVs (let's say it is 100 again) and the denominator is the total number of people on ARVs, including all men and women and children receiving treatment (let's say it's 5000). In other words, this indicator would tell us, of all the people on ARVs, the percentage who are pregnant women is 100/5000 or 1/50 or 2%.
If this indicator increased over time, say from 2% to 20%, it could be because more pregnant women were receiving ARV treatment (1000/5000, the desired effect of the program) but it could also be because fewer people overall were receiving this treatment (100/500) and the number of pregnant women receiving treatment did not actually change.
Similarly, if the indicator decreased, it might be because more people overall were receiving treatment or because fewer women were HIV positive or because there were fewer pregnant women. So the information provided by this indicator would be difficult or impossible to interpret accurately.
Let's try one more example: Would the indicator percentage of HIV-positive pregnant women who are on ARVs be appropriate?
Answer:
Yes, this indicator would provide the needed information.
Here the numerator is the number of HIV-positive pregnant women who are on ARVs and the denominator is the total number of HIV-positive pregnant women.
With this indicator, interpretation is not complicated by factors unrelated to the IR, such as a decrease in HIV prevalence among pregnant women or the number of non-pregnant women receiving ARVs.
Guidelines for Selecting Indicators
Some general guidelines for the selection of indicators are:
Select indicators requiring data that can realistically be collected with the resources available.
Select at least one or two indicators (ideally, from different data sources) per key activity or result.
Select at least one indicator for each core activity (e.g., training event, social marketing message, etc.).
Select no more than 8-10 indicators per area of significant program focus.
Use a mix of data collection sources whenever possible. (We will discuss data sources in the next section of this course.)
