What Is an Indicator?
An indicator is a variable that measures one aspect of a program or project that is directly related to the program’s objectives.
Let’s take a moment to go over each piece of this definition.
An indicator is a variable whose value changes from the baseline level at the time the program began to a new value after the program and its activities have made their impact felt. At that point, the variable, or indicator, is calculated again.
Secondly, an indicator is a measurement. It measures the value of the change in meaningful units that can be compared to past and future units. This is usually expressed as a percentage or a number.
Finally, an indicator focuses on a single aspect of a program or project. This aspect may be an input, an output, or an overarching objective, but it should be narrowly defined in a way that captures this one aspect as precisely as possible.
A reasonable guideline recommends one or two indicators per result, at least one indicator for each activity, but no more than 10-15 indicators per area of significant program focus.
Quantitative and Qualitative Indicators
Indicators can be either be quantitative or qualitative.
Quantitative indicators are numeric and are presented as numbers or percentages.
Qualitative indicators are descriptive observations and can be used to supplement the numbers and percentages provided by quantitative indicators. They complement quantitative indicators by adding a richness of information about the context in which the program has been operating. Examples include "availability of a clear, strategic organizational mission statement" and "existence of a multi-year procurement plan for each product offered."
Why Are Indicators Important?
Indicators provide M&E information crucial for decision-making at every level and stage of program implementation.
- Indicators of program inputs measure the specific resources that go into carrying out a project or program (for example, amount of funds allocated to the health sector annually).
- Indicators of outputs measure the immediate results obtained by the program (for example, number of multivitamins distributed or number of staff trained).
- Indicators of outcomes measure whether the outcome changed in the desired direction and whether this change signifies program “success” (for example, contraceptive prevalence rate or percentage of children 12-23 months who received DTP3 immunization by 12 months of age).
What Is a Metric?
An important part of what comprises an indicator is the metric, the precise calculation or formula on which the indicator is based. Calculation of the metric establishes the indicator’s objective value at a point in time. Even if the factor itself is subjective or qualitative, like the attitudes of a target population, the indicator metric calculates its value at a given time objectively.
Here is an example:
- Indicator: Percentage of urban facilities scoring 85-100% on a Quality of Care Checklist
Note that because this indicator calls for a percentage, a fraction is required to calculate it. - Possible metrics:
- Numerator, or top number of the fraction: number of urban facilities scoring 85-100% on a Quality of Care Checklist
- Denominator, or bottom number of the fraction: total number of urban facilities checked and scored
Clarifying Indicators
In many cases, indicators need to be accompanied by clarifications of the terms used. For instance, let's look at the indicator: number of antenatal care (ANC) providers trained.
If such an indicator were used by a program, definitions would need to be included. For example, providers would need to be defined, perhaps as any clinician providing direct clinical services to clients seeking ANC at a public health facility. For the purposes of this indicator then, providers would not include clinicians working in private facilities.
Trained would also need to be defined, perhaps as those staff who attended every day of a five-day training course and passed the final exam with a score of at least 85%.
Another indicator for this program could be percentage of facilities with a provider trained in ANC.
In this example, because the indicator is a proportion or fraction, a numerator and a denominator are needed to calculate it.
- The numerator would be the number of public facilities with a provider who attended the full five days of the ANC training and scored at least 85% on the final exam. Note that the numerator must still specify that the facilities are public and that the providers must have attended all five days and passed the exam in order to be counted. This information need not be included in the indicator itself as long as it is in the definitions that accompany it.
- The denominator would be the total number of public facilities offering ANC services. This requires that this number be obtainable. If it is not known and it is not possible to gather such information, this percentage cannot be calculated.
In this example, it is also necessary to know at which facility each trained provider works. This information could be obtained at the time of the training. If it is not, all facilities would have to be asked if they have any providers who attended the training.
Characteristics of Indicators
A good indicator should:
- Produce the same results each time it is used to measure the same condition or event
- Measure only the condition or event it is intended to measure
- Reflect changes in the state or condition over time
- Represent reasonable measurement costs
- Be defined in clear and unambiguous terms
Indicators should be consistent with international standards and other reporting requirements. Examples of internationally recognized standardized indicators include those developed by UNAIDS and those included in the UNDP Millennium Development Goals.
Indicators should be independent, meaning that they are non-directional and can vary in any direction. For instance, an indicator should measure the number of clients receiving counseling rather than an increase in the number of clients receiving counseling. Similarly, the contraceptive prevalence rate should be measured, rather than the decrease in contraceptive prevalence.
Indicator values should be easy to interpret and explain, timely, precise, valid, and reliable. They should also be comparable across relevant population groups, geography, and other program factors.
Linking Indicators to Frameworks
Let’s use this generic results framework for a family planning program to demonstrate how indicators are linked to frameworks.
For this program, the strategic objective (SO) is to increase the use of family planning services. There are two intermediate results (IRs) feeding into this objective.
- Under the IR of increasing availability of quality services, there are three sub-intermediate results (sub-IRs): services increased, practitioners’ skills and knowledge increased, and sustainable effective management.
- Under the other IR (increasing demand for services), the only sub-IR listed is to improve customer knowledge of family planning.
![](file:///C:/DOCUME%7E1/SMAHMO%7E1/LOCALS%7E1/Temp/moz-screenshot-5.jpg)
|
In order to develop indicators for this framework, the activities to be undertaken by the program must first be recognized. |
This portion of the results framework shows what activities are planned in order for the program to achieve IR1 and its sub-IRs. These activities are:
A. Provision of support and supplies to community-based distributors
B. Expanding family planning services to additional clinics
C. Clinical training for providers
D. The development of a checklist to monitor the quality of care
E. Management training for supervisors
Note that some of these activities can affect several of the sub-IRs.
Next, indicators that measure these activities would be identified. Here you can see the indicators that are linked to the IR and sub-IR1. Other indicators would be linked to the other sub-IRs.
Although it is important to avoid assigning so many indicators that their measurement becomes unachievable, it is risky to rely on a single indicator to measure the significant effects of a project. If the data for that one indicator became unavailable for some reason, it would be difficult to document a significant impact on that result. Therefore, some diversification of indicators tends to strengthen M&E plans.
Linking Indicators to Logic Models
This example depicts how indicators are related to logic models. Here is a logic model for the same activity that was just depicted in the results framework.
Three indicators are linked with this activity:
- Number of providers who have completed clinical training is linked to the output of having trained providers. This indicator can provide information about whether the program is meeting its targets for training providers.
- Percentage of providers scoring 85-100 on the practitioners’ skills and knowledge checklist relates to the intended outcome of improving the knowledge and skills of practitioners.
- Number of facilities providing family planning services links to the intended outcome of increasing the availability of services. The assumption is that increasing the skills and knowledge of more providers will result in more facilities being able to offer services.
Challenges to Selecting Indicators
We will now look at some common challenges to selecting indicators.
Choosing an indicator that the program activities cannot affect
For instance, imagine a program that planned to train health care providers in AIDS prevention and treatment services in an effort to expand access to these services.
The authors of the M&E plan selected the UNAIDS indicator the proportion of health care facilities with adequate conditions to provide care. However, many elements can affect this indicator, such as supervision, availability of supplies and equipment, and the drafting of appropriate treatment protocols. None of these factors would be addressed by the planned training program. In using this global indicator, the planners overlooked the fact that it did not accurately reflect their program activities.
Better indicators would be the number of clinicians trained or the number of facilities with a trained provider.
Choosing an indicator that is too vague
For example, imagine a radio campaign aimed at dispelling specific myths about HIV/AIDS transmission. Although the goal of the campaign is ultimately to increase knowledge about HIV/AIDS, the indicator percentage of the population with knowledge about HIV/AIDS does not specify the exact area of knowledge in question.
A better indicator would be one that measured precisely the objective of the campaign: percentage of the population not believing myths X and Y about HIV/AIDS transmission.
Selecting an indicator that relies on unavailable data
For instance, a program working on drug supply issues selected an indicator that stated percentage of days per quarter that service delivery points have stock-outs of drugs. However, information on stock-outs may not be collected often enough to provide this information.
A better indicator would be percentage of service delivery points that experienced a stock-out of drugs at some time during the last quarter.
Population-level data may also be unavailable or difficult to collect. For example, baseline numbers for immunization coverage in a certain population may be unknown.
Selecting an indicator that does not accurately represent the desired outcome
For instance, if an IR states expanded access to antiretroviral (ARV) treatment for pregnant women to prevent mother-to-child transmission (PMTCT) of HIV, what would an appropriate indicator be?
Would the indicator percentage of women on ARVs who are pregnant be appropriate?
Answer:
No, this would not be an appropriate indicator because it tells us how many women are pregnant out of all women on ARVs, rather than how many HIV-positive pregnant women are on ARVs.
In other words, the numerator for this indicator is the number of women on ARVs who are pregnant and the denominator is the number of women who are on ARVs. Let's say that there were 100 pregnant women on ARVs and a total of 400 women on ARVs. The percentage would be 100/400 or 1/4 or 25%.
If the denominator increased, that is, if more non-pregnant women received treatment for HIV but the number of pregnant women receiving treatment stayed the same, the indicator would decrease. For instance, if 1000 women were on ARVs, the percentage would become 100/1000 or 1/10 or 10%. The indicator would reflect this change, but this change is irrelevant to the desired outcome of the program,which is increasing the number of pregnant women on ARVs.
Similarly, if the indicator increased, for instance if the percentage of women on ARVs who were pregnant out of all women on ARVs went from 25% to 50%, this may be because more pregnant women received ARV treatment (the desired outcome) but it also could be because fewer non-pregnant women were on ARVs, which would not be related to the desired outcome of the program. Because it is not clear which change occurred, this would not be a good indicator to use.
Let's try another example. Would the indicator percentage of people on ARVs who are pregnant women be appropriate?
Answer:
No, this also would not be an appropriate indicator.
Here the numerator is the number of pregnant women on ARVs (let's say it is 100 again) and the denominator is the total number of people on ARVs, including all men and women and children receiving treatment (let's say it's 5000). In other words, this indicator would tell us, of all the people on ARVs, the percentage who are pregnant women is 100/5000 or 1/50 or 2%.
If this indicator increased over time, say from 2% to 20%, it could be because more pregnant women were receiving ARV treatment (1000/5000, the desired effect of the program) but it could also be because fewer people overall were receiving this treatment (100/500) and the number of pregnant women receiving treatment did not actually change.
Similarly, if the indicator decreased, it might be because more people overall were receiving treatment or because fewer women were HIV positive or because there were fewer pregnant women. So the information provided by this indicator would be difficult or impossible to interpret accurately.
Let's try one more example: Would the indicator percentage of HIV-positive pregnant women who are on ARVs be appropriate?
Answer:
Yes, this indicator would provide the needed information.
Here the numerator is the number of HIV-positive pregnant women who are on ARVs and the denominator is the total number of HIV-positive pregnant women.
With this indicator, interpretation is not complicated by factors unrelated to the IR, such as a decrease in HIV prevalence among pregnant women or the number of non-pregnant women receiving ARVs.
Guidelines for Selecting Indicators
Some general guidelines for the selection of indicators are:
- Select indicators requiring data that can realistically be collected with the resources available.
- Select at least one or two indicators (ideally, from different data sources) per key activity or result.
- Select at least one indicator for each core activity (e.g., training event, social marketing message, etc.).
- Select no more than 8-10 indicators per area of significant program focus.
- Use a mix of data collection sources whenever possible. (We will discuss data sources in the next section of this course.)
do not forget to have your say.
No comments:
Post a Comment