1. Overview Metrics Challenges Developing Local Capacity for Metrics
Primer on Metrics
Terminology
Data collection methodologies
Logframe process
Metrics Challenges
Developing Local Capacity for Metrics
Metrics Exercise
WORKED WITH PHIL—UNDERSTAND HE BRIEFED YOU. WHAT DID HE COVER.
HOW MANY OF YOU HAVE EVER DEVELOPED METRICS? WHAT VALUE DID THEY HAVE FOR YOU?
- IS YOUR STRATEGY WORKING—OR ARE WE CONSTANTLY REINVENTING THE FLAT TIRE?
USE SCARCE RESOURCES MORE EFFECTIVELY
BENCHMARKS: HOW MUCH IS ENOUGH?
YOU WILL BE A CONSUMER OF THE DATA BEING PROVIDED TO TEST THE “THEORY OF CHANGE” THAT WAS DEVELOPED BEFORE YOU DEPLOYED
TO AVOID REINVENTING THE FLAT TIRE YOU NEED TO UNDERSTAND HOW THE METRICS WERE DEVELOPED AND HOW TO USE THEM--CONTINUITY
YOU NEED TO DEVELOP A LOCAL CAPACITY FOR M & E
ROES: FIRE AWAY; USAFA
WE WILL CONCLUDE WITH AN EXERCISE THAT WILL INTRODUCE YOU TO A CATALOGUE OF RELEVANT METRICS AND TAKE YOU THROUGH THE STEPS OF THE SELECTION AND TAILORING PROCESS SO YOU WILL BE ABLE TO USE THIS RESOURCE IN YOUR STRATEGIC PLANNING RESPONSIBILITIES.

2. Metrics Terminology: Inputs, Outputs, Outcomes/Impact
Resources required (funds or personnel).
Outputs:
Activities or programs we implement.
Key point: WE control this.
Outcomes/Impact:
The result that the outputs are expected to achieve so that our goals and objectives are met.
Key point: We DON’T control this.
WHICH OF THESE ARE EASIEST TO MEASURE?
WHICH OF THESE ARE MOST LIKELY TO BE USED TO DEMONSTRATE A PROGRAM IS WORKING?
WHAT HAS NTM-A TENDED TO USE TO CLAIM ITS PROGRAMS ARE WORKING?

3. Metrics Terminology: Input, output, or outcome/impact?
- 25,000 police recruits graduated from the police academy in 2012
- 16 MODA personnel are assigned to NTM-A
- Increasing % of respondents who say the ANP is having a “very positive” or “somewhat positive” impact on the level of security in their area.
- 3,000 students completed counter-terrorism training
- More than X% of ANP units in critical locations are housed in adequate facilities
- Less than X % of inspections of construction projects reveal structural defects or misallocation of resources.
WHICH OF THESE ARE OUTCOMES/IMPACT?
THESE ARE FICTITIOUS FIGURES
THE LAST TWO ARE WRITTEN IN A WAY THAT FACILITATES BENCHMARKING.
WHAT IS BENCHMARKING?
HOW DO WE KNOW HOW MUCH IS ENOUGH?
Trend in # of security incidents reported by the ANP per month before and after districts have transitioned from ISAF to ANSF primacy. (By District)

4. Metrics Terminology: Goal and Theory of Change
Impact that an intervention is intended to achieve (comprised of the outcomes of various related programs).
Theory of Change:
Although “theory” is used, this is normally stated as a hypothesis: If X program(s) is/are undertaken, then Y outcome/impact will be achieved.
NOTE: Metrics cannot be developed unless these are articulated.
NTM-A HAS SO 30 PROGRAMS FOR THE MOI. EACH HAS ITS OWN OUTCOMES. TOGETHER THEY ARE INTENDED TO HAVE THE IMPACT IDENTIFIED IN US POLICY. THE IMPACT IS MEASURE BY OVERARCHING INDICATORS.

5. Metrics Terminology: Indicators and Measures
Variable that provides a valid means to determine whether the goal is being met.
Measure:
Empirical data (quantitative or qualitative) that provide a reliable method of assessing the extent to which an indicator is present or absent.
HOW WOULD YOU KNOW WHICH INDICATORS SHOULD BE TAKEN INTO ACCOUNT?
ASSESSMENT OF THE PROBLEM (LOGFRAME): IS THE MILITARY PART OF THE PROBLEM OR PART OF THE SOLUTION?
HOW MANY MEASURES DO YOU NEED FOR AN INDICATOR?
ONE MEASURE IS NECESSARY PER INDICATOR
WHY MIGHT YOU WANT MORE?
BUT IT MAY NOT BE POSSIBLE TO GATHER THE DATA FOR A PREFERRED MEASURE SO ALTERNATIVES ARE PROVIDED.
IF MULTIPLE METHODOLOGIES ARE USED, AND THE RESULTS ARE CONSISTENT WITH EACH OTHER, THIS PROVIDES GREATER CONFIDENCE IN THE VALIDITY OF THE RESULTS

6. Metrics Terminology Goal
Impunity, injustice, and criminalization of state institutions have been diminished to the point that the domestic justice system, supported by a sustainable level of essential international involvement, maintains public order and safety, brings perpetrators of major crimes to justice, holds governing authorities accountable through an independent judiciary, protects fundamental human rights, and applies the law equally, in increasing compliance with international norms and standards.
THIS IS TAKEN FROM MPICE
ASK RIGHT QUESTIONS?
IS THE LEGAL SYSTEM PART OF THE PROBLEM OR PART OF THE SOLUTION?
IS THE PROBLEM A LACK OF CAPACITY OR ABUSE OF POWER?
INSTITUTUIONS ARE BOTH FORMAL AND INFORMAL
What would indicate whether impunity is present? Is decreasing?

7. Metrics Terminology: Indicator
Impunity
Can politically motivated crimes (e.g. Inter-group murder, political violence, terrorism) be resolved by the criminal justice system?
WE HAVE PROVIDED AN INDICATOR OF IMPUNITY THAT IS SENSITIVE TO A SOCIETY EMERGING FROM CONFLCIT.
ALSO RELEVANT FOR CONFLCIT PREVENTION.
COMMENT ON TAILORING:
DEFINE WHICH POLITICALLY MOTIVATED CRIMES ARE RELEVANT IN THE GIVEN CONTEXT.
IN AFGHANISTAN, WHAT MIGHT YOU WANT TO ADD TO THE LIST?
- DEFINE WHICH GROUPS MUST BE TAKEN INTO ACCOUNT
HOW WOULD YOU MEASURE THIS?

8. Metrics Terminology: Measures
Impunity
Can politically motivated crimes (e.g. Inter-group murder, political violence, terrorism) be resolved by the criminal justice system?
Ratio of incidence of inter-group murder, political violence, and terrorism to investigations, prosecutions, and convictions for these crimes
Perceptions of law enforcement officials and victims of the crimes cited above that suspects involved are untouchable and that cases are abandoned for this reason
WHAT DATA COLLECTION METHODOLOGY WOULD YOU USE?

9. Data Collection Methodologies
Methodologies should be selected because they are best suited to collect the data required for a given measure
Data for MPICE are collected using one of four methodologies:
Content Analysis (CA)
Quantitative Data (QD)
Survey and Polling Data (S/PD)
Expert Knowledge (EK)
Q- If public perceptions of the armed forces are desired? This is measured by surveys;
Survey/Polling Data: RANDOM SAMPLE TO Measure perceptions
Q- If hate speech is a concern? This is best measured by Content analysis;
Content analysis: Uses frequency of specific (boolean) phrases in print media as proxies to identify increasing or decreasing sentiments.
Q-If performance/corruption of an institution is needed? Use expert knowledge.
Expert Knowledge: Assemble PANEL OF experts to conduct focused assessment of an issue like performance of an institution
Q- If the number of arrests or convictions for terrorist acts is appropriate? This is a statistic.
Quantitative Data: May be limited availability initially.
SEE MPICE USERS HANDBOOK FOR EXPLANATION
NOT FOCUS GPS: Not replicable; useful for analysis

10. Logframe Metrics Process
Assess the “problems” that stand in the way of achieving your objectives (e.g., Is the security sector part of the problem or part of the solution?)
Identify factors you don’t control (barriers and dependencies)
State what needs to be done to address those problems (i.e., a theory of change: If X outputs, and no Z barriers, and A dependencies are met, then Y outcomes)
Select indicators and measures
Collect relevant data to establish baseline
Analysis of change after program implementation
Reallocate resources and adjust theory of change based on results
BEFORE YOU CAN DEVELOP METRICS, WHAT DID YOU NEED?
WHAT OBJECTIVES YOU ARE STRIVING TO ACHIEVE
(AND HOW) WHAT IS THE THEORY OF CHANGE
WHAT ACTIONS BY OTHERS DO YOU DEPEND ON TO ACCOMPLISH YOUR GOALS
IF CONCERNED WITH STABILIZATION THEN BEGIN WITH AN ASSESSMENT OF THE DYNAMICS THAT FUEL CONFLICT
DOES THE SAME THEORY OF CHANGE APPLY IF THE ARMED FORCES ARE THE PROBLEM RATHER THAN THE SOLUTION?
IF RESOURCES WILL NOT BE REALLOCATED ON THE BASIS OF THE ANALYSIS OF THE RESULTS OF YOUR DATA, THEN THE ENTIRE EXERCISE IS NOTHING BUT AN EXERCISE IN SELF-FLAGILLATION.

11. Metrics Challenges: Using the Wrong Metric
“The development of Afghan National Army and National Police in isolation of dealing with other weaknesses of the Afghan Government such as the rule of law, corruption, the influence of non-state power brokers seriously undermines the effectiveness of the force no matter how numerically strong it may be. Efforts should be aimed at consolidation of institutions to curb the influence of power brokers. Otherwise, institutions will continue to serve personal and group interests of non-state actors.”
Presentation to USIP by Ali Jalali, former Afghan Minister of Interior
ANECDOTE: ANTI-CRIME POLICE: PRESUMED MEASURE—CRIME CLEARANCE RATE; I ASKED WHAT THEIR BIGGEST IMPEDIMENT/BARRIER WAS. ANS: CAN’T ARREST WARLORDS. HAD WRONG MEASURE SINCE THIS WOULD HAVE MEASURED INCREASING ARRESTS OF THE DISENFRANCHIZED WHILE THE WARLORDS ENJOYED IMPUNITY. WHAT WOULD THE IMPACT OF SUCH A STRATEGY BE ON LEGITIMACY OF THE KARZAI GOVT?
HOW TO MEASURE? WE HAD POMLTs ASSIGNED TO THE DISTRICTS. SURVEY THEM. HOW OFTEN IS THE ANTI-CRIME POLICE PREVENTED FROM INVESTIGATING POWERFUL LOCAL ELITES? LIKERT SCALE. AN EXAMPLE OF EXPERT KNOWLEDGE
Increasing trend in the # of “no” responses to a survey of Police Operational Mentors, Liaisons and Trainers (POMLETs??): “In your experience, are there certain individuals in the community where you are assigned who are suspected of criminal activity but who cannot be investigated or arrested.”
(Source: Survey of POMLTs)
Increasing trend in the % of POMLTs who respond “seldom happen” and “only happen sometimes” to the survey question, “If yes (to the above question), does this seldom happen, only happen sometimes, or frequently happen?”

12. Solution to Metrics Challenge : Measuring Conflict Transformation
Conflict Drivers
Motivations and means
for violent conflict
Institutional Performance
Capacity of indigenous institutions to
overcome conflict peacefully
ASSESRTION: ONE OF GREATEST MISTAKES IS BUILDING CAPACITY WITHOUT ASKING IF THE POLICE OR MILITARY ARE PART OF THE PROBLEM (EG Iraq UNTIL 2006).
WHAT SHOULD THE PRIORITY FOR INTERNATIONAL ACTION BE?
THE PURPOSE OF THE MPICE FRAMEWORK IS TO MEASURE CONFLCIT TRANSFORMATION
WHAT IS CONFLCIT TRANSFORMATION? IT WAS ADOPTED BY S/CRS AND USED TO DEVELP THE CAMPAIGN PLAN
A SUCCESSFULTRANSITION FROM WAR TO PEACE ENTAILS TWO SIMULTANEOUS PROCESSES: DIMINISHING THE MEANS AND MOTIVATIONS THAT ARE RESPONSIBLE FOR VIOLENT CONFLICT IN A PARTICULAR SOCIETY WHILE NURTURING INSTITUTIONS (FORMAL AND INFORMAL) THAT PROVIDE PEACEFUL ALTERNATIVES FOR RESOLUTION OF DISPUTES, ESPECIALLY THE COMPETITIVE PURSUIT OF WEALTH AND POWER.
OTHER TERMS USED FOR INSTITUTIONAL PERFORMANCE ARE “MITIGATORS OF CONFLICT” OR RESILIENCE.
PEACE BECOMES VIABLE WHEN THERE IS A CONVERGENCE BETWEEN THE TWO FACTORS. VIABILITY DOES NOT MEAN EXIT, HOWEVER, BECAUSE WITHOUT SOME FORM OF SAFEGUARD AND CONTINUING INTERNATIONAL ENGAGEMENT, THE PROCESS WILL NOT BE SUSTAINABLE. VIABLE PEACE IS THE POINT WHERE THE LEVEL OF INTERNATIONAL INVOLVEMENT -- MEASURED IN TERMS OF RESOURCE EXPENDITURES-- IS SUSTAINABLE OVER THE LONGER TERM. TRANSITION IN THE NATURE OF THE INTERNATIONAL REGIME RFEQUIRED TO MANAGE THE PROCESS UNTIL SELF SUSTAINING.
FROM QVP PUBLISHED IN 2005
CONFLCIT TRANSFORMATION ADOPTED BY S/CRS AS BASIS FOR STRATEGIC PLANNING
WE WERE ASKED TO DEVELOP A METRICS FRAMEWORK; TEAMED WITH PKSOI AND ARMY CORPS OF ENGINEERS

13. Metrics Challeges
Politicization: Will you be allowed to report disappointing results?
Wrong Objectives: Does the mandate or policy goal actually confront the essence of the conflict?
Attribution: Did your strategy cause the outcome?
Timeframes: Lag time between project completion and desired effect?
Role of Locals: Should locals (politicians, civil society, masses) determine strategic goals and how to measure them?
Cost and Availability of Data: Can the data be collected and at what cost?
POLITICIZATION: AFFECTS THE MEASURES THAT ARE SELECTED (WHY DO OUTPUTS TEND TO BE USED?) AND ANALYSIS OF RESULTS. USG REFUSED TO USE METRICS FOR IRAQ SURGE
SOLUTION: BETTER TO ESTABLISH CORE SET OF METRICS AND USE NEUTRAL ENTITY.
WRONG OBJECTIVES: MANDATES/PEACE AGREEMENTS ARE COMPROMISES THAT MAY NOT REFLECT THE REAL ISSUES OR IDENTIFY WHAT REALLY NEEDS TO BE DONE TO RESOLVE A CONFLCIT (BOSNIA/RWANDA-ARUSHA). SOLUTION: PROPER ASSESSMENT; ASK THE RIGHT QUESTIONS (EG IS LEGAL SYSTEM PART OF THE PROBLEM OR PART OF THE SOLUTION?
ATTRIBUTION: MPICE IS NOT FOR EVALUATING PROGRAMS; IS FOR EVALUATING STRATEGIES. SOLUTION: IS CONFLICT BEING TRANSFORMED OR NOT? IS GAP BETWEEN DRIVERS AND INSTITUTIONAL CAPACITY NARROWING? NEED TO KNOW IF ENDS/GOALS ARE CORRECT; IF THEORY OF CHANGE (WAY) IS RIGHT; IF HAVE ADEQUATE RESOURCES
TIMEFRAMES: LAG TIME BETWEEN THE COMPLETION OF PROJECTS AND CHANGE IN BEHAVIOR OR ATTITUDE?
SOLUTION: A MATTER OF ANALYSIS/EXPECTATIONS
ROLE OF LOCALS: WHAT IF THE POLITICIANS ARE THE PROBLEM AND THEIR TRANSFORMATION IS WHAT YOU NEED TO MEASURE?
SOLUTION: USE SEPARATE METRICS FOR DRIVERS AND CAPACITY
COST TOO MANY METRICS

14. Developing Local Metrics Capacity
Suggested measures for NTM-A Overarching Indicator:
“MOI is able to translate strategy and policy into operational delivery and delivery activity through clear lines of command and control and measure the effect of that activity on the achievement of ministerial goals.”
- MOI has developed a monitoring and evaluation system for its National Police Plan and Strategy that allows for a comprehensive assessment of progress by all ANP pillars, Provinces, and Districts at least annually.
(Source: NTM-A advisors in discussion with ANP counterparts)
- Fewer than X% of audits of Force Readiness reports (personnel and equipment), personnel status reports, and facilities status reports indicate major discrepancies.
(Source: An audit system will need to be implemented. This will be a crucial task for NTM-A to monitor.)

15. Exercise Transformational Effect to be Measured: Population Security
Afghans feel free from violence and coercion by insurgents, criminals, and terrorists and increasingly trust security forces to protect them, enabling resistance to the insurgency and support for GIRoA.
Directions:
Using the MPICE Handbook for a Safe and Secure Environment, select and tailor 3-4 indicators and associated measures for this goal.

16. Selection Process Match your strategic goal to generic MPICE goals
Select indicators that are valid representations of your goals
Identify measures that tap the essence of the indicator in your context
Can the data required actually be collected?
If not, use a different measure or methodology
WHAT GOALS ARE BEING SOUGHT?
WHAT IS SALIENT TO ACHIEVING THE GOALS IN A GIVEN ENVIRONMENT?
The MMEs/goals may not be the same as the MPICE end states/sectors—look in ROL for example.
Trend in # of security incidents reported by the ANP per month before and after districts have transitioned from ISAF to ANSF primacy. (By District)

17. Tailoring Process
Word the measures you have selected to match the context
In Haiti “violent armed groups” equate to gangs
Identify the level at which the data should be collected (e.g. national, regional, provincial)
Determine how often the data should be collected (monthly, quarterly, annually)