"Not everything that counts can be counted." MEPA Qualitative Tools "Not everything that counts can be counted."

SESSION OBJECTIVES At the end of the session, the participants are expected to: Discuss the basics and nuances of using qualitative tools in MEPA; Explain how to prepare and use appropriately tools such as: Interview (KII & IDI), Most Significance Change (MSC) Story Technique MSC, Participatory Rapid Appraisal (PRA), and Focus Group Discussion (FGD); Develop deeper understanding on the importance of .both quantitative and qualitative tools in preparing Completed Staff Work (CSW) for MEPA in order to provide story to quantitative data.

ACTIVITY: Self-Assessment Directions: Assess how knowledgeable and competent you are in using qualitative tools in the preparation of CSW for MEPA, using the following rating scale: 1 – Developing 3 – Advanced 2 - Maturing No. Qualitative Tools I have adequate knowledge and have already used: Pre-test Post-test 1 Key Informant Interview (KII) 2 3 In-depth Interview (IDI) Most Significance Change (MSC) 4 Focus Group Discussion (FGD) 5 Participatory Rapid Appraisal (PRA) Total Score Level of Competence

How is your pre-test result? Directions: Sum-up the total score and divide it by 5. 2. Determine your level of competence based on the table below: Range Descriptive Rating Interpretation 1.0-1.49 Developing You are still in the process of acquiring the skill; hence, you need Technical Assistance. 1.50-2.49 Maturing You have acquired the skill to a larger extent, but you need to enhance it further. 2.50-3.0 Advanced Congratulations! You have fully acquired the skill to a largest extent, you are very much ready to prepare CSW for MEPA.

ANALYSIS How did you find the activity? What is your level of competence in terms of using qualitative tools? What facilitating/hindering factors contributed for having such level? Which specific tool do you need more TA? Why do you need to have high level of competence in all these qualitative tools?

Qualitative Monitoring Data-gathering techniques that are focused on the significance of observations made in a study rather than the raw numbers themselves.

4 (Quantity) Happy (Quality) Kids Quantitative: numbers breadth generalizability Qualitative: words depth specific Remember, "Not everything that counts can be counted."

Quantitative methods vs Qualitative methods Surveys Focus groups Tests Unstructured interviews and observations Questionnaires Rapid Appraisal

Mixing Quantitative Methods & Qualitative Methods Both quantitative and qualitative methods have their strengths and weaknesses. What qualitative methods lose on reliability they gain in terms of validity.  They provide a more in depth and rich description of change and learning. Quantitative methods are reliable and can be projected to a large population but are weak in terms of describing actual experience. Combining quantitative and qualitative research provides a more holistic picture of outcomes and impacts achieved.

Mixing Quantitative Methods & Qualitative Methods Quantitative Monitoring Mid Term Results Region A Physical Outputs Accomplishment 30% Financial Funds Utilization: 18% Funds Liquidated: 5% Qualitative Monitoring Mid Term Results Region A During the Orientation on DO 62 I was full of questions. Why do we have to give so much emphasis on IP education when they do not show interest in going to school. Going to the immersion in Mindoro was an eye opener, here the IPs were both teacher and student. They learned mainstream concepts together with their own indigenous knowledge. The river, land and air were all part of the learning environment. After the immersion, the Region decided to conduct a second round of orientations on DO 62. With the benefit of our experience we understood the principles behind the order. We also decided to repeat the proposal preparation training to emphasize community participation. As a result, although late the second round of proposals submissions were much better than the first.

Mixing Quantitative Methods & Qualitative Methods Use QUALITATIVE Research To: look for a range of ideas and feelings about something understand different perspectives between groups and categories of people uncover underlying motivations and factors that influence decision making and opinions Use QUANTITATIVE Research To: find whether there is consensus on a particular issue projects results to a larger population test specific hypotheses and examine specific relationships

Qualitative Data Gathering Qualitative research takes place in the real world, as opposed to the laboratory, and deals with how people give meaning to their own experience. These often involve face-to-face interactions between researcher and participant The researchers need to be flexible and sensitive to the needs of the social context within which the data is obtained.

Qualitative Data Analysis Then it is followed by an attempt to interpret the behaviour and the meanings that people have given to their experience. The data is then analyzed and interpreted. To look for themes rather than than trying to confirm a hypothesis. The objective of qualitative research is to describe and possibly explain events and experiences.

Participatory Rapid Appraisal (PRA) Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal

Participatory Rapid Appraisal (PRA) Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal

Participatory Rapid Appraisal (PRA) Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal

Participatory Rapid Appraisal (PRA) Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal

Participatory Rapid Appraisal (PRA) Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal

Participatory Rapid Appraisal (PRA) Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal

Participatory Rapid Appraisal (PRA) Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal

Participatory Rapid Appraisal (PRA) Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal

Participatory Rapid Appraisal (PRA) Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal

THE KEYS TO KEY INFORMANT INTERVIEW (KII) Who is a Key Informant? - Someone who can ‘unlock’ key information for you. What is KII? - Discussion between an individual respondent and an interviewer which can be done via telephone or face-to-face ranging from informal to highly structured chats. Source: Performance M&E, USAID Center for Dev’t Info. And Eval; http://pdf.dec.org/pdf _docs/PNABS541.PDF

THE KEYS TO KEY INFORMANT INTERVIEW (KII) Purpose of KII? Collect qualitative, in-depth information from a wide range of people about a pressing issue to understand beliefs, motivations and sensitive topics Types of KII: Informal, conversational interview General, guided interview Standardized, open-ended interview Closed, fixed-response interview Source: Performance M&E, USAID Center for Dev’t Info. And Eval; http://pdf.dec.org/pdf _docs/PNABS541.PDF

THE KEYS TO KEY INFORMANT INTERVIEW (KII) Key to Success? Diversity is important. If saturation is reached, it’s time to stop. Interviewer skills Location: private, no distractions and non-threatening Tips for Writing Questions: Open-ended, neutral, one at a time, clear, use probing questions (can you give me examples, what changes have you noticed) Source: Performance M&E, USAID Center for Dev’t Info. And Eval; http://pdf.dec.org/pdf _docs/PNABS541.PDF

THE KEYS TO KEY INFORMANT INTERVIEW (KII) Analyzing the Data: - Create interview summary sheet Descriptive codes: themes, concepts, questions and/or ideas KII is perfect when: Descriptive information is needed Subject matter is complex. Respondents are of high status. Subject matter is highly sensitive. Source: Performance M&E, USAID Center for Dev’t Info. And Eval; http://pdf.dec.org/pdf _docs/PNABS541.PDF

IN-DEPTH INTERVIEW (IDI) What is IDI? Questions are based upon the responses collected Objectives of IDI: Discover the interviewee’s own framework of meanings Obtain rich, contextualized, in-depth information Source: Crystal Graphics, Inc. (2014) Retrieved on Dec. 7, 2014 from http://www.powershow.com/view3/50df42-MTE1M/Observations_and_in-depth_interviews_powerpoint_ppt_presentation

IN-DEPTH INTERVIEW (IDI) IDI Guide Questionnaire List of core questions: Topics Variations: Order, terminology (adapt to the terminology of the respondent) Additional questions may be introduced Types of Questions for IDI: - Behavior, opinion/belief, feelings, knowledge, sensory, background Source: Crystal Graphics, Inc. (2014) Retrieved on Dec. 7, 2014 from http://www.powershow.com/view3/50df42-MTE1M/Observations_and_in-depth_interviews_powerpoint_ppt_presentation

IN-DEPTH INTERVIEW (IDI) Qualities of Questions in IDI Open-ended, neutral, sensitive, clear to the interviewee Practical Aspects of IDI: - Number of interviewees, status of informant, recording process, dynamics of interview, leading role of informant, questions, sequential interviews, probing, relationship Source: Crystal Graphics, Inc. (2014) Retrieved on Dec. 7, 2014 from http://www.powershow.com/view3/50df42-MTE1M/Observations_and_in-depth_interviews_powerpoint_ppt_presentation

FOCUS GROUP DISCUSSION What is FGD? Involves a whole group answering questions together (Bailey, 1994: 192). ‘Guided group discussions’ designed to provide information on a certain topic from a certain population (Ward et al., 1991: 267). Are groups of unrelated individuals that are formed by a researcher and then led in group discussion of a topic (Schutt, 1999: 308).

FOCUS GROUP DISCUSSION When to use FGD? To familiarize a research topic To polish research objectives To deepen understanding of survey data To validate survey data When interested in capturing diverse and ‘detailed’ views and/or opinions

FOCUS GROUP DISCUSSION Characteristics of FGD Most focus groups involve 7 to 10 people May be semi-structured or unstructured Discussion is guided/facilitated by an FGD facilitator/leader FGD proceedings are recorded/documented by an FGD documenter

FOCUS GROUP DISCUSSION Pre-FGD Activities Defining the objectives Construction of the FGD guide especially for semi-structured FGD Familiarizing the FGD guide/questions Preparation of recording devices/materials Identifying and inviting FGD participants Confirming the attendance of FGD participants

FOCUS GROUP DISCUSSION Stages in the conduct of FGD Orientation of the FGD participants—setting discussion rules Getting to know (especially if participants are unrelated) Discussion proper Synthesizing and closing the discussion Distribution of tokens/snacks

FOCUS GROUP DISCUSSION Post-FGD Activities Checking the documentation/recorded data Transcribing the recorded discussion Encoding the finalized transcriptions Data Analysis

MOST SIGNIFICANCE CHANGE (MSC) Most Significant Change (MSC) involves the collection and systematic participatory interpretation of stories of significant change. Unlike conventional approaches to monitoring, MSC does not employ quantitative indicators, but is a qualitative approach.

MOST SIGNIFICANCE CHANGE (MSC) It is a form of participatory monitoring and evaluation. It is participatory because many project stakeholders are involved both in deciding the sorts of change to be recorded and in analyzing the data.

HISTORY OF MSC The MSC approach was originally developed by Rick Davies through his work with a participatory rural development project in Bangladesh in 1994. It has since been adapted and widely promoted in many developing countries.

HISTORY OF MSC It is a form of monitoring because it occurs throughout the program cycle and provides information to help people manage the program. It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole.

MSC AS M&E TOOL It is a form of monitoring because it occurs throughout the program cycle and provides information to help people manage the program. It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole.

MSC PROCESS Essentially, the process involves: the collection of significant change (SC) stories emanating from the field level, and the systematic selection of the most significant of these stories by panels of designated stakeholders or staff.

MSC QUESTION SC stories are collected from those most directly involved, such as participants and field staff. The stories are collected by asking a simple question such as: ‘During the last quarter, in your opinion, what was the most significant change that took place for participants in the program?’

MSC PROCESS The designated staff and stakeholders are initially involved by ‘searching’ for project benefits. Once changes have been captured, various people sit down together, read the stories aloud and have regular and often in-depth discussions about the value of these reported changes. When the technique is implemented successfully, whole teams of people begin to focus their attention on program impact.

WHY STORIES? Storytelling is the vivid description of ideals, beliefs, personal experiences, and life-lessons through stories or narratives that evoke powerful emotions and insights. It enables expression of emotional aspects, factual content, and tacit knowledge. It increases the potential of meaningful knowledge-sharing. It augments the likelihood that learning will take place and be passed on.

Stories are the creative conversion of life itself into a more powerful, clearer, more meaningful experience. They are the currency of human contact. — Robert McKee Everyone is necessarily the hero of his own life story. JOHN BARTH

If you don’t know the trees you may be lost in the forest, but if you don’t know the stories you may be lost in life. — Siberian Elder There is no agony like bearing an untold story inside of you. - Maya Angelou

STEPS IN IMPLEMENTING MSC TECHNIQUE 1. Raise interest in MSC 2. Defining the domains of change 3. Determine the process and responsibilities 4. Collect significant change stories 5. Select the most significant of the stories 6. Verify stories 7. Analyze the top stories for program learning

Sample of a Promising Story I had my first PRIME experience when I attended the immersion at the IP Center on February 10, 2012. In my encounter with the IP students, teachers, folks and the community, I had developed in myself pride of one’s culture, love and care of one’s identity with rich tradition and diverse cultural heritage. It was an eye-opener to me that no one could certainly preserve one’s culture other than ourselves.

Sample MSC Questionnaire Sample 1: MSC Form for Teachers Sample 2: MSC Form on Drop-outs for School Head Sample 3: MSC Template on New Systems, Technology

Initial Gain Evaluation MSC SCATTERGRAM Dimensions Stages of M & E Institutionalization Policy Formulation * * Managing & Monitoring * * * * * * * * * Initial Gain: Improving Teacher’s Competence * * * * * * * * Technical Assistance Innovation * * * * * * * Implementation of the Program * * * * * * * * * * * * New Process/ System/ Technology Experience or Encountered * * * * * * * * * * Awareness Progress M & E Initial Gain Evaluation Outcome Evaluation Impact Evaluation Note: * This Indicator shows that such program component is embedded explicitly / implicitly in the MSC story of the respondents.

APPLICATION: MSC Answer the following questions based on your experience: Tell us about your involvement in program/projects addressing dropout in your school/district/division. What is the most significant change/experience of the school/community as a result of the dropout reduction program/project? Can you cite observable changes among the children who have availed of the dropout reduction programs/schemes? Relate a personal account of a particular student/s. How were the parents & community leaders involved in bringing about these changes? Why is this story significant to you? What lessons can we learned from this experience and what could be improved?