(5) Moderators - Coding

Overview General Information to keep in mind:  Coded variables - Objective – which are usually study characteristics that can be directly coded from text by 1-2 people  Judged variables - Subjective – which are inferential judgments of new variables not in text, coded by 2-3 people  If more than one coder, need consistency and reliability between coders

What to code? Depends upon purpose of meta:  Coded variables Descriptive information like publication form, date, experimenters, etc. Demographic information like age, gender, education, ethnicity, etc. Design – type of design such as survey, experiment, lab, field, etc. Ss – type, characteristics, etc. IV – type, levels, description, etc. DV – type, level, description, etc. Hypothesis/Theory – type, generalization, etc. Manipulations – type, level, description, etc. Measures – type, level, description, etc. Existing moderators/mediators – type, level, quantity, etc. See Table 1 of my Quals for coded variables

What to code? Depends upon purpose of meta:  Judged variables Identify existing factors from primary studies, and then code all the studies by those factors e.g., Moderators, mediators, variables, theories, processes e.g., read method sections for differences between studies Identify new factors not from primary studies and then code all the studies by those factors e.g., Moderators, mediators, variables, theories, processes e.g., try looking at other metas for ideas

How to code: Part 1 For many of the variables you can start coding them yourself within your excel file (Part 1) and then later we will do formal content coding (Part 2) with multiple coders using a coding manual Part 1  When reading the primary studies, start making a list of possible variables to be included in the meta  After we have decided on the variables to be included, start coding by creating new columns in the excel file  The first row is the coding scheme, and then the rows for each study are the ratings/numbers for the variables See the “Example-DataSet1” for columns, G, H, I

How to code: Part 2 (1) Create Coding Manual (definitions, examples, scale ranges) (2) Photocopy “Method” sections (3) Create Coding Sheet (for coders to enter codes, in excel) (4) Training Phase - using samples - Meet and discuss coding manual - Code the samples separately - Samples are ideally methods not used in meta - Conduct reliability analysis (alpha, or frequency) - If discrepancies, repeat and refine (5) Coding Phase - using full data - Read/skim the articles to get familiar - Code question-by-question across all studies - Randomize order of studies for each question

Other issues: Consistency of a single coder from occasion to occasion  Did the coder code the same way from the first entry to the last entry?  Is there a break in coding of days, week, months?  Might become more proficient and knowledgeable at end so code different than first entry  Try re-coding the same study at different time to see if you are reliably coding

Keep a record of everything Within text of paper, or in Tables, you will report:  Why was that variable included in coding manual?  Has that variable been used before, and if so how/why?  If variable is new, where did it come from?  If variable is new, what is reasoning behind inclusion?  What is prediction for each variable?  Is there a counter prediction?  What is underlying theory for the variable?  What was text of question from coding manual  What is the scale range of the question  What is an example of low on scale range  What is example of high on scale range