The basics of Social Science Research Lecture 4 Dr. Jordy Gosselt Department of Communication Science

Today Pilot test Data collection Data analysis Self-study  Book Chapter 9, 10, 14 & 15 [Bha2012] Bhattacherjee, A. (2012). Social Science Research. Principles, Methods, and Practices. Tampa, Florida: University of South Florida. PDF http://scholarcommons.usf.edu/cgi/viewcontent.cgi?article=1002&context=oa_textbooks

Your research process Research method: Validity/reliability & Pilot Testing Research execution: Data collection Data Analysis Make your data file Check reliability of your variables Calculate frequencies, Means, SD Compare groups

Research method: Reliability/Validity & Pilot Testing Reliability: extent to which a measurement tool gives consistent results Validity: extent to which a measurement tool measures what it is supposed to measure Example: “To what extent do media ratings instigate a forbidden fruit effect” Media ratings (=stimulus material): What games to select + What media ratings? => pretest Forbidden fruit effect: How to measure this? Existing scales / New scale / Adjusted scale? Reliability: e.g. you weight yourself several times during a day and the scale does not vary much = high reliability you weight yourself several times during a day and the scale drastically varies = low reliability Validity: e.g. Intelligence test that actually measures intelligence = high validity Intelligence test that measures memory instead = low validity

Research method: Reliability/Validity & Pilot Testing Report about validity and reliability; what did you do to establish sufficient validity and reliability for your research: Have you used existing measurement scales? Have you performed a pre-study (pilot test) to test stimulus materials? Is the wording of the questions correct? Do you measure what you intent to measure? (validity) Gives your measurement consistent results? (internal consistency reliability)  Cronbach’s Alpha Calculation How did you select participants? Take into account possible threats regarding validity and reliability and address those threats in your methods section. Reliability: e.g. you weight yourself several times during a day and the scale does not vary much = high reliability you weight yourself several times during a day and the scale drastically varies = low reliability Validity: e.g. Intelligence test that actually measures intelligence = high validity Intelligence test that measures memory instead = low validity

Your research process Research method: Validity/reliability & Pilot Testing Research execution: Data collection Data Analysis Make your data file Check reliability of your variables Calculate frequencies, Means, SD Compare groups

Data Collection Type of data collection: qualitative vs. quantitative Expected amount of collected data Online questionnaire: Between 200 and 250 respondents Hard-copy questionnaire: Between 100 and 150 respondents Experimental designs: Depending on number of conditions (min 30 resp. per condition) Interviews: Between 15 and 20 participants Focus groups: Between 4 and 6 group sessions Content analysis: Depends on the research design Observations: Depends on the research design

Data Collection: Qualtrics Data collection procedure Online survey tool “Qualtrics” is available in cases of quantitative data collection (i.e. surveys, experiments, sorting tasks, heat maps...) Easy to use and distribute Direct export to SPSS

Data Collection: Qualtrics

Data Collection: Qualtrics

Data Collection: Qualtrics

Data Collection: Qualtrics Many features and options: see video’s on YouTube for instruction and on Qualtrics’ website Information to register for an account can be found at: http://www.utwente.nl/cw/qualtrics/

Your research process Research method: Validity/reliability & Pilot Testing Research execution: Data collection Data Analysis Make your data file Check reliability of your variables Calculate frequencies, Means, SD Compare groups

Make your data file

Data analysis: Make your data file Export your data from Qualtrics to SPSS:

Data analysis: Make your data file Data and variable view in SPSS 1=male 2=female Age in numbers 1=School A 2=School B

Data analysis: Make your data file Make labels (=questions) Fill in the values (=answer options) Data analysis: Make your data file Data and variable view in SPSS 1=male; 2=female Age in numbers (no value) 1=School A; 2=School B

Data analysis: Make your data file Data and variable view in SPSS

Data analysis: Make your data file Clean your data: Make your data anonymous (delete names, ID‘s) Complete? Time duration realistic? Only include the variables you need for your analysis in your final data file

Data analysis: Make your data file Re-coding of variables Translate open answers into quantitative codes (manually)

Data analysis: Make your data file Re-coding of variables Re-coding of “smartphone ownership“(manually) So, better option: Answering categories from the start! 1= 0-1 yrs 2= 1-2 yrs 3= 2-3 yrs etc,.

Data analysis: Make your data file Re-coding of variables Re-coding of “age in years“ into “age categories“ (automatically)

Data analysis: Make your data file Re-coding of variables Re-coding of “age in years“ into “age categories“ (automatically)

Data analysis: Make your data file Re-coding of variables Re-coding of “age in years“ into “age categories“ (automatically)

Data analysis: Make your data file Re-coding of variables Re-coding negative item (automatically) Emotional Appeal I have a good feeling about the company. I admire and respect the company. I do not trust this company. Products and Services Stands behind its products and services. Develops innovative products and services. Offers low quality products and services. Offers products and services that are a good value for money. Vision and Leadership Has excellent leadership. Has a clear vision for its future. Recognizes and takes advantage of market opportunities. Workplace and Environment Is well managed. Looks like a bad company to work for. Looks like a company that would have good employees. Social and Environmental Responsibility Supports good causes. Is an environmentally responsible company. Maintains high standards in the way it treats people. Financial Performance Has a strong record on profitability. Looks like a low risk investment. Tends to outperform its competitors. Looks like a company with strong prospects for future growth.

Data analysis: Make your data file SPSS discriminates between: Scale Nominal Ordinal measurement

Data analysis: Make your data file Scale: ordered categories, has a „true zero“ point e.g. age (if open question)

Data analysis: Make your data file Nominal: e.g. region, Zip-code, operating system, gender, yes/no (numbers that are simply used as identifiers)

Data analysis: Make your data file Ordinal: ranking (e.g. Likert Scale)

Data analysis: Missing value

Data analysis: Check reliability of variables Gives your measurement consistent results? (internal consistency reliability)  Calculate Cronbachs Alpha

Data analysis: Check reliability of variables

Data analysis: Check reliability of variables

Data analysis: Check reliability of variables Cronbach‘s Alpha for the construct Perceived Security (PS) is .72 In the original scale .82 Look at „if item deleted“ Cronbach's alpha Internal consistency α ≥ 0.9 Excellent 0.9 > α ≥ 0.8 Good 0.8 > α ≥ 0.7 Acceptable 0.7 > α ≥ 0.6 Questionable 0.6 > α ≥ 0.5 Poor 0.5 > α Unacceptable

Data analysis: Check reliability of variables Might consider to delete 4th item

Check Cronbach‘s alpha for all variables. Then: Make variables in spss

Data analysis: Making your variables

Data analysis: Making your variables

Data analysis: Making your variables

Dataset is finished, time for analysis: Calculate frequencies, Means, SD

Data analysis: Calculate frequencies

Data analysis: Calculate frequencies Gender: 82.9 % male and 17.1 % female Study program: 82.9 % Computer Science 8.6 % Electrical Engineering etc. … Describe all the characteristics of your sample that you consider as important to mention! Be as specific as possible but do not report nonsense…

Data analysis: Calculate Means + SD

Data analysis: Calculate Means + SD Age: 24.17 (SD 6.14 yrs) Average time s.o. has owned a smartphone (in months): 79.97 (SD 36.15) Average time s.o. uses a smartphone (in years): 6.66 (SD 3.01) etc. . Standard Deviation = amount of variation within a given data set. A low standard deviation indicates that the data points tend to be close to the mean (also called the expected value) of the set, while a high standard deviation indicates that the data points are spread out over a wider range of values.

Data analysis: Calculate Means + SD

Data analysis: Calculate Means + SD

Sub groups in your sample. Or: different conditions Sub groups in your sample? Or: different conditions? Compare the answers of groups of respondents

Data analysis: Compare Means Example: To what extent do females score differently on perceived security compared to males?

Data analysis: Compare Means Or: condition (high tech campaign vs green campaign)

Data analysis: Compare Means The difference between gender (female vs male) and the mean score on Perceived Security is not statistically significant (p = .642) (so there is no difference!) We usually consider results as statistically significant at a threshold of .05 (or below) (=95% certainty)

Data analysis: Compare Means Fundamentalists = are most protective and concerned about their privacy Pragmatists = weigh the potential pros and cons of sharing personal data, but are less concerned than fundamentalists Perceived Security of the app store the individual usually uses, measured on a Likert Scale going from 1 = completely disagree to 7 = completely agree Conclusion: Fundamentalists perceive the security of the app store they usually use as significantly (p<.05) less secure (M=3.71) than pragmatists (M=4.83).

Data analysis And much more tests… Depends on your RQ‘s Read Andy Fields‘ Book Have a look at https://statistics.laerd.com/ Have a look at https://www.spss-tutorials.com/ Go to methodology shop (methodologie winkel BMS) https://www.utwente.nl/en/bms/m-store/ Ask the supervisor team if you need help [Fie2009] Field, A. (2009). Discovering statistics using SPSS. LA: Sage.

Now: Self-study: developing/finalizing your questionnaire developing an analysis strategy working on your study design or experimental setting/manipulation etc… What you eventually will be working on depends on the status of your project.