By Franklin Kramer.   Crowdsourcing web service  Have turkers complete HITs for small amounts of money (most being 1-25 cents)  Can filter workers.

Slides:



Advertisements
Similar presentations
1 How to Overcome Staff Objections Develop incentive program 1.$3-$7 for each ad you assumptivlly sell into the program for the first 30-days 2.Special.
Advertisements

1 Welcome to CASSEL Welcome to the CASSEL Lab, and thank you for participating in todays experiment. It is very important that you do not touch the computer.
A measurement of fairness game 1: A box contains 1red marble and 3 black marbles. Blindfolded, you select one marble. If you select the red marble, you.
Introduction to Mechanized Labor Marketplaces: Mechanical Turk Uichin Lee KAIST KSE.
Running Experiments on Mechanical Turk: Day 1 Tim Brady What is Turk? Who are Turkers? How much do I pay? Are Turk workers good subjects? Setting up a.
Bringing the crowdsourcing revolution to research in communication disorders Tara McAllister Byun, PhD, CCC-SLP Suzanne M. Adlof, PhD Michelle W. Moore,
Rethinking Grammatical Error Detection and Evaluation with the Amazon Mechanical Turk Joel Tetreault[Educational Testing Service] Elena Filatova[Fordham.
Chapter 4 Conducting & Reading Research Baumgartner et al Chapter 4 Developing the Research Plan.
CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility Alex Quinn, Ben Bederson, Tom Yeh, Jimmy Lin.
Fair Games/Expected Value
EXAMPLE 1 Finding a Sale Price You are shopping for a guitar and find one with an original price of $160. The store is offering a 30% discount on all guitars.
How Math Can Help You Choose the Right College Dr. Geoff Turner.
Christopher Harris Informatics Program The University of Iowa Workshop on Crowdsourcing for Search and Data Mining (CSDM 2011) Hong Kong, Feb. 9, 2011.
Amari,Jada,Taurus. Petrol Direct Accuracy The website is supported by a series of authors and other websites who have studied the pricings of gas. The.
Banking for Safety Discussion Why was it difficult for many workers in New Orleans to get their paychecks after Hurricane Katrina hit the city? How did.
Book Fair by: Stephanie Milton. Come Get Lost in a World of Books When: May 23 rd 2011 Where: St. Patricks gymnasium Time: 10:00-2:00 Cost: $
Points in Distributions n Up to now describing distributions n Comparing scores from different distributions l Need to make equivalent comparisons l z.
Crowdsourcing: Ethics, Collaboration, Creativity KSE 801 Uichin Lee.
Evaluating Economic Performance. Section 2 Vocab Social Security Inflation Fixed Income.
Crowdsourcing using Mechanical Turk Quality Management and Scalability Panos Ipeirotis – New York University.
Session 4 Social Entrepreneurship Interterm Course.
Let’s Learn About Money!
Name: Date: Counting Up to Make Change 1 DO NOT subtract to find the answer. Use change to find the answer. Example: 1)Count up, using coins, from the.
Chapter 5: Probability Section 5.1: Probability Rules.
Improving Search Results Quality by Customizing Summary Lengths Michael Kaisser ★, Marti Hearst  and John B. Lowe ★ University of Edinburgh,  UC Berkeley,
Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson.
Ch 7. What Makes a Great Analysis? Taming The Big Data Tidal Wave 31 May 2012 SNU IDB Lab. Sengyu Rim.
Second International Seville Seminar on Future-Oriented Technology Analysis (FTA): Impacts on policy and decision making 28th- 29th September 2006 Use.
For slides 2 to 12, count the change as the coins and bills appear on the screen. Before clicking the final answer, count the change to yourself to see.
Understanding Randomness.  Many phenomena in the world are random: ◦ Nobody can guess the outcome before it happens. ◦ When we want things to be fair,
L56 – Discrete Random Variables, Distributions & Expected Values
Chapter 8 Beginning a New JobSucceeding in the World of Work What You Can Expect From Your Employer 8.2 SECTION OPENER / CLOSER INSERT BOOK COVER ART Section.
Ethical Use of Amazon Mechanical Turk Kristy Milland Ryerson University
1 Welcome to CASSEL Welcome to the CASSEL Lab, and thank you for participating in today’s experiment. It is very important that you do not touch the computer.
Bernoulli Trials, Geometric and Binomial Probability models.
© 2009 Amazon.com, Inc. or its Affiliates. Insights into Mechanical Turk (or, “Mistakes Requesters Make”) Adam D. Bradley
Franklin Kramer.   Background  Experiment  Results  Conclusions Overview.
Making Predictions with Theoretical Probability. Warm Up You flip a coin three times. 1.Create a tree diagram to find the sample space. 2.How many outcomes.
Building Loyalty to Online Communities through Bond and Identity-based Attachment to Sub-Groups Yla R. Tausczik Laura A. Dabbish Robert E. Kraut Carnegie.
16-3 The Binomial Probability Theorem. Let’s roll a die 3 times Look at the probability of getting a 6 or NOT getting a 6. Let’s make a tree diagram.
Jane wins $21 if a die roll shows a six, and she loses $2 otherwise
Probability Statistics Introduction This stuff can be a bit hard, but don’t be afraid We use probability for our purposes, so it will be a tool,
Welcome to SSEL Welcome to the SSEL Lab, and thank you for participating in today’s experiment. It is very important that you do not touch the computer.
When could two experimental probabilities be equal? Question of the day.
Open data, implications for teaching and learning Dun Laoghaire Institute of Art Design & Technology Robert Griffin EdTech 2013.
Fisher, M., Voracek, M., Rekkas, P., & Cox, A. (2008). Sex differences in feelings of guilt arising from infidelity. Evolutionary Psychology, 6, 3,
Algebra Applications of Linear Systems. Amusement Park Two families go to Busch Gardens. In Family A, they buy 3 children (ages 3 to 9) tickets.
IMPACT EVALUATION PBAF 526 Class 5, October 31, 2011.
Uncertainty and Games (Ch. 15). Uncertainty If a game outcome is certain can it achieve meaningful play? –Example of such a game? Two kinds of uncertainty:
28. June Crowdsourcing – Challenges and Opportunities in Web Science Imaad Mohamed Khan
Conducting Behavioral Research on Amazon‘s Mechanical Turk
Claire M. Renzetti, Ph.D. Judi Conway Patton Endowed Chair, CRVAW
High Deductible Health Plans and Health Savings Accounts
Standardized Coefficients (Beta)
Culture and Group decision Making exercise
Statewide Community Infrastructure Program (SCIP)
The need for standardization
Web Design And Development Company
Conducting Behavioral Research on Amazon’s Mechanical Turk
Best Web Design Company in Chennai at India
Counting Money Directions: 1. Do not view slide show.
Chapter 17 Thinking about Chance.
Probability Trees By Anthony Stones.
Adverse Event Post mentions
Performance Management
Probability of TWO EVENTS
Looking at care opinion from both sides
Cumulative Changes in Components of U. S
Per cent optimal adherence over time among the study population by 5-day, 30-day and adherence behaviour scales, by exposure status to community health.
Open and Online Experimental Philosophy (X-Phi)
Presentation transcript:

By Franklin Kramer

  Crowdsourcing web service  Have turkers complete HITs for small amounts of money (most being 1-25 cents)  Can filter workers by specific criteria  Typical worker earns $1.40 (Horton & Chilton)  US Turkers are fairly representative of general population- much better than University participants (Ipeirotis) What is Mechanical Turk?

  Original experiment (Robert Mille)  31/19  Coin Flips with Bonuses (same or different)  32/18, 23/27  Coin Flips: A further look (time)  Die rolls (Suri et. al) Background

  If people are going to lie, they will tend to choose the first result offered.  People are more likely to give honest answers when dishonesty is more transparent in potential results given.  Price will have little effect on results.  People are less likely to lie when they are more aware of what is at stake.  People will be less likely to lie when they feel representative of a group they belong to. Hypotheses to Test:

  Lying is a very interesting social phenomena  Benefit to lying minimal  Only detectable on large scale  MT is fairly representative  Valuable to understand how to best give valid results in MT Why?