1. Task 2: Design a new task-based evaluation challenge Jack Mostow, Lee Becker, Julius Goth, Claire McConnell, Itziar Aldabe

2. Task 2 Guessing whether a question is human or computer generated – Single player – Multiplayer For single player, compete against clock – Point value depreciates over time – Reward structure Correct, receive point value Incorrect, lose point value

3. At some very basic level, there are three ways in which businesses can respond to the green imperative. What businesses do to the green imperative? Human Computer Correct! This question came from a computer.

4. At some very basic level, there are three ways in which businesses can respond to the green imperative. In how many basic ways can businesses respond to the green imperative? Human Computer Incorrect! This question came from a human.

5. Reward structure Quickness of Response FastSlow Polarity of Response Accept12 Reject43

6. Gold Standard Questions Turk-it to collect Turk-it to filter for usability