Experiment „Natural Language Processing“ Christian Gütl, IICM, TU-GRAZ Instructions for Experiment „Natural Language Processing“

Experiment „Natural Language Processing“ Christian Gütl, IICM, TU-GRAZ2 The Experiment Procedure  Topic: “Natural Language Processing (NLP)”  Objective of the Experiment  Gain insight about extracting and evaluating concepts and generating questions  Informed consent  Please fill in the informed consent

Experiment „Natural Language Processing“ Christian Gütl, IICM, TU-GRAZ3 The Experiment Procedure  Step 1: Learning the text  Step 2: Questionnaire 1  Step 3: Learning Activity I (Questionnaire 2)  Step 4: Test (Questionnaire 3)  Step 5: Learning Activity II (Questionnaire 4)  Step 6: Questionnaire 5a  Please allow yourself breaks between tasks!

Experiment „Natural Language Processing“ Christian Gütl, IICM, TU-GRAZ4 Please learn the text (max. 30 minutes)  See (Subject: „NLP – text“) Step 1

Experiment „Natural Language Processing“ Christian Gütl, IICM, TU-GRAZ5 Please fill in Questionnaire 1 (max. 10 min)  See (Subject „phd Questionnaire 1“)  Please give all answers in English!  Please do not use the text while answering the questions!  Click on the „Submit“-Button once you have finished the task! Step 2

Experiment „Natural Language Processing“ Step 3a: Extracting concepts  Please extract the main concepts of the NLP-text, using Questionnaire 2 (max. 10 min)  (Example: Speed of light; theory of relativity, Einstein).  Please separate your concepts with semicolons (;) or commas (,)  See (Subject „ phd Questionnaire 2“)  After you have finished this step, please continue working on Questionnaire 2 (Step 3b) as described on the next slide Christian Gütl, IICM, TU-GRAZ6

Experiment „Natural Language Processing“ Step 3b: Generating questions  Please generate eight questions from the NLP-text, using Questionnaire 2.  Construct two Open Ended questions using the text (Example: What do you know about theory of light?)  Construct two Fill In The Blank questions using the text (Example: The founder of this university was ________.)  Construct two Single Choice questions using the text. Their answer should be “no”! (Example: Albert Einstein was born in Norway)  Construct two Multiple Choice questions with three distractors for each chapter. Only one of the distractors should be correct! (Example: The sun is _____. blue, green, yellow, red) Note that you have to generate one question for each category first and then another one afterwards Christian Gütl, IICM, TU-GRAZ7

Experiment „Natural Language Processing“ Christian Gütl, IICM, TU-GRAZ8 Step 4 Please fill in Questionnaire 3 (max 15 Minutes) and allow yourself a break afterwards.  See (Subject „ phd Questionnaire 3“).  Please do not use the text while answering the questions!

Experiment „Natural Language Processing“ Christian Gütl, IICM, TU-GRAZ9 Step 5a: Evaluation of concepts Evaluation Task I (max. 10 min.)  Please evaluate the concepts you will find in Questionnaire 4. Do you think the concept is relevant for the text?  1 = not relevant; 5 = very relevant  See (Subject „ phd Questionnaire 4“). After you have finished this task, please continue working on Questionnaire 4 (Step 5b) as described on the next slide

Experiment „Natural Language Processing“ Christian Gütl, IICM, TU-GRAZ10 Step 5b: Evaluation of questions Evaluation Task II (max. 35 min) Please evaluate the questions provided in Questionnaire 4, using the criteria as described below. For each criterion described below, please assign a score between 1 ( = very bad) and 5 (very good) to each question. Pertinence: Relevancy of the formulated questions. Has the question a precise and logical connection to the topic? Terminology: Appropriateness of the words chosen to formulate the question. Level: Extent in approaching the topic. Is the question trivial or expresses it a significant meaning? (1 = trivial, easy; 5 = difficult, significant meaning) Answer (only if provided!): Is the provided answer relevant? Distractors (only for multiple choice!): Quality of the computed distractors of the multiple choice questions

Experiment „Natural Language Processing“ Step 5c General Survey Please fill in the last section of Questionnaire 4 (max. 15 min) Christian Gütl, IICM, TU-GRAZ11

Experiment „Natural Language Processing“ Step 6 Evaluating questions Christian Gütl, IICM, TU-GRAZ12 Evaluation Task III Please evaluate the questions provided in Questionnaire 5a (see ), using again the criteria as described below. For each criterion, please assign a score between 1 ( = very bad) and 5 (very good) to each question. Pertinence: Relevancy of the formulated questions. Has the question a precise and logical connection to the topic? Terminology: Appropriateness of the words chosen to formulate the question. Level: Extent in approaching the topic. Is the question trivial or expresses it a significant meaning? (1 = trivial, easy; 5 = difficult, significant meaning) Answer (only if provided!): Is the provided answer relevant? Distractors (only for multiple choice!): Quality of the computed distractors of the multiple choice questions

Experiment „Natural Language Processing“ Thank you for participating in this Experiment! Christian Gütl, IICM, TU-GRAZ13