Nebraska 4-H Robotics Effectiveness of Educational Robotics in the Classroom

Presenters Dr. Bradley S. Barker, 4-H Science and Technology Specialist – Dr. Neal Grandgenett, Professor of Mathematics Education – Dr. Gwen Nugent, Associate Research Professor, Nebraska Center for Research on Children, Youth, Families and Schools –

Presentation Overview Provide background of using robots in non- formal education The three iterations (pilot study, large scale study, NSF ITEST program) and impact data from the 4-H robotics program. Embedded assessment (Reasoning/ Communication) Effects of robotics on attitudes Summary Questions

Purpose of Program To address the shortage of students pursuing careers in science, technology engineering, and mathematics (STEM). Goals –promote youths’ interest in STEM fields (including IT), –introduce basic STEM skills, –foster problem solving and inquiry, –and encourage teamwork

Background – Why Robotics? Integrate many content areas (CS, Engineering, Mathematics) LEGOs are familiar to youth Durable and reusable Relative low-cost Programming language works on Macs and PCs Widely available Motivating

NXT Robot with GPS trailer

RCX Robot

Review of Literature Fundamental gap in literature – what is the impact of using robotics on STEM learning? Most studies are qualitative, examine self-evaluations, project descriptions, and student reactions. We set out to examine the impact on STEM learning in a nonformal environment.

Pilot Study: Gibbon Elementary Purpose was to develop assessment instrument and examine impact on learning STEM concepts. Developed 24-item multiple choice test one item from each unit in the curriculum. –Test items reviewed by CMU robotic experts and revised. Robotics group met twice a week for six weeks. Pre test prior to intervention and post on last day for both groups.

Pilot StudyParticipants Participants –32 students in study ages 9-11 median age was 9.0 –14 students in (9 male, 5 female) experimental group –18 students in control group (11 male, 7 female) –Control group selected by instructor in same grade but not part of the afterschool program.

Pilot Study Results Cronbach’s alpha score of 0.86 on posttest. LEGO Robotics questions removed STEM assessment items alpha was calculated at 0.76 The assessment instrument seemed to be valid and reliable. Used Pell and Jarvis instrument to measure attitudes towards science.

Pilot Study Results Observed impact on learning –No significant difference (t(30) = 11.60, p =.70 on pretest scores between groups (M=7.50, SD = 2.58, control) and M = 7.93, SD=3.71, experimental). –Significant difference (t(22,17) = 12.93, P <.000 between groups on posttest (M=7.44, SD = 2.98, control) and M = 17.00, SD=.88, experimental). No change in attitudes

Pilot Study

Pilot Results Refined the assessment instrument. Felt confident students had an increase in STEM content areas as well as specific robotic concepts. Used instrument for larger study.

Large Scale Study (RCX) Participants –121 students ages 7-14 from 6 afterschool programs and 3 4-H clubs. –36 youth ages acted as a control group from 3 afterschool programs Interventions (not concurrent) lasted for 8 weeks. The pretest was administered prior to the intervention and the post test was administered immediately after.

Large Scale Study Results Impact on learning –ANCOVA analysis used posttest as dependent and pretest, gender as covariates –Main effect was significant F(1,141) = p=.001 –Posttest (M=10.68, SD = 3.93, control) and (M = 11.09, SD= 3.93, experimental). –No difference based on gender F(1, 141) =.833 p=.478

Boxplots Pretest to Posttest

NSF ITEST Program Expanded 4-H robotics program –Includes the integration of robotics with Geospatial technologies (GIS, GPS, aerial photography) –Looking at applications in precision agriculture and natural resources –Provide career exploration with visits from scientist and engineers. –New robotic kits

ITEST Program Components Open to middle school students (200) Year 1 - Pilot camps Year 2 –Start with 40-hour summer camp –1 day camp for educators and leaders –Youth then complete 80 hours in clubs and after- school programs Year 3 –40-hour summer camp –80 hours in clubs and after school programs

Results of Pilot Camps Site 1 – Gretna, NE 6 day overnight camp. –N = 12 ages, 11-14, median age –8 males, 4 females –Paid to attend the camp Site 2 - Grand Island, NE 5 day camp. –N = 26 ages 11-15, median age –18 males, 8 females –Offered through CLCs at Barr M.S. Modified content examine and added new questions, piloted an embedded assessment

Camp Results Academic Site 1 pretest m = 14.5, sd = 4.42 and posttest m = 17.5 sd = 4.89 Site 2 pretest m = 11.77, sd = 2.99 and posttest m = 16.04, sd 3.68 Overall significant increase in scores using ANCOVA analysis using posttest as dependent and pretest as covariate F(1,31) = 21.24, p =.000. KR20 score =.81, Alpha score =.799

Boxplots Pretest to Posttest

Boxplots by Content Area

Camp Results Attitude Developed new attitude instrument focusing on robotics –Modeled after the Motivated Strategies for Learning Questionnaire (Pintrick, Smith, Garcia, & McKeachie, 1991) –Multiple scales focusing on task value, motivation, self-efficacy, problem solving, cooperative learning Conducted interviews with selected youth in Grand Island

Camp results Attitude Goal - promote youths’ interest in STEM fields 4-point scale

Camp results Attitude Promote youth’s interest in STEM –Six of the seven Grand Island youth interviewed said the camp made them like math and science more –Four said it increased their interest in a career in STEM

Camp results Attitude Goal – foster problem solving and inquiry skills Significant increase for “make a plan”

Camp results Attitude Interviews –There was kind of a big task, so it made us break it down into little parts and then get it done step by step. –For the robot, you have to start with nothing and tell it exactly what to do. –If your robot did something wrong, you have to figure out after how many steps it was and then you fix the particular problem.

Camp results Attitude Goal – Encourage teamwork Significant increase for “work with others” and “listening to others”

Camp results Attitude Interviews –I like the hands on aspect of it. I hate just being stuck in a chair and writing stuff down. –It was fun when we did challenges.

Future Directions: Attitude Revise the instrument –Factor analyze the instrument and revise Use results to guide instructional improvement Initiate new study that measures attitude mid- way through the camp

Piloting Embedded Assessment Content Tests Attitude Test Pretest Measures Envelope Activity Posttest Measures Robotics Instruction Content Tests Attitude Test Think-Aloud Interview Task Rubrics Design Tasks Control Groups: Partners and Partner Schools

Embedded Asssessment: Envelope Activity (mathematical communication/reasoning, and also used to discuss programming instruction) Steps to Mailing a Letter: Student Directions Directions: In order to do something well, like playing a sport or a board game, you usually need to know the "procedures" of how to do it. Thus, it is often helpful for you to know the steps in doing some task, before you are able to explain that task or procedure to someone else. For example, if you are given a letter that you need to mail, with an envelope, a stamp, a ruler, and an address of where you want to send the letter, can you describe how you would mail that letter? Or in other words, can you describe the general procedure for mailing a letter? Let's further pretend that this person who will be mailing the letter is a friendly alien from Outer Space, and that they have absolutely no idea about how to mail a letter here in the United States. List the steps that the alien should use to mail a letter. Use the blank sheets of paper provided to make your list. You have 15 minutes to accomplish this task.

Sample Products

Scoring:

Envelope Activity Analysis:

Student Interviews: Are there any similarities between the envelope activity and programming? Are there any differences between the envelope activity and programming? “You have to do the steps very carefully” “Both big things need to be broken down into little steps” “I had to tell a thing that knows nothing how to do something” “It is easier to see the result when you tell a robot” “It is more fun to write steps on the computer than with pencils” “I am better at thinking than a robot…they just do what they are told”

Summary Robotics seems to have a promising potential impact on academic achievement Robotics impact on attitudes is difficult to measure; current results suggest that the impacts are limited to specific areas. More research is needed into long term effects

Summary Robotics seems to have a potential impact on learning of robotics concepts and principles based on pre to post test scores More research is needed into long term effects Intervention seems to increase interest in STEM but it is difficult to measure directly and mathematical interest is particularly challenging Embedded assessment is a promising area of investigation for robotics activities

Questions?

Back Page A final quote following questions….. “We have not succeeded in answering all of your problems. The answers we have found only serve to raise a whole set of new questions. In some ways, we feel we are as confused as ever, but we believe we are confused on a higher level and about more important things.” Omni Magazine, 1992