NRTs and CRTs Group members: Camila, Ariel, Annie, William

NRTs—Norm-referenced Item Analysis  Item facility  Item discrimination

Why do we do item analysis?  Assemble a large number of items of the type you want on the test  make sure the items are well written and clear  Pilot the items  Analyze the results of the pilot testing  Select the most effective items/get rid of the ineffective items or revise the weak ones

The basic purpose of NRTs  To spread students out along a general continuum of language abilities making aptitude, proficiency, or placement decisions

Two item statistics used in IA of NRTS  Item facility(IF): the proportion of students who answered a particular item correctly. 45/50= % of the students answered the item correctly — the item is very easy  Item discrimination(ID):Caculate IF for the upper group and the lower group using AVERAGE (C2:C6) and AVERAGE (C15:C19) ID=IFu-IFl ex: IFu-IFl=.20

Item Discrimination (ID)  can be calculated by first figuring out who the upper and lower students are on the test  using their total scores to sort them from the highest score to the lowest  Equal numbers of students in three groups — high/middle/low  ID=If upper - If lower

Ideal items in an NRT  Should have an average IF of.50 50% of the students answered correctly and 50% of them answered incorrectly  In reality, items rarely have an IF of exactly.50  Those that fall in a range between.30 and.70 are usually considered acceptable for NRT purpose

Items within.30 to.70 range  The items among them that have the highest IDs should be further selected for inclusion in the revised test.  This process would help the test designer to keep only those items that are well centered and discriminate well between the high and the low scoring students

Conclusion for NRT  IF and ID are only appropriate for developing and analyzing norm- referenced tests  Used at the institutional level overall English language proficiency tests or placement tests  Not appropriate for developing and analyzing classroom oriented criterion- referenced tests like the diagnostic, progress, and achievement tests

CRTs-Criterion-referenced Item Analysis  Purpose of CRTs — to measure the amount (or percent) of material in a course or program of study that students know  Usually for purposes of making diagnostic, progress, or achievement

Two item statistics used in IA of CRTs 1.The difference index(DI): the item facility on the particular item for the posttest minus the item facility for that same item on the pretest pretest:10/50=.20; posttest:45/50=.90 DI= =.70  DI tells how much the students are improving between the pretest and posttest on each item  The higher the value of the DI, the better.  A value of 1.00 is a perfect difference index

Two item statistics used in IA of CRTs 2.The B-index: the item facility on the particular item for the students who passed the test minus the item facility for the students who failed  The B-index show how well each item is contributing to the pass/fail decisions that are often made with CRTs B-index=IF pass - IF fail =14/14-0/6= =1

P.21  The B-index tell us how well each item is contributing to the pass/fail decision on this test at the cut-point.  Like ID and DI statistics, the higher the B-index, the better.  A perfect B-index would be 1.00.

Conclusion-CRTs  When should these indices be used? To analyze the items on a CRTs for purposes of revising the test  The items with the highest values should generally be kept in both cases, DI and BI  Making these decisions not as simple as it is for NRT development, because a CRT item may not be performing well in terms of these statistics for many reasons.

Reasons that a CRT item may not be performing well  The item is written/ working poorly  The objective the item is testing is vague  Ss are not ready to learn this particular objective  One/ all of the teachers did not teach this particular objective or teaching it poorly  The materials are confusing with regard to this particular objective  The statistics cannot tell you exactly what is wrong even though they can point you to places in your curriculum where something is not working well

 Some common-sense analysis of the entire situation needs to be done  To revise the CRT or other aspects of your curriculum such as the objectives, the materials, the teaching, etc.,

What the statistics help you?  Help you figure out where to focus your energies  The DI will tell you how well each item fits the objectives of your curriculum  The BI will tell you how each item is contributing to the pass/fail decision that you must make at whatever cut- point you are using.

2 Definitions for “Criterion” in CRT 1.Refers to the material being taught in the course --CRT would assess the particular learning points of a particular course or program  This definition fits very well with the difference index, which indicates how well each item fits objectives of the curriculum

2 Definitions for “Criterion” in CRT 2. The standard of performance (or cut-point for decision making) that is expected for passing the test/course --CRTs – used to assess whether students pass or fail at a certain criterion level (or cut-point)  Fits very well with the B-index, which indicates how well each item is contributing to the pass/fail decision that you must make at whatever cut- point your are making

What should you focus?  If you are interested in the degree to which your items are reflecting the material in your course ….DI  If you are interested in the degree to which your items are helping you make decisions at a certain cut- point … BI  If you are interested in both statistics … DI and BI

What should they not be used?  Not used to analyze the effectiveness of norm-referenced items

What is the Ultimate Goal?  To produce a curriculum and CRTs that match each other such that you get high difference indexes and high B-indexes.