Comparison of two Normalization Procedures Results of some statistical analysis

Normalization procedure 1 Get percentile (P) of a student on the basis of 5- subject aggregate scores in board. Determine the JEE-Main aggregate marks corresponding to percentile P at the All-India level. Regard this as the normalized board score of the student (B 1 ). Get JEE-Main aggregate marks (A 0 ) of the student. The composite marks used for drawing merit list is A 1 = 0.6 A B 1.

Normalization procedure 2 Get percentile (P) of a student on the basis of 5-subject aggregate scores in board. Determine the JEE-Main aggregate marks corresponding to percentile P from the set of aggregate marks obtained by different students of that board in JEE-Main. Regard this as the normalized board score of the student (B 2 ). Get JEE-Main aggregate marks (A 0 ) of the student. The composite marks used for drawing merit list is A 2 = 0.6 A B 2.

Common assumptions aggregate scores of students in a board examination are in the order of their general merit within that board (scholastic achievement indicates ability). aggregate scores of the students in JEE-Main are in the order of their merit in respect of engineering admissions (achievement indicates aptitude). Assumptions may be violated if a student has a poor exam, A student has a special advantage. If these effects are confounded with merit, violations become difficult to rectify. No fairer assumption is possible.

Exclusive assumptions Procedure 1 General merit distribution does not vary from one board to another. Procedure 2 Merit distribution varies from board to board; this difference is captured by differences in performance patterns of students of various boards in JEE-Main.

Data for analysis One data common data set: Aggregate marks in AIEEE 2012 and the corresponding percentiles computed from the set of all students taking the examination. Three sets of data in respect of six boards: Aggregate marks in board examinations 2012 (expressed as percentage) and the corresponding percentiles for all students taking the examination. Aggregate marks in AIEEE 2012 and the corresponding percentiles computed from the set of students from the chosen board. Matched pair of AIEEE aggregate marks and board aggregate percentage.

Issue 1: Extent of ‘compensation’ for board percentile ‘disparity’

‘Compensation’ for ‘disparity’: some specific examples Transformed score of a student at the 50th percentile of CBSE is more than that of – a student at the 80th percentile of Maharashtra Board, – a student at the 85th percentile of Jharkhand Board. Transformed board scores of top five students of three boards: – CBSE: 346, – Maharashtra board : 331, – Jharkhand board : 274. Amount of compensation (by Procedure 2) is LARGE!

What is being ‘compensated’? Poor performance of students of some boards in AIEEE may be due to – Poor merit/ability (not a tenable explanation for large boards), – Non-alignment of board examination pattern with AIEEE (rank correlations indicate this), – Lack of instruction in English and Hindi (only available languages for AIEEE/JEE-Main), – Less access to coaching, – Load of an extra subject in board (for some boards). All these effects are confounded. If confounding factors are ignored, solution may be worse than the problem.

Issue 2: Performance pattern of different boards (distribution of students in different percentile ranges) Current procedure Procedure 1 Procedure 2 Disparity among the distributions is the least for Procedure 1.

Issue 3: The question of fairness In 2012, five students (4 from CBSE, 1 from Maharashtra board) were tied at – AIEEE aggregate marks of 130, – board percentile of According to Procedure 1, they shared rank 20,352. According to Procedure 2 – CBSE students had shared rank 18,077, – Maharashtra board students had rank 34,175. The differential treatment is entirely due to performance of other students of the two boards in AIEEE. This goes against the principle of fairness.

Issue 4: Sampling bias Procedure 2 requires the assumption Board to board difference in merit distribution is captured by differences in performance patterns of students of various boards in JEE-Main. Proportion of AIEEE candidates in percentile range (board aggregate) Board Above 80Total Assam0% 2%98%100% CBSE29%15%19%37%100% Jharkhand16%5% 74%100% Maharashtra11%8%11%69%100% Mizoram1% 2%96%100% Uttarakhand14%3%4%79%100% For this assumption to hold, sampling of AIEEE candidates from the boards should follow the same pattern. However, 2012 data shows that AIEEE attracts different profiles of students from different boards.

In summary Students of different boards face different types/levels of difficulty in AIEEE. The presence of these factors confound any presumed board-to-board difference in merit distributions. Board-to-board difference in performance patterns in AIEEE is vast. Use of AIEEE marks for ‘tracking’ would produce drastic amount of compensation. Such levels of unequal treatment of percentiles of different boards, based on presumptions, is unfair. Procedure 2 also penalizes a student for poor performance of peers from same board. For large boards, there is a strong reason to presume a common merit distribution. Procedure 1 produces a more equitable pattern of performance of students of different boards.

Criteria that are NOT appropriate for choosing between procedures Correlation of new and old ranks. – How much correlation is ideal? – The more the merrier? Movement of students from one merit list to another. – How much movement is ideal? – The less the merrier?

A benchmark for rank correlations Rank correlation between board and AIEEE marks Board Spearman rank correlation with AIEEE aggregate marks Assam CBSE Jharkhand Maharashtra Mizoram Uttarakhand All matched samples Observations No rank correlation is more than 0.7. Board examinations are different from AIEEE (measure something else). CBSE marks have highest rank correlation with AIEEE: – Similarity of examination pattern, – Same agency administers CBSE exams and AIEEE.

Correlation of new and old ranks Board Rank correlation between A 0 and A 1 Rank correlation between A 0 and A 2 Assam CBSE Jharkhand Maharashtra Mizoram Uttarakhand All matched samples Not much difference overall, and for CBSE. For some boards, Procedure 1 produces lower correlation. This is in line with observed low correlation between board and AIEEE marks. Why should large correlation be a good thing?

Movement from old to new rank list Procedure 1 produces greater movement. This is in line with observed disparity between board and AIEEE marks.

Some questions of interest to CBSE 1.What do the correlation patterns and rank movements indicate about the fate of CBSE students? 2.Will the CBSE students be at a disadvantage under Procedure 1? 3.Will they get a better deal under Procedure 2? 4.Will smaller boards gain unfairly from Procedure 1? 5.Is there a possibility of ‘board hopping’ if Procedure 1 is used?

Q1: Correlation patterns and rank movements Correlation and movement patterns for the two procedures are SIMILAR. Correlations indicate degree of association/alignment between new and old ranks. Correlations do not reveal ANYTHING about the relative advantage or disadvantage to CBSE students. If all CBSE students are pushed down 10,000 places in AIEEE merit list, their rank correlation with board scores would stay the same. Scatter-plots of rank movement show that students exchange fate with one another. – no clue about advantage/disadvantage to CBSE students.

New simulations to answer Q2-Q5 Need simulated data for ALL the boards Simulation conditions – 10% Random samples chosen from AIEEE 2012 scores data. – Presumed correlation between AIEEE and board scores: Observed correlations for AS, CBSE, JK, MR, MZ, UK 0.4 for other boards. – 100 separate simulation runs.

Actual performance of CBSE students in AIEEE % of all AIEEE candidates are CBSE students. However, this groups constitutes – 66 of the top 100 AIEEE rand holders, – 608 of the top 1,000 AIEEE rand holders, – 5,977 of the top 10,000 AIEEE rand holders, – 54,430 of the top 100,000 AIEEE rand holders.

Share of boards in different percentile ranges of AIEEE 2012

How Procedure 1 or 2 would change the shares Procedure 1 reduces marginally the dominance of CBSE students in merit list, Procedure 2 increases drastically that dominance.

…and the numbers behind the charts Procedure 1: Share of boards in different percentile ranges of AIEEE 2012 Board CBSE15%18%21%24%26%29%33%38%44%53% MR17%16%14%13%12%11%10%8%7%5% AP7%8% 9% 10%13% BR10%9% 8% 7%6% 5%3% All other51%49%48%47%46%44%42%39%35%26% Procedure 2: Share of boards in different percentile ranges of AIEEE 2012 Board CBSE11%14%17%21%25%29%35%42%48%59% MR20%18%16%14%12%11%9%7%5%4% AP6% 7%8% 9% 10%11%13% BR11%10% 9%8%7%6%5%4%2% All other53%51%50%49%47%44%41%36%31%22% Current scenario: share of boards in different percentile ranges of AIEEE 2012 Board Overall CBSE14%17%20%22%26%28%33%39%46%54%30% MR17% 15%14%12%11%9%8%6%5%11% AP7%8% 9% 10%13%9% BR10%9% 8% 7%6%5%4%3%7% All other51%49%48% 46%45%42%38%33%25%43%

Findings from simulation Present scenarioProcedure 1Procedure 2 Top 0.01%65.1%61.9%82.7% Top 0.1%60.5%60.4%80.1% Top 1%60.0% 69.6% Top 10%54.4%53.4%58.8% Procedure 1 produces a marginal reduction in CBSE’s share at the top; 2% reduction at the top 0.01%. Procedure 2 brings a great windfall to CBSE students. CBSE students gain more and more from Procedure 2 as one goes up the merit list (share at top 1% and top 0.1% increases by 10% and 20%, respectively ). If interest of top CBSE students had been the criterion, Procedure 2 would be a clear winner!

How does Procedure 2 dramatically increase CBSE’s share of top-rankers? It does so by practically turning the board score into another AIEEE score. The composite score A 2 for each board has same mean as A 0, but has much smaller spread. Each set of board scores get more concentrated around their respective means. In effect, every students gets a second chance to perform at the same mean level. As a result, CBSE students gain substantially.

Why does Procedure 1 have only a mild effect on CBSE’s share of top-rankers? For CBSE, the students who do well in AIEEE are generally those who do well in board exam. – They mostly hold on to their position in merit list. For boards with less rank correlation with AIEEE, this is not the case. – Many AIEEE top-rankers from other boards move towards middle in terms of composite score. – If their board exams had been better aligned with AIEEE, they would have gained more from Procedure 1.

Answers to Q2 and Q3 Q1: Will the CBSE students be at a disadvantage under Procedure 1? – Yes, but marginally so. Q2: Will they get a better deal under Procedure 2? – Yes, the top third of the merit list gets PACKED with CBSE students. – It is not just a better deal, it is a great deal. – It would not be fair to other boards!

Q4: Will smaller boards gain unfairly from Procedure 1? A small board may not have same merit distribution as others. It may have – Stronger students, or – Weaker students. To say that they gain more (from Procedure 1) than larger boards is to PRESUME that they have weaker students. Percentiles generally correct for board size. Here, percentile of a student is defined as percentage of candidates strictly below that student. Toppers of a smaller board will always have a smaller percentile than the toppers of a larger board. Students of smaller boards are also more vulnerable to variations of all kinds.

Q5: Will Procedure 1 encourage board hopping? Yes, but TOWARDS CBSE (continuation of current trend) – CBSE’s loss and other boards’ gain is marginal. – This loss is only in relation to the present scenario; 2013 students cannot hop into 2012 system. – Even after this minor loss, CBSE students will have continued advantage over students of most other boards in terms of alignment of examination pattern alignment of language access to coaching a well administered examination system (less uncertainty) – If a CBSE student hops to a smaller board, good marks are not guaranteed (exam is likely to be of a different type) Confounding advantages in AIEEE should not give them false confidence.

Composition of different percentile groups in 2012 by Procedure 1