Class 3: Thursday, Sept. 16 Reliability and Validity of Measurements Introduction to Regression Analysis Simple Linear Regression (2.3)

Correlation and Reliability of Measurements Measurement theory: Branch of applied statistics that attempts to describe, categorize, evaluate and improve the quality of measurements. Measurement theory for psychological attributes such as intellectual ability or personality is called psychometrics. Reliability of a measurement: The degree of consistency with which a trait or attribute is measured. A perfectly reliable measurement will produce the same value each time assuming the trait or attribute remains constant. Validity of a measurement: The degree to which a measurement measures what it purports to measure. The reliability of a measurement is often determined by the correlation between repeated measurements of the same trait/attribute.

Reliability of Pulse Measurement Correlation =

Types of Reliability External reliability (stability): Does the measurement vary from one use to another? Internal reliability (Internal consistency): Is the measurement consistent within itself? Checking external reliability: Test-retest method. Correlation between the scores of the same people on two different administrations of test (e.g., pulse measurement). Checking internal reliability of a test: Split-half method. Randomly split a test into two halves. Compute correlation of scores on two halves.

Validity Validity of a measurement: The degree to which a measurement measures what it purports to measure. To what extent is one’s pulse a valid measure of one’s general state of health. What time did you go to bed last night? –Is it reliable as a measure of when you went to bed last night? –Is it reliable as a measure of your average bed time? –Is it valid as a measure of what time you actually went to bed? –Is it valid as a measure of time spent studying?

Evidence for Validity of a Test 1.Face validity: –Definition: Items on a test should look like they are covering the proper topics. –Example: Math test should not have history items. 2.Content validity: –Definition: Test covers range of material it is intended to cover. –Example: SAT should cover different areas of math and verbal ability.

Evidence for Validity of a Test 3.Predictive validity (criterion validity) –Definition: Test scores should be positively associated with real world outcomes that the test is supposed to predict. –Example: Correlation between SAT verbal scores and freshman grades is 0.47, correlation between SAT math scores and freshman grades is Construct validity –Definition: Test scores should be positively associated with other similar measures and should not be associated with irrelevant measures. –Example: SAT scores should be positively associated with high school GPA and other academic ability tests. SAT scores should not be associated with political attitudes.

Regression Analysis Setup: Response variable Y. Explanatory (predictor) variables Goal: (1) Understand how changes in are associated with changes in Y; (2) Predict Y based on Examples of applications: – Building an spam filter. Predict whether an e- mail is spam based on the frequency of certain words and punctuation. –The product manager in charge of a brand of children’s cereal would like to predict demand during the next year. She has available the following “predictor” variables: price of the product, number of children in target market, price of competitors’ products, effectiveness of advertising, annual sales this year and previous year

Simple Regression Analysis There is only one explanatory variable X. Example: A real estate agent would like to understand the relationship between Y=selling price of a house (dollars) and X=house size (square feet) and to predict selling price based on house size. To build a regression model, the real estate agent (who lives in Gainesville, FL) randomly samples 93 recently sold homes in Gainesville and finds out their selling price and their size (Data from Agresti and Finlay, Statistical Methods for the Social Sciences)

Regression Model There is a population of units, each of which has a (Y,X) value. Regression model: A model for the mean (expected value) of Y for the subpopulation of units with, denoted by. The model has a deterministic and a random component: The Y-value for unit i equals the mean value of Y given plus a random error that has mean zero. Application of regression model to prediction: A good prediction of Y for a unit with is. Example: A house has a size of 1000 feet. A good prediction of the house’s selling price is the mean selling price of all houses of size 1000 feet.

Simple Linear Regression Model The simple linear regression model is that the mean of Y given X is a straight line: Example:. For each additional one unit increase in X, the amount by which the mean of Y given X increases (or decreases), e.g., for each additional additional square foot of house size, mean selling price increases by 75. intercept. The mean of Y for X=0. Mean selling price of an empty lot is $25,000.

House size House Cost Most lots sell for $25,000 Building a house costs about $75 per square foot. House cost = (Size) Simple Linear Regression Model The model has a deterministic and a probabilistic components

House cost = (Size) House size House Cost Most lots sell for $25,000   However, house cost vary even among same size houses! Simple Linear Regression Model Since cost behave unpredictably, we add a random component.

Estimating the Slope and Intercept The estimates are determined by –drawing a sample from the population of interest, –calculating sample statistics. –producing a straight line that cuts into the data.           Question: What should be considered a good line? x y

The Least Squares Principle A good line is one that minimizes the sum of squared differences between the points and the line.

The Least Squares (Regression) Line 3 3     (1,2) 2 2 (2,4) (3,1.5) Sum of squared differences =(2 - 1) 2 +(4 - 2) 2 +( ) 2 + (4,3.2) ( ) 2 = 6.89 Sum of squared differences =(2 -2.5) 2 +( ) 2 +( ) 2 +( ) 2 = Let us compare two lines The second line is horizontal The smaller the sum of squared differences the better the fit of the line to the data.

Regression Analysis in JMP Use Analyze, Fit Y by X. Put response variable in Y and explanatory variable in X (make sure X is continuous by clicking on the X column, clicking Cols and Column Info and checking that the Modeling Type is Continuous). Click on fit line under red triangle next to Bivariate Fit of Y by X.

Prediction Least squares estimate of regression model: = Predicted selling price for a house that is 1000 square feet = $50,398 Predicted selling price for a house that is 0 square feet = -$25,222 !

Extrapolation Extrapolation: Use of the regression model to predict far outside the range of values of the explanatory variable X that you used to obtain the line. Such predictions are often not accurate.

Olympic Long Jump: Length of gold medal jump (Y) vs. Year (X)

Predictions from Long Jump Simple Linear Regression Model Predicted olympic gold medal winning long jumps: – 2008 (Beijing): *2008 = feet –2028: *2028 = feet –3000: *3000 = feet

Summary A good test is both reliable and valid. Correlation is useful in measuring reliability and predictive validity. Regression Analysis: Model for the mean of Y given X. Useful for predicting Y from X. Simple Linear Regression Model: Interpretation of slope and intercept, estimation of slope and intercept by least squares. Be careful about extrapolation when using a regression model to predict Y from X. Next class: Finish Chapter 2.3, start Chapter 2.4.