Chapter 2 Simple Linear Regression – What is it, Why do we do it?

Remember Statistics is an applied branch of mathematics. When you apply mathematics to describe the world we live in, we call this mathematical modeling. Up to this point in your mathematical studies you have been learning about the language of algebra, which involves learning the about the objects – like functions – that make up this language, its characteristics, syntax, and grammar. The big concept in algebra is the understanding of relationships between variables. The simplest mathematical relationship being linear relationships. After a while we learn about a special type of relationship called a function, symbolized by f(x) = y.

The types of relationships/functions we learn about in algebra are called deterministic. What is deterministic? This means that we have a perfect relationship between the variables in question. If we know a particular value of one variable, then knowing what the relationship is – meaning knowing the equation- gives us complete knowledge of what is the corresponding value of the other variable. Here is a simple example. You work at a job that pays you $10.00 per hour. Let h equal the number of hours you work, d equal the amount of dollars you earn and f(h) = d equal the symbolic representation of the relationship between h and d. So, if on your first week on the job you work 20 hours your gross pay will be $ without question, f(20) = $ The relationship is exact!

If the following week, you work 20 hours again your pay will be $ again. This is different from a probabilistic model. In a probabilistic model there is a relationship between two variables but that relationship is not perfect. For the value of a particular variable we have an expectation of a value for the other variable in the relationship, but we can not expect to get that exact value. Here is a simple example. You are a vendor at an outdoor market. Let h represent the number of hours you work at the outdoor market, and let d equal the net amount of money you earn from selling your products. If you work for 8 hours you will earn some amount of money, but you can not predict from day to day what that amount will be. You have an expectation of what you will earn, otherwise you would not be involved in this endeavor.

But suppose that you worked for 8 hours, for many days at your stall in the outdoor market. Eventually you would have gathered enough data to create a distribution of sales and would begin to see a pattern emerge. Soon you would recognize that this pattern repeats itself, except that you can not predict exactly how much money you will earn in any one week. But depending on the time of the season, or month you can expect a certain return for your work. Lets say that sales always seem to be higher at the end of the month. So there is a relationship between the two variables, except that it is not deterministic- exact - it is probabilistic.

So here is where we begin talking about the topic of chapter 2, simple linear regression. The attempt here is to understand what relationship exists between the two variables under a probabilistic model. Before we start let us cover a couple of preliminary concepts. As was mentioned, we are dealing with a probabilistic model, which means our functions will also be probabilistic. More on what this means in a moment. An important fact is that we will be dealing with linear equations/functions only to make our task easier.

Now, what do we mean by a probabilistic linear function? Let x be our independent variable, which we will call, most of the time, the explanatory variable. In algebra the letter y represent the dependent variable, which we now call the response variable (more on the name change later). Using algebra symbols, f is the name of the linear function, and f(x) = y. But since we are dealing with a probabilistic model, that is x and y are not perfectly related it seems that we have the following situation: f(3) = ?. What exactly is the output of a probabilistic equation that defines the relationship between two variables?

Alright, here it is. Let the variable h, the explanatory variable be the number of hours you work at the market selling your product. Let us say you work 8 hours every Saturday. The variable d represents the gross amount of dollars earned for working 8 hours on Saturday. The function f represents the mathematical relationship between the two; so f(8) equals the expected (mean) gross amount of dollars earned. In other words f(8) = the mean for the given situation. Thus, f(8) represents the mean of some distribution.

We will make some assumptions about our distribution; f(8) is the mean dollar value of the sales. Secondly, we will assume a normal distribution.

We realize that there is also then a value associated for any value of h, number of hours worked. Thus f(4) = has some value, and f(5) = has some value, and f(3.5) = has some value and so on. The output is always the mean number of sales in dollars associated with that number of hours worked, and we will assume that the distributions are all normal. Also all the normal distributions have the same standard deviation.

Ok, this is understandable so far. But what exactly is simple linear regression? What is chapter 2 about?

We want to find the equation f(x) = y or in this example f(h) = d. What is this equation that explains the relationship between the variable x and the variable y, or in our example h and d, which represents the mean of a normal distribution. We will only be considering situation when f(x) is a linear equation, linear relationship. So one task is to determine if our relationship is linear, how good is the linear relationship, and if it will it provide any good information We will assume that for each value of x, f(x) is the mean of a normal distribution.