Physics 114: Lecture 6 Measuring Noise in Real Data John Federici NJIT Physics Department

Physics Cartoons

REMINDER Quiz on weeks 1,2,3 material in TUESDAY’s class. About 15 min. Please be on time. No extra time will be allotted for late students.

Mean and Standard Deviation Sample Mean Parent population mean Standard Deviation from sample mean Standard Deviation from parent population mean

Averaging of Data - Example Readout from Infrared Detector Array Bottom curve – No averaging. Is there a signal present? Top curve (Offset) - more averaging Detected Voltage (Arb. Units) Wavelength (Arbitrary Units) Fowler and Gatly, Astro. J 353, L33-34, (1990)

Averaging of Data - Example Readout from Infrared Detector Array Bottom curve – No averaging. Is there a signal present? Top curve (Offset) - more averaging Detected Voltage (Arb. Units) SIGNAL level is comparable to or smaller than NOISE Level AVERAGING will reduce the noise to a level that SIGNAL is LARGER than noise. Wavelength (Arbitrary Units) Fowler and Gatly, Astro. J 353, L33-34, (1990)

How Many Times Should I Average? If the Parent Distribution has a standard deviation of 10% x=10+randn(1,1000); % create data with a 'true' value of 10 and noise with standard deviation % of 1. True value will vary within about 10% for i=2:1000 j(i)=i; MyStanDev(i)=std(x(1:i)); end plot(j,MyStanDev)

How Many Times Should I Average? If the Parent Distribution has a standard deviation of 30% With increased noise, you need to AVERAGE MORE to get close to the PARENT Distribution standard Deviation. Question: Are the averaging of three data values in Physics Lab sufficient to achieve the PARENT DISTRIBUTION standard deviation? x=10+3*randn(1,1000); % create data with a 'true' value of 10 and noise with standard deviation % of 1. True value will vary within about 30% for i=2:1000 j(i)=i; MyStanDev(i)=std(x(1:i)); end plot(j,MyStanDev)

Hw#3 Data – THz Transmission Terahertz Time-Domain Data – Recall the basic experimental layout Think of THz pulses of radiation as similar to a Radar pulse. Pulses through sample DELAYED in time because then travel SLOWER

Refractive Index From time delay between pulses, one can measure the speed of light in the sample material and its ‘index of refraction’ Speed of light in vacuum (~3×108 m/s) Speed of light in material L

Measuring Dt between two pulses In order to measure Dt accurately between two pulses, you need to determine where the peak of the pulse is located in time. However, each measurement of the THz voltage at a particular time has some error associated with it. 1 Average Blow up section of figure using MAGNIFYING GLASS icon in figure

How do you remove ‘Statistical’ Uncertainty? AVERAGING will remove statistical noise, but SYSTEMATIC Noise (and the signal you are looking for) will remain. 1 Avg 10 Avg 100 Avg 3 Avg

How do you remove ‘Statistical’ Uncertainty? AVERAGING will remove statistical noise, but SYSTEMATIC Noise (and the signal you are looking for) will remain. 1 Avg 100 Avg Now it is easy to determine the time of the MINIMUM of the peak

Systematic and Statistical Noise Example If one were to BLOCK the THz Beam in the transmission measurement, what waveform would you measure?

HomeWork #3 Is the noise in the THz transmission measurement statistical? 1 average

Does Averaging help? 10 Avg 1 Avg 10Avg 1 Avg

What if we average more? 600 Averages What is THIS oscillatory signal? The THZ beam is blocked. What type of ERROR is this signal? SYSTEMATIC Noise due to problem with experimental instrumentation

Data, Systematic Error, Random Noise How do we know that this oscillatory waveform is not the ‘Real DATA’ ? ie. not the THz signal? How do we know that this waveform is NOT statistical noise? How do we know that this waveform is probably a Systematic Error? THz signal is blocked! After many averages, the persistent shape of the waveform (ie. sine wave) does not average out. Statistics do not look Gaussian THz Signal Blocked. After many averages, waveform persists.

This Systematic Error is Intermittent 10 Avg 100 Avg Best solution is to FIX the instrumentation to remove Systematic Errors.

More evidence of Systematic Error

Correct for Systematic Errors If Systematic Errors are NOT intermittent, then we can subtract out the Systematic Errors Subtract

In Class Exercise Begin on weekly HW assignment Problem 2. (a) Download the THZ data set for Week 3 Homework from the course web site, in text format. Use the IMPORT DATA tool to read the data from the various files. Each file corresponds to a DIFFERENT number of averages as indicated in the file name. (b) Import each data file and then calculate the standard deviation for the E Field (second column of data). Save the standard deviation values in an array. Save a second array whose elements are the NUMBER of averages. As a guide, look at the table below as a guide to the two vector arrays which you will create: # of averages Standard deviation 1 std of AVG1 data 3 std of AVG3 data 6 std of AVG6 data 10 std of AVG10 data … continuation of data