Introduction to Smartphone Sensors Uichin Lee KAIST KSE Slides based on: Sensor Fusion on Android Devices – A Revolution in Motion Processing, David Sachs, Feb. 2010 Android Sensors, Stefan Varga, Michal Kostic, Oct. 2010

Contents Sensor applications Programming Android sensors Setting up sensors Processing events Smartphone sensors Accelerometer, Magnetometer, Gyroscope, Light Sensor, Proximity Sensor, GPS, Microphone, etc.

Applications Resizing screen/tilt Environment adjustment of apps for user’s comfort Adjustment in cinema, movement prediction Gaming Augmented Reality (AR) AR Gaming AR Navigation Bar codes Geo–tagging, recommendations.. Network of objects, locations and people, 3D social networking? Distributed sensor system Noise mapping, traffic information, etc. And anything you can imagine… 

Android Phones and Sensors http://en.wikipedia.org/wiki/Nexus_S From web sources - might not be complete, plus some brands have several versions of their phones with different hw setups! Updates: Nexus S, iPhone4 and iPad2 have gyroscope

Smartphone Sensors Accelerometer Magnetometer (digital compass) Gyroscope Light Pressure Proximity Temperature Camera Voice

Android API (I.) Package: android.hardware Classes: SensorManager – android service Sensor – specific sensor SensorEvent – specific event of the sensor = data

SensorManager Most sensors interfaced through SensorManager or LocationManager Obtain pointer to android service using Context.getSystemService(name) For name, use constant defined by Context class SENSOR_SERVICE for SensorManager LOCATION_SERVICE for LocationManager Check for available sensors using List<Sensor> getSensorList(int type) Type constants provided in Sensor class documentation

SensorManager Use getDefaultSensor(int type) to get a pointer to the default sensor for a particular type Sensor accel = sensorManager.getDefaultSensor( Sensor.TYPE_ACCELEROMETER); Register for updates of sensor values using registerListener(SensorEventListener, Sensor, rate) Rate is an int, using one of the following 4 constants SENSOR_DELAY_NORMAL (delay: 200ms) SENSOR_DELAY_UI (delay: 60ms) SENSOR_DELAY_GAME (delay: 20ms) SENSOR_DELAY_FASTEST (delay: 0ms) Use the lowest rate necessary to reduce power usage Registration will power up sensor: mSensorService.enableSensor(l, name, handle, delay);

SensorManager Unregister for sensor events using unregisterListener(SensorEventListener, Sensor) or unregisterListener(SensorEventListener) Undregistering will power down sensors: mSensorService.enableSensor(l, name, handle, SENSOR_DISABLE) Perform register in OnResume() and unregister in OnPause() to prevent using resources while your activity is not visible SensorListener is deprecated, use SensorEventListener instead See documentation for Sensor, SensorManager, SensorEvent and SensorEventListener

SensorEventListener Must implement two methods SensorEvent onAccuracyChanged(Sensor sensor, int accuracy) onSensorChanged(SensorEvent event) SensorEvent int accuracy Sensor sensor long timestamp Time in nanoseconds at which event happened float[] values Length and content of values depends on sensor type

API – Setup public class MainActivity extends Activity implements SensorEventListener { .. private SensorManager sm = null; … public void onCreate(Bundle savedInstanceState) { sm = (SensorManager) getSystemService(SENSOR_SERVICE); } protected void onResume() { List<Sensor> typedSensors = sm.getSensorList(Sensor.TYPE_LIGHT); // also: TYPE_ALL if (typedSensors == null || typedSensors.size() <= 0) … error… sm.registerListener(this, typedSensors.get(0), SensorManager.SENSOR_DELAY_GAME); // Rates: SENSOR_DELAY_FASTEST, SENSOR_DELAY_GAME, // SENSOR_DELAY_NORMAL, SENSOR_DELAY_UI

API – Processing Events public class MainActivity extends Activity implements SensorEventListener { .. private float currentValue; private long lastUpdate; … public void onSensorChanged(SensorEvent event) { currentValue = event.values[0]; lastUpdate = event.timestamp; } It is recommended not to update UI directly!

API – Cleanup public class MainActivity extends Activity implements SensorEventListener { … protected void onPause() { sm.unregisterListener(this); } protected void onStop() { ..

Accelerometer Mass on spring -1g 1g Gravity Free Fall Linear Acceleration Linear Acceleration plus gravity 1g = 9.8m/s2

Accelerometer http://www.chipworks.com/ja/technical-competitive-analysis/resources/recent-teardowns/2010/06 STMicroelectronics STM331DLH three-axis accelerometer (iPhone4)

Accelerometer Sensor.TYPE_ACCELEROMETER Values[3] = m/s2, measure the acceleration applied to the phone minus the force of gravity (x, y, z) GRAVITY_EARTH, GRAVITY_JUPITER, GRAVITY_MARS, GRAVITY_MERCURY, GRAVITY_MOON, GRAVITY_NEPTUNE

Compass Magnetic field sensor (magnetometer) Magnetic declination Magnetic north Geographic north Magnetic inclination Horizontal Gravity Magnetic field vector X Y Z 3-Axis Compass? Z X Y

Compass 3-Axis Compass Hall Effect The Hall Effect More complicated magnetometers, such as those used on spacecraft, use a variety of methods to detect magnetic field strength and detection. The most common magnetometers are called Solid-State Hall Effect sensors. These sensors use properties of electrical current that are affected by the presence of a magnetic field that does not run parallel to the direction of the current. When there is a magnetic field present, the electrons (or their opposite, electron holes, or both) in the current gather on one side of the conductive material. When it is absent, the electrons or holes run in a basically straight line. The way a magnetic field affects the motion of the electrons or holes can be measured and used to determine the direction of a magnetic field. Hall Effect sensors also produce a voltage that is proportional to the strength of the magnetic field, making them both vector and scalar magnetometers. Read more: How Does a Magnetometer Work? | eHow.com http://www.ehow.com/how-does_4913575_a-magnetometer-work.html#ixzz1ISbutxUX Hall Effect

Compass Sensor.TYPE_MAGNETIC_FIELD values[3] = in micro-Tesla (uT), magnetic field in the X, Y and Z axis SensorManager’s constants MAGNETIC_FIELD_EARTH_MAX: 60.0 MAGNETIC_FIELD_EARTH_MIN: 30.0

Orientation Sensor Sensor.TYPE_ORIENTATION Deprecated A fictitious sensor: orientation is acquired by using accelerometer and compass Deprecated Now use getOrientation (float[] R, float[] result) Values[3] – (Azimuth, Pitch, Roll) Azimuth, rotation around the Z axis 0 <= azimuth <= 360, 0 = North, 90 = East, 180 = South, 270 = West Pitch, rotation around the X axis -180 <= pitch <= 180 0 = sunny side up 180, -180 = up side down -90 = head up (perpendicular) 90 = head down (perpendicular) Roll, rotation around the Y axis -90<=roll <= 90 Positive values when the z-axis moves toward the x-axis.

Orientation: Why Both Sensors? We need two vectors to fix its orientation! (gravity and magnetic field vectors) Tutorial: http://cache.freescale.com/files/sensors/doc/app_note/AN4248.pdf

Gyroscope Angular velocity sensor Coriolis effect – “fictitious force” that acts upon a freely moving object as observed from a rotating frame of reference http://www.sensorwiki.org/doku.php/sensors/gyroscope http://www.cs.cmu.edu/afs/cs.cmu.edu/academic/class/16311/www/current/ppp/INS3.pdf

Gyroscope http://memscentral.com/Secondlayer/mems_motion_sensor_perspectives-coriolis-force-mems-gyroscope-principle.htm

Gyroscope Sensor.TYPE_GYROSCOPE Measure the angular velocity of a device Detect all rotations, but few phones have it iPhone4, iPad2, Samsung Galaxy S, Nexus S Values[] – iPhone 4 gives radians/sec, and makes it possible to get the rotation matrix

Accelerometer vs. Gyroscope Senses linear movement, but worse rotations, good for tilt detection, Does not know difference between gravity and linear movement Shaking, jitter can be filtered out, but the delay is added Gyroscope Measure all types of rotation Not movement Does not amplify hand jitter A+G = both rotation and movement tracking possible

How to Use the Data – Example float[] matrixR = new float[9]; float[] matrixI = new float[9]; SensorManager.getRotationMatrix( matrixR, matrixI, matrixAccelerometer, matrixMagnetic); float[] lookingDir = MyMath3D.matrixMultiply(matrixR, new float[] {0.0f, 0.0f, -1.0f}, 3); float[] topDir = MyMath3D.matrixMultiply(matrixR, new float[] {1.0f, 0.0f, 0.0f}, 3); GLU.gluLookAt(gl, 0.4f * lookingDir[0], 0.4f * lookingDir[1], 0.4f * lookingDir[2], lookingDir[0], lookingDir[1], lookingDir[2], topDir[0], topDir[1], topDir[2]);

Accelerometer Noise - Simple Exponential moving average const float kFilteringFactor = 0.1f; //play with this value until satisfied float accel[3]; // previous iteration //acceleration.x,.y,.z is the input from the sensor accel[0] = acceleration.x * kFilteringFactor + accel[0] * (1.0f - kFilteringFactor); accel[1] = acceleration.y * kFilteringFactor + accel[1] * (1.0f - kFilteringFactor); accel[2] = acceleration.z * kFilteringFactor + accel[2] * (1.0f - kFilteringFactor); result.x = acceleration.x - accel[0]; result.y = acceleration.y - accel[1]; result.z = acceleration.z - accel[2]; Return result;

Accelerometer Noise – Notes If it is too slow to adapt to sudden change in position, do more rapid changes when angle(accel, acceleration) is bigger You can throw away single values that are way out of average. |acc| does not have to be equal to |g| ! Kalaman filters – too complicated?

Other sensors Light sensor Proximity sensor Temperature sensor Pressure sensor

Light sensor Sensor.TYPE_LIGHT values[0] = ambient light level in SI lux units SensorManager’s constants LIGHT_CLOUDY: 100 LIGHT_FULLMOON: 0.25 LIGHT_NO_MOON: 0.001 LIGHT_OVERCAST: 10000.0 (cloudy) LIGHT_SHADE: 20000.0 LIGHT_SUNLIGHT: 110000.0 LIGHT_SUNLIGHT_MAX: 120000.0 LIGHT_SUNRISE: 400.0

Proximity sensor Sensor.TYPE_PROXIMITY values[0]: Proximity sensor distance measured in centimeters (sometimes binary near-far)

Temperature sensor Sensor.TYPE_TEMPERATURE values[0] = temperature

Pressure sensor Sensor.TYPE_PRESSURE values[0] = pressure no constants

Videos Sensor Fusion: 15:22-20:27 Jobs video..