1. Introduction to Smartphone Sensors
Uichin Lee
KAIST KSE
Slides based on:
Sensor Fusion on Android Devices – A Revolution in Motion Processing, David Sachs, Feb. 2010
Android Sensors, Stefan Varga, Michal Kostic, Oct. 2010

2. Contents Sensor applications Programming Android sensors
Setting up sensors
Processing events
Smartphone sensors
Accelerometer, Magnetometer, Gyroscope, Light Sensor, Proximity Sensor, GPS, Microphone, etc.

3. Applications Resizing screen/tilt
Environment adjustment of apps for user’s comfort
Adjustment in cinema, movement prediction
Gaming
Augmented Reality (AR)
AR Gaming
AR Navigation
Bar codes
Geo–tagging, recommendations..
Network of objects, locations and people, 3D social networking?
Distributed sensor system
Noise mapping, traffic information, etc.
And anything you can imagine… 

4. Android Phones and Sensors
From web sources - might not be complete, plus some brands have several versions of their phones with different hw setups!
Updates: Nexus S, iPhone4 and iPad2 have gyroscope

5. Smartphone Sensors Accelerometer Magnetometer (digital compass)
Gyroscope
Light
Pressure
Proximity
Temperature
Camera
Voice

6. Android API (I.) Package: android.hardware Classes:
SensorManager – android service
Sensor – specific sensor
SensorEvent – specific event of the sensor = data

7. SensorManager
Most sensors interfaced through SensorManager or LocationManager
Obtain pointer to android service using
Context.getSystemService(name)
For name, use constant defined by Context class
SENSOR_SERVICE for SensorManager
LOCATION_SERVICE for LocationManager
Check for available sensors using List<Sensor> getSensorList(int type)
Type constants provided in Sensor class documentation

8. SensorManager
Use getDefaultSensor(int type) to get a pointer to the default sensor for a particular type
Sensor accel = sensorManager.getDefaultSensor( Sensor.TYPE_ACCELEROMETER);
Register for updates of sensor values using registerListener(SensorEventListener, Sensor, rate)
Rate is an int, using one of the following 4 constants
SENSOR_DELAY_NORMAL (delay: 200ms)
SENSOR_DELAY_UI (delay: 60ms)
SENSOR_DELAY_GAME (delay: 20ms)
SENSOR_DELAY_FASTEST (delay: 0ms)
Use the lowest rate necessary to reduce power usage
Registration will power up sensor:
mSensorService.enableSensor(l, name, handle, delay);

9. SensorManager Unregister for sensor events using
unregisterListener(SensorEventListener, Sensor) or
unregisterListener(SensorEventListener)
Undregistering will power down sensors:
mSensorService.enableSensor(l, name, handle, SENSOR_DISABLE)
Perform register in OnResume() and unregister in OnPause() to prevent using resources while your activity is not visible
SensorListener is deprecated, use SensorEventListener instead
See documentation for Sensor, SensorManager, SensorEvent and SensorEventListener

10. SensorEventListener Must implement two methods SensorEvent
onAccuracyChanged(Sensor sensor, int accuracy)
onSensorChanged(SensorEvent event)
SensorEvent
int accuracy
Sensor sensor
long timestamp
Time in nanoseconds at which event happened
float[] values
Length and content of values depends on sensor type

11. API – Setup
public class MainActivity extends Activity implements SensorEventListener { ..
private SensorManager sm = null; …
public void onCreate(Bundle savedInstanceState) {
sm = (SensorManager) getSystemService(SENSOR_SERVICE); }
protected void onResume() {
List<Sensor> typedSensors = sm.getSensorList(Sensor.TYPE_LIGHT);
// also: TYPE_ALL
if (typedSensors == null || typedSensors.size() <= 0) … error…
sm.registerListener(this, typedSensors.get(0),
SensorManager.SENSOR_DELAY_GAME);
// Rates: SENSOR_DELAY_FASTEST, SENSOR_DELAY_GAME,
// SENSOR_DELAY_NORMAL, SENSOR_DELAY_UI

12. API – Processing Events
public class MainActivity extends Activity implements SensorEventListener { ..
private float currentValue;
private long lastUpdate; …
public void onSensorChanged(SensorEvent event) {
currentValue = event.values[0];
lastUpdate = event.timestamp; }
It is recommended not to update UI directly!

13. API – Cleanup
public class MainActivity extends Activity implements SensorEventListener { …
protected void onPause() {
sm.unregisterListener(this); }
protected void onStop() { ..

14. Accelerometer Mass on spring -1g 1g Gravity Free Fall
Linear Acceleration
Linear Acceleration plus gravity
1g = 9.8m/s2

15. Accelerometer
STMicroelectronics STM331DLH three-axis accelerometer (iPhone4)

16. Accelerometer Sensor.TYPE_ACCELEROMETER
Values[3] = m/s2, measure the acceleration applied to the phone minus the force of gravity (x, y, z)
GRAVITY_EARTH, GRAVITY_JUPITER,
GRAVITY_MARS, GRAVITY_MERCURY,
GRAVITY_MOON, GRAVITY_NEPTUNE

17. Compass Magnetic field sensor (magnetometer) Magnetic declination
Magnetic north
Geographic north
Magnetic inclination
Horizontal
Gravity
Magnetic field vector X Y Z
3-Axis Compass? Z X Y

18. Compass 3-Axis Compass Hall Effect The Hall Effect
More complicated magnetometers, such as those used on spacecraft, use a variety of methods to detect magnetic field strength and detection. The most common magnetometers are called Solid-State Hall Effect sensors. These sensors use properties of electrical current that are affected by the presence of a magnetic field that does not run parallel to the direction of the current. When there is a magnetic field present, the electrons (or their opposite, electron holes, or both) in the current gather on one side of the conductive material. When it is absent, the electrons or holes run in a basically straight line. The way a magnetic field affects the motion of the electrons or holes can be measured and used to determine the direction of a magnetic field. Hall Effect sensors also produce a voltage that is proportional to the strength of the magnetic field, making them both vector and scalar magnetometers.
Read more: How Does a Magnetometer Work? | eHow.com
Hall Effect

19. Compass Sensor.TYPE_MAGNETIC_FIELD
values[3] = in micro-Tesla (uT), magnetic field in the X, Y and Z axis
SensorManager’s constants
MAGNETIC_FIELD_EARTH_MAX: 60.0
MAGNETIC_FIELD_EARTH_MIN: 30.0

20. Orientation Sensor Sensor.TYPE_ORIENTATION Deprecated
A fictitious sensor: orientation is acquired by using accelerometer and compass
Deprecated
Now use getOrientation (float[] R, float[] result)
Values[3] – (Azimuth, Pitch, Roll)
Azimuth, rotation around the Z axis
0 <= azimuth <= 360, 0 = North, 90 = East, 180 = South, 270 = West
Pitch, rotation around the X axis
-180 <= pitch <= 180
0 = sunny side up
180, -180 = up side down
-90 = head up (perpendicular)
90 = head down (perpendicular)
Roll, rotation around the Y axis
-90<=roll <= 90
Positive values when the z-axis moves toward the x-axis.

21. Orientation: Why Both Sensors?
We need two vectors to fix its orientation! (gravity and magnetic field vectors)
Tutorial:

22. Gyroscope Angular velocity sensor
Coriolis effect – “fictitious force” that acts upon a freely moving object as observed from a rotating frame of reference

23. Gyroscope

24. Gyroscope Sensor.TYPE_GYROSCOPE
Measure the angular velocity of a device
Detect all rotations, but few phones have it
iPhone4, iPad2, Samsung Galaxy S, Nexus S
Values[] – iPhone 4 gives radians/sec, and makes it possible to get the rotation matrix

25. Accelerometer vs. Gyroscope
Senses linear movement, but worse rotations, good for tilt detection,
Does not know difference between gravity and linear movement
Shaking, jitter can be filtered out, but the delay is added
Gyroscope
Measure all types of rotation
Not movement
Does not amplify hand jitter
A+G = both rotation and movement tracking possible

26. How to Use the Data – Example
float[] matrixR = new float[9];
float[] matrixI = new float[9];
SensorManager.getRotationMatrix(
matrixR, matrixI,
matrixAccelerometer, matrixMagnetic);
float[] lookingDir = MyMath3D.matrixMultiply(matrixR,
new float[] {0.0f, 0.0f, -1.0f}, 3);
float[] topDir = MyMath3D.matrixMultiply(matrixR,
new float[] {1.0f, 0.0f, 0.0f}, 3);
GLU.gluLookAt(gl,
0.4f * lookingDir[0], 0.4f * lookingDir[1], 0.4f * lookingDir[2],
lookingDir[0], lookingDir[1], lookingDir[2],
topDir[0], topDir[1], topDir[2]);

27. Accelerometer Noise - Simple
Exponential moving average
const float kFilteringFactor = 0.1f; //play with this value until satisfied
float accel[3]; // previous iteration
//acceleration.x,.y,.z is the input from the sensor
accel[0] = acceleration.x * kFilteringFactor + accel[0] * (1.0f - kFilteringFactor);
accel[1] = acceleration.y * kFilteringFactor + accel[1] * (1.0f - kFilteringFactor);
accel[2] = acceleration.z * kFilteringFactor + accel[2] * (1.0f - kFilteringFactor);
result.x = acceleration.x - accel[0];
result.y = acceleration.y - accel[1];
result.z = acceleration.z - accel[2];
Return result;

28. Accelerometer Noise – Notes
If it is too slow to adapt to sudden change in position, do more rapid changes when angle(accel, acceleration) is bigger
You can throw away single values that are way out of average.
|acc| does not have to be equal to |g| !
Kalaman filters – too complicated?

29. Other sensors Light sensor Proximity sensor Temperature sensor
Pressure sensor

30. Light sensor Sensor.TYPE_LIGHT
values[0] = ambient light level in SI lux units
SensorManager’s constants
LIGHT_CLOUDY: 100
LIGHT_FULLMOON: 0.25
LIGHT_NO_MOON: 0.001
LIGHT_OVERCAST: (cloudy)
LIGHT_SHADE:
LIGHT_SUNLIGHT:
LIGHT_SUNLIGHT_MAX:
LIGHT_SUNRISE: 400.0

31. Proximity sensor Sensor.TYPE_PROXIMITY
values[0]: Proximity sensor distance measured in centimeters (sometimes binary near-far)

32. Temperature sensor
Sensor.TYPE_TEMPERATURE
values[0] = temperature

33. Pressure sensor
Sensor.TYPE_PRESSURE
values[0] = pressure
no constants

34. Videos
Sensor Fusion:
15:22-20:27
Jobs video..