1. CIS 470 Mobile App Development
Lecture 19
Wenbing Zhao
Department of Electrical Engineering and Computer Science
Cleveland State University
11/14/2018
CIS 470: Mobile App Development

2. Google Mobile Vision API
Face tracking
11/14/2018
CIS 470: Mobile App Development

3. CIS 470: Mobile App Development
Face Tracking
<?xml version="1.0" encoding="utf-8"?> <manifest xmlns:android=" package="com.wenbing.facetracker"> <uses-feature android:name="android.hardware.camera" /> <uses-permission android:name="android.permission.CAMERA" /> <application android:allowBackup="true" android:hardwareAccelerated="true" android:label="FaceTracker"> <meta-data android:name="com.google.android.gms.version" <meta-data android:name="com.google.android.gms.vision.DEPENDENCIES" android:value="face" /> <activity android:name=".MainActivity" android:label="Face Tracker" android:screenOrientation="fullSensor"> <intent-filter> <action android:name="android.intent.action.MAIN" /> <category android:name="android.intent.category.LAUNCHER" /> </intent-filter> </activity> </application> </manifest>
Create a new app and name it FaceTracker
Modify manifest
11/14/2018
CIS 470: Mobile App Development

4. CIS 470: Mobile App Development
Face Tracking
Modify build.gradle (Module: app)
dependencies { compile fileTree(dir: 'libs', include: ['*.jar']) androidTestCompile('com.android.support.test.espresso:espresso-core:2.2.2', { exclude group: 'com.android.support', module: 'support-annotations' }) compile 'com.android.support:appcompat-v7:26.+' compile 'com.android.support:design:26.+' compile 'com.google.android.gms:play-services-vision:9.4.0+' testCompile 'junit:junit:4.12' }
11/14/2018
CIS 470: Mobile App Development

5. CIS 470: Mobile App Development
Face Tracking
<?xml version="1.0" encoding="utf-8"?> <LinearLayout xmlns:android=" android:orientation="vertical" android:layout_width="match_parent" android:layout_height="match_parent" android:keepScreenOn="true"> <com.wenbing.facetracker.CameraSourcePreview android:layout_width="match_parent" android:layout_height="match_parent"> <com.wenbing.facetracker.GraphicOverlay android:layout_width="match_parent" android:layout_height="match_parent" /> </com.wenbing.facetracker.CameraSourcePreview> </LinearLayout>
Add three Java classes: CameraSourcePreview, FaceGraphic, GraphicOverlay. Populate all four classes with the java files posted on the web
Modify activity_main.xml layout:
11/14/2018
CIS 470: Mobile App Development

6. CIS 470: Mobile App Development
Face Tracking
<?xml version="1.0" encoding="utf-8"?> <LinearLayout xmlns:android=" android:orientation=”horizontal" android:layout_width="match_parent" android:layout_height="match_parent" android:keepScreenOn="true"> <com.wenbing.facetracker.CameraSourcePreview android:layout_width="match_parent" android:layout_height="match_parent"> <com.wenbing.facetracker.GraphicOverlay android:layout_width="match_parent" android:layout_height="match_parent" /> </com.wenbing.facetracker.CameraSourcePreview> </LinearLayout>
Add one more layout with the same name but change the default directory name to: layout-land
11/14/2018
CIS 470: Mobile App Development

7. CIS 470: Mobile App Development
Face Tracking
Add one more layout with the same name but change the default directory name to: layout-land
11/14/2018
CIS 470: Mobile App Development

8. CIS 470: Mobile App Development
Face Tracking
Change values/strings.xml
<resources> <string name="app_name">FaceTracker</string> <string name="ok">OK</string> <string name="permission_camera_rationale">Access to the camera is needed for detection</string> <string name="no_camera_permission">This application cannot run because it does not have the camera permission. The application will now exit.</string> <string name="low_storage_error">Face detector dependencies cannot be downloaded due to low device storage</string> </resources>
11/14/2018
CIS 470: Mobile App Development

9. CIS 470: Mobile App Development
Face Tracking
11/14/2018
CIS 470: Mobile App Development

10. Google Mobile Vision Face Tracking
Face detection is the process of automatically locating human faces in visual media (digital images or video)
A face that is detected is reported at a position with an associated size and orientation
Once a face is detected, it can be searched for landmarks (points of interest within a face) such as the eyes and nose
Classification is determining whether a certain facial characteristic is present. For example, a face can be classified with regards to whether its eyes are open or closed. Another example is whether the face is smiling or not
11/14/2018
CIS 470: Mobile App Development

11. CIS 470: Mobile App Development
Face Tracking
Pose angle examples where y==Euler Y, r==Euler Z
11/14/2018
CIS 470: Mobile App Development

12. CIS 470: Mobile App Development
Face Tracking
Landmarks: The left eye, right eye, and nose base
Landmark detection is not done by default, since it takes additional time to run. You can optionally specify that landmark detection should be done:
Euler Y angle
detectable landmarks
< -36 degrees
left eye, left mouth, left ear, nose base, left cheek
-36 degrees to -12 degrees
left mouth, nose base, bottom mouth, right eye, left eye, left cheek, left ear tip
-12 degrees to 12 degrees
right eye, left eye, nose base, left cheek, right cheek, left mouth, right mouth, bottom mouth
12 degrees to 36 degrees
right mouth, nose base, bottom mouth, left eye, right eye, right cheek, right ear tip
> 36 degrees
right eye, right mouth, right ear, nose base, right cheek
11/14/2018
CIS 470: Mobile App Development

13. Face Tracking: Classification
The Android Face API currently supports two classifications: eyes open and smiling
Classification is expressed as a certainty value, indicating the confidence that the facial characteristic is present. For example, a value of 0.7 or more for the smiling classification indicates that it is likely that a person is smiling
“eyes open” and “smiling” classification only works for frontal faces, that is, faces with a small Euler Y angle (at most about +/- 18 degrees)
11/14/2018
CIS 470: Mobile App Development

14. CIS 470: Mobile App Development
Face Tracking APIs
Creating the face detector:
Detecting faces with facial landmarks
Classification:
FaceDetector detector = new FaceDetector.Builder(context) setClassificationType(FaceDetector.ALL_CLASSIFICATIONS).build();
for (Landmark landmark : face.getLandmarks()) {         int cx = (int) (landmark.getPosition().x * scale);         int cy = (int) (landmark.getPosition().y * scale);         canvas.drawCircle(cx, cy, 10, paint);     }
face.getIsSmilingProbability()
face.getIsRightEyeOpenProbability()
face.getIsLeftEyeOpenProbability()
11/14/2018
CIS 470: Mobile App Development

15. CIS 470: Mobile App Development
Exercise
Draw all landmarks on the detected face(s).
11/14/2018
CIS 470: Mobile App Development