1. Android Camera2 Api
By Tony Constantinides
Date July 2nd 2015

2. Introduction to Mobile Multimedia on Android
Why you should care?
A brief history of multimedia on Android
Brief design of the Camera1 api
The detailed design of the Camera2 api
Key differences
DEMO
Q & A

3. Wy you should care?
Cameras are used as key differentiators for selling Android devices
Excellent support for Camera support can be important to increase the perceived value of your apps
End users love using mobile cameras, so that love can spill over to be love of your android app
Cameras are just plain cool and fun

4. Multimedia Design - Android

5. Camera 1 using HAL 1.0

6. Camera 1 API was simple

7. Camera2 APIs android.hardware.camera2
CameraManager, CameraCharacteristics, CameraMetadata, CameraDevice, CameraCaptureSession, CaptureRequest, TotalCaptureResult, DngCreator

8. Overview of
Camera2
and
moving
parts.

9. Camera 2 using HAL 3.0

10. Google’s new approach

11. Camera2 API Core Operation Model

12. Android Camera HAL v3
Based on capture requests, each of which translate to a captured image in one or more buffers
The full capture configuration is part of the request
User queues capture requests to the device and receives completed requests back later on
Note all phone support HAL v3

13. The heart of the system

14. ISP Configuration
Hardware ISPs have low level image processing configuration
—Lens shading compensation tables
—Black level correction
—Linearization
—Color space conversion (RGB to YUV)
—Statistics configuration (e.g. windows of interest --- location and size)

15. 3A Library works with Sensor and ISP
Input
– Exposure and white balance statistics
– Histogram
– Frame metadata
Output
– Sensor exposure time and gain
– ISP parameters

16. 3A Control Loop or Camera Love triangle
frame metadata,
image statistics 3A
Library
exposure time,
gain
ISP
parameters
sensor
ISP
images,
frame metadata

17. Camera2 Hardware levels- Full
• 30fps operation at sensor resolution preferred, 20fps required for at least uncompressed YUV output: useful for burst capture and computational photography
• Per frame control
• Manual sensor control: Like exposure time, gain
• Manual post-processing control: color rendition, contrast/gamma curve, noise reduction
• Arbitrary cropping, in size and coordinates
• At least 3 processed non-stalling output streams formats: like YUV 4:2:0 used for preview.
• The ability to list and auto-configure output formats according to a list of supported streams

18. Camera2 Hardware level-Legacy = Camera 1 API
No support for per frame control
No Manual Sensor control
No Manual post-processing
No Arbitrary cropping regions
Relaxed performance constraints
No Raw Capture
No 4K

19. Camera2 Hardware Level - Limited
Can turn Auto Exposure off
Support for turning AF off
Support for 30 frames a second recording
Support for control over the AE Compensation range
May not support Raw capture
Minimum Focus distance lens support
Noise reduction available

20. So Limited is not so bad?
You must query for the following using the CameraCharacteristics class
BACKWARD_COMPATIBLE
MANUAL_SENSOR - 3A algorithms
MANUAL_POST_PROCESSING
RAW
READ_SENSOR_SETTINGS
BURST_CAPTURE

21. Start with CameraManager
Provides an system service manager fro detecting, characterizing, and connecting to CameraDevice
You create an instance by CameraManager m = (CameraManager)activity.getSystemService(Context.CAMERA_SERVICE);
Call “String[] getCameraIdList( )”returns a list of currently connected camera devices by identifier
Next call “CameraCharacteristics getCameraCharacteristics(String cameraId)” to return information about your device.

22. Now Open the Camera and set the status callback
mCameraManager,openCamera(“0”,new CameraDevice.StateCallback {
public void onOpened(CameraDevice cameraDevice) { }
public void onClosed(CameraDevice camera) { }
public void onDisconnected(CameraDevice cameraDevice) {}
public void onError(CameraDevice cameraDevice, int error) {}
},null};

23. Different types of surfaces can be used as output
TextureView - surface for taking a picture
MediaRecorder - recording video and audio
MediaCodec - feeding output to GPU H264 Decoder
ImageReader - convert Raw to Adobe DLG files
RenderScriptAllocation - convert from YUV to NV image or i420 video formats

24. Creating surfaces, adjust size to fit the video or image
CameraCharacteristics ch = mCamerManager.getCameraCharacteristics(“0”);
StreamConfigurationMap map = ch.get(CameraCharacteristics.SCALER_STREAM_CONFIGURA TION_MAP);
mVideoSize = chooseVideoSize(map.getOutputSize(MediaRecorder.class));
mPreviewSize = chooseOptimalSize (map.getOutputSizes(SurfaceTexture.class), width, height, mVideoSize);

25. Capture Request options
TEMPLATE_PREVIEW - A request suitable for a camera preview window. A High Frame rate is given priority.
TEMPLATE_MANUAL - A basic template for direct application control of capture parameters
TEMPLATE_RECORD - For Video Recording(Stable FR)
TEMPLATE_STILL_CAPTURE - For Still Image capture
TEMPLATE_VIDEO_SNAPSHOT - Image capture while recording video
TEMPLATE_ZERO_SHUTTER_LAG - zero shutter lag still capture

26. Create a CaptureRequest
mPreviewBuilder = mCameraDevice. createCaptureRequest(CameraDevice.TEMPLA TE_RECORD); // used to record video
Surface recorderSurface = mMediaRecorder.getSurface();
surfaces.add(recorderSurface);
mPreviewBuilder.addTarget(recorderSurface);

27. Create a CaptureSession
mCameraDevice.createCaptureSession(surfaces, new CameraCaptureSession.StateCallback) {
public void onConfigured(CameraCaptureSession cs) {}
public void onConfigureFailed(CameraCaptureSession cs) {}
public void onReady(CameraCaptureSession s) {}
public void onActive(CameraCaptureSession s) {}
public void onClosed(CameraCaptureSession s) {}
}, mBackGroundHandler);

28. Before submitting CaptureRequests adjust camera settings
Builder builder = mCameraDevice.createCaptureRequest(TEMPLATE_RECORD);
// turn on auto focus
builder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.AF_MODE_AUTO);
builder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_SCENE_MODE_HIGH_SPEED_VIDEO); // 60- fps for Nexus 5
builder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_SCENE_MODE_HDR);
builder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_SCENE_MODE_BEACH); // video on the beach

29. Taking a picture
Use mCaptureSession.capture(request, new CaptureListener) {
public void onCaptureStarted(CameraCaptureSession session, CaptureRequest request, long timestamp) {}
public void onCaptureComplete(CameraCaptureSession session, CaptureRequest request, TotalCaptureResult result) {} }
}, null);

30. Demo time
Still, Stillwith raw, and video

31. Q&A Most camera are still reported “Limited” Hardware support
Be prepared to query to see if you can adjust Sensor settings, auto white balance, Edge control, etc
The Camera2 API is very powerful but can be complex at first look
The device manufactures need to be ousted to support the full camera2 api as they like their prioparitay api
The API is unbelievable powerful but will require the developer to query to make sure the camera supports the features it wants to use
Some camera (Nexus 6( have white balance and flash support bugs