Android Camera2 Api By Tony Constantinides Date July 2nd 2015

Introduction to Mobile Multimedia on Android Why you should care? A brief history of multimedia on Android Brief design of the Camera1 api The detailed design of the Camera2 api Key differences DEMO Q & A

Wy you should care? Cameras are used as key differentiators for selling Android devices Excellent support for Camera support can be important to increase the perceived value of your apps End users love using mobile cameras, so that love can spill over to be love of your android app Cameras are just plain cool and fun

Multimedia Design - Android

Camera 1 using HAL 1.0

Camera 1 API was simple

Camera2 APIs android.hardware.camera2 CameraManager, CameraCharacteristics, CameraMetadata, CameraDevice, CameraCaptureSession, CaptureRequest, TotalCaptureResult, DngCreator

Overview of Camera2 and moving parts.

Camera 2 using HAL 3.0

Google’s new approach

Camera2 API Core Operation Model

Android Camera HAL v3 Based on capture requests, each of which translate to a captured image in one or more buffers The full capture configuration is part of the request User queues capture requests to the device and receives completed requests back later on Note all phone support HAL v3

The heart of the system

ISP Configuration Hardware ISPs have low level image processing configuration —Lens shading compensation tables —Black level correction —Linearization —Color space conversion (RGB to YUV) —Statistics configuration (e.g. windows of interest --- location and size)

3A Library works with Sensor and ISP Input – Exposure and white balance statistics – Histogram – Frame metadata Output – Sensor exposure time and gain – ISP parameters

3A Control Loop or Camera Love triangle frame metadata, image statistics 3A Library exposure time, gain ISP parameters sensor ISP images, frame metadata

Camera2 Hardware levels- Full • 30fps operation at sensor resolution preferred, 20fps required for at least uncompressed YUV output: useful for burst capture and computational photography • Per frame control • Manual sensor control: Like exposure time, gain • Manual post-processing control: color rendition, contrast/gamma curve, noise reduction • Arbitrary cropping, in size and coordinates • At least 3 processed non-stalling output streams formats: like YUV 4:2:0 used for preview. • The ability to list and auto-configure output formats according to a list of supported streams

Camera2 Hardware level-Legacy = Camera 1 API No support for per frame control No Manual Sensor control No Manual post-processing No Arbitrary cropping regions Relaxed performance constraints No Raw Capture No 4K

Camera2 Hardware Level - Limited Can turn Auto Exposure off Support for turning AF off Support for 30 frames a second recording Support for control over the AE Compensation range May not support Raw capture Minimum Focus distance lens support Noise reduction available

So Limited is not so bad? You must query for the following using the CameraCharacteristics class BACKWARD_COMPATIBLE MANUAL_SENSOR - 3A algorithms MANUAL_POST_PROCESSING RAW READ_SENSOR_SETTINGS BURST_CAPTURE

Start with CameraManager Provides an system service manager fro detecting, characterizing, and connecting to CameraDevice You create an instance by CameraManager m = (CameraManager)activity.getSystemService(Context.CAMERA_SERVICE); Call “String[] getCameraIdList( )”returns a list of currently connected camera devices by identifier Next call “CameraCharacteristics getCameraCharacteristics(String cameraId)” to return information about your device.

Now Open the Camera and set the status callback mCameraManager,openCamera(“0”,new CameraDevice.StateCallback { public void onOpened(CameraDevice cameraDevice) { } public void onClosed(CameraDevice camera) { } public void onDisconnected(CameraDevice cameraDevice) {} public void onError(CameraDevice cameraDevice, int error) {} },null};

Different types of surfaces can be used as output TextureView - surface for taking a picture MediaRecorder - recording video and audio MediaCodec - feeding output to GPU H264 Decoder ImageReader - convert Raw to Adobe DLG files RenderScriptAllocation - convert from YUV to NV image or i420 video formats

Creating surfaces, adjust size to fit the video or image CameraCharacteristics ch = mCamerManager.getCameraCharacteristics(“0”); StreamConfigurationMap map = ch.get(CameraCharacteristics.SCALER_STREAM_CONFIGURA TION_MAP); mVideoSize = chooseVideoSize(map.getOutputSize(MediaRecorder.class)); mPreviewSize = chooseOptimalSize (map.getOutputSizes(SurfaceTexture.class), width, height, mVideoSize);

Capture Request options TEMPLATE_PREVIEW - A request suitable for a camera preview window. A High Frame rate is given priority. TEMPLATE_MANUAL - A basic template for direct application control of capture parameters TEMPLATE_RECORD - For Video Recording(Stable FR) TEMPLATE_STILL_CAPTURE - For Still Image capture TEMPLATE_VIDEO_SNAPSHOT - Image capture while recording video TEMPLATE_ZERO_SHUTTER_LAG - zero shutter lag still capture

Create a CaptureRequest mPreviewBuilder = mCameraDevice. createCaptureRequest(CameraDevice.TEMPLA TE_RECORD); // used to record video Surface recorderSurface = mMediaRecorder.getSurface(); surfaces.add(recorderSurface); mPreviewBuilder.addTarget(recorderSurface);

Create a CaptureSession mCameraDevice.createCaptureSession(surfaces, new CameraCaptureSession.StateCallback) { public void onConfigured(CameraCaptureSession cs) {} public void onConfigureFailed(CameraCaptureSession cs) {} public void onReady(CameraCaptureSession s) {} public void onActive(CameraCaptureSession s) {} public void onClosed(CameraCaptureSession s) {} }, mBackGroundHandler);

Before submitting CaptureRequests adjust camera settings Builder builder = mCameraDevice.createCaptureRequest(TEMPLATE_RECORD); // turn on auto focus builder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.AF_MODE_AUTO); builder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_SCENE_MODE_HIGH_SPEED_VIDEO); // 60- fps for Nexus 5 builder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_SCENE_MODE_HDR); builder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_SCENE_MODE_BEACH); // video on the beach

Taking a picture Use mCaptureSession.capture(request, new CaptureListener) { public void onCaptureStarted(CameraCaptureSession session, CaptureRequest request, long timestamp) {} public void onCaptureComplete(CameraCaptureSession session, CaptureRequest request, TotalCaptureResult result) {} } }, null);

Demo time Still, Stillwith raw, and video

Q&A Most camera are still reported “Limited” Hardware support Be prepared to query to see if you can adjust Sensor settings, auto white balance, Edge control, etc The Camera2 API is very powerful but can be complex at first look The device manufactures need to be ousted to support the full camera2 api as they like their prioparitay api The API is unbelievable powerful but will require the developer to query to make sure the camera supports the features it wants to use Some camera (Nexus 6( have white balance and flash support bugs