Camera metadata设计与应用

前言

Android的Camera Metadata是一种数据结构，用于表示图像特征的参数，例如常见的曝光、AE、AWB、flash等参数。在新的Camera API2 / HAL3架构中，这种结构被用来在app-hal之间IPC传输，取代了原先的SetParameter()/GetParameter()方法。

一，metadata设计剖析

1.1 数据结构

源码路径：

system/media/camera/include/system/camera_metadata.h
system/media/camera/include/system/camera_metadata_tags.h
system/media/camera/src/camera_metadata.c
system/media/camera/src/camera_metadata_tag_info.c

http://androidxref.com/9.0.0_r3/xref/system/media/camera/src/camera_metadata.c

Camera Metadata将参数以有序结构体的形式保存在一块连续的内存中。这个结构体的内存分布大致可以分为以下几个区域：

1. 区域一：存储camera_metadata_t结构体的定义，占用内存48Byte。该区域数据结构如下：

struct camera_metadata;
typedef struct camera_metadata camera_metadata_t;#define METADATA_ALIGNMENT ((size_t) 4)
struct camera_metadata {metadata_size_t          size;uint32_t                 version;uint32_t                 flags;metadata_size_t          entry_count;metadata_size_t          entry_capacity;metadata_uptrdiff_t      entries_start; // Offset from camera_metadatametadata_size_t          data_count;metadata_size_t          data_capacity;metadata_uptrdiff_t      data_start; // Offset from camera_metadatauint32_t                 padding;    // padding to 8 bytes boundarymetadata_vendor_id_t     vendor_id;
};
_Static_assert(sizeof(camera_metadata_t) == 48,"Size of camera_metadata_t must be 48");

2. 区域二：保留区，供未来使用。

3. 区域三：序号为0~entry_count-1的camera_metadata_buffer_entry_t结构体对象，简称metadata_entry结构体对象，每个对象有自己的tag。该区域数据结构如下：

typedef struct camera_metadata_entry {size_t   index;uint32_t tag;uint8_t  type;size_t   count;union {uint8_t *u8;int32_t *i32;float   *f;int64_t *i64;double  *d;camera_metadata_rational_t *r;} data;
} camera_metadata_entry_t;

4. 区域四：剩余未使用的Tag结构体的内存保留，该区域大小为(entry_capacity – entry_count)个TAG，也就是(entry_capacity – entry_count)个camera_metadata_buffer_entry_t的空间。

5. 区域五：所有Tag对应的具体metadata数据。仅camera_metadata.c函数使用，外部函数不能使用该数据结构。该区域数据结构如下：

#define DATA_ALIGNMENT ((size_t) 8)
typedef union camera_metadata_data {uint8_t u8;int32_t i32;float   f;int64_t i64;double  d;camera_metadata_rational_t r;
} camera_metadata_data_t;_Static_assert(sizeof(camera_metadata_data_t) == 8,"Size of camera_metadata_data_t must be 8");

6. 区域六：剩余未使用的Tag占用的内存。

从metadata源码定义来看这个结构体包括Head区、Entry区和Data区，典型TLV结构，描述每个Tag对应模型是key-value对，键值映射关系。数据模型概述如下：

metadata增删改查操作都以camera_metadata_t 的结构体数据为操作对象，相当于handle。

1.2 增删改查

camera_metadata_t* allocate_camera_metadata(size_t entry_capacity, size_t data_capacity)    //metadata 内存分配
void free_camera_metadata(camera_metadata_t *metadata) //释放 metadata 对应内存
int add_camera_metadata_entry(camera_metadata_t *dst, uint32_t tag, const void *data, size_t data_count)    //在 metadata 中增加 tag 和 value
int delete_camera_metadata_entry(camera_metadata_t *dst, size_t index)    //删除 tag
int update_camera_metadata_entry(camera_metadata_t *dst, size_t index, const void *data, size_t data_count, camera_metadata_entry_t *updated_entry)    //更新 tag 的 value 值
int find_camera_metadata_entry(camera_metadata_t *src, uint32_t tag, camera_metadata_entry_t *entry)    //根据 tag 查找 value

1.allocate_camera_metadata创建metadata

camera_metadata_t *allocate_camera_metadata(size_t entry_capacity,size_t data_capacity) {size_t memory_needed = calculate_camera_metadata_size(entry_capacity,data_capacity);void *buffer = calloc(1, memory_needed);camera_metadata_t *metadata = place_camera_metadata(buffer, memory_needed, entry_capacity, data_capacity);if (!metadata) {/* This should not happen when memory_needed is the same* calculated in this function and in place_camera_metadata.*/free(buffer);}return metadata;
}size_t calculate_camera_metadata_size(size_t entry_count,size_t data_count) {size_t memory_needed = sizeof(camera_metadata_t);// Start entry list at aligned boundarymemory_needed = ALIGN_TO(memory_needed, ENTRY_ALIGNMENT);memory_needed += sizeof(camera_metadata_buffer_entry_t[entry_count]);// Start buffer list at aligned boundarymemory_needed = ALIGN_TO(memory_needed, DATA_ALIGNMENT);memory_needed += sizeof(uint8_t[data_count]);// Make sure camera metadata can be stacked in continuous memorymemory_needed = ALIGN_TO(memory_needed, METADATA_PACKET_ALIGNMENT);return memory_needed;
}

sizeof(uint8_t[data_count])表示sizeof(uint8_t)*data_count，即数组长度为data_count内存size。

二，provider进程IPC传输hal层metadata

Android metadata通过Binder机制实现app与hal传输，而不是共享内存。

通过源码分析，hal是如何做到可以上报metadata到app。

2.1 metada回调接口注册

这个回调是binder通信机制。

1，获取client注册的ICameraDeviceCallback

ICameraDeviceCallback有processCaptureResult上报metadata接口，根据hidl机制，实体是cameraserver client的。

CameraDeviceSession.cpp - OpenGrok cross reference for /hardware/interfaces/camera/device/3.2/default/CameraDeviceSession.cpp (aospxref.com)

CameraDeviceSession::CameraDeviceSession(camera3_device_t* device,const camera_metadata_t* deviceInfo,const sp<ICameraDeviceCallback>& callback) :camera3_callback_ops({&sProcessCaptureResult, &sNotify, nullptr, nullptr}),mDevice(device),mDeviceVersion(device->common.version),mFreeBufEarly(shouldFreeBufEarly()),mIsAELockAvailable(false),mDerivePostRawSensKey(false),mNumPartialResults(1),mResultBatcher(callback) {mDeviceInfo = deviceInfo;camera_metadata_entry partialResultsCount =mDeviceInfo.find(ANDROID_REQUEST_PARTIAL_RESULT_COUNT);if (partialResultsCount.count > 0) {mNumPartialResults = partialResultsCount.data.i32[0];}mResultBatcher.setNumPartialResults(mNumPartialResults);camera_metadata_entry aeLockAvailableEntry = mDeviceInfo.find(ANDROID_CONTROL_AE_LOCK_AVAILABLE);if (aeLockAvailableEntry.count > 0) {mIsAELockAvailable = (aeLockAvailableEntry.data.u8[0] ==ANDROID_CONTROL_AE_LOCK_AVAILABLE_TRUE);}// Determine whether we need to derive sensitivity boost values for older devices.// If post-RAW sensitivity boost range is listed, so should post-raw sensitivity control// be listed (as the default value 100)if (mDeviceInfo.exists(ANDROID_CONTROL_POST_RAW_SENSITIVITY_BOOST_RANGE)) {mDerivePostRawSensKey = true;}mInitFail = initialize();
}

2，processCaptureResult上报metadata数据

void CameraDeviceSession::sProcessCaptureResult(const camera3_callback_ops *cb,const camera3_capture_result *hal_result) {CameraDeviceSession *d =const_cast<CameraDeviceSession*>(static_cast<const CameraDeviceSession*>(cb));CaptureResult result = {};camera3_capture_result shadowResult;bool handlePhysCam = (d->mDeviceVersion >= CAMERA_DEVICE_API_VERSION_3_5);std::vector<::android::hardware::camera::common::V1_0::helper::CameraMetadata> compactMds;std::vector<const camera_metadata_t*> physCamMdArray;sShrinkCaptureResult(&shadowResult, hal_result, &compactMds, &physCamMdArray, handlePhysCam);status_t ret = d->constructCaptureResult(result, &shadowResult);if (ret == OK) {d->mResultBatcher.processCaptureResult(result);}
}

d->mResultBatcher.processCaptureResult(result);为回调接口，通过binder向cameraserver client发送数据，包含metadata数据。

3，vendor camera.XXX.so库提供给provider调用的camera3接口

 int (*initialize)(const struct camera3_device *, const camera3_callback_ops_t *callback_ops);

camera3.h - OpenGrok cross reference for /hardware/libhardware/include/hardware/camera3.h (aospxref.com)

camera3_callback_ops_t是camera3_callback_ops typedef起的别名。

initialize主要是设置callback回调接口，包含processCaptureResult，为CameraDeviceSession::sProcessCaptureResult。

4，CameraDeviceSession调用内部initialize方法

到这里完成回调接口在vendor camera.XXX.so注册。

2.2 dlopen camera.XXX.so

1，camera.XXX.so hal入口

open中完成camera3_device_ops_t获取。camera_common.h - OpenGrok cross reference for /hardware/libhardware/include/hardware/camera_common.h (aospxref.com)

/*** The id of this module
*/
#define CAMERA_HARDWARE_MODULE_ID "camera"

2，camera hal 接口camera3_device_t

camera3.h - OpenGrok cross reference for /hardware/libhardware/include/hardware/camera3.h (aospxref.com)

typedef struct camera3_device {/*** common.version must equal CAMERA_DEVICE_API_VERSION_3_0 to identify this* device as implementing version 3.0 of the camera device HAL.** Performance requirements:** Camera open (common.module->common.methods->open) should return in 200ms, and must return* in 500ms.* Camera close (common.close) should return in 200ms, and must return in 500ms.**/hw_device_t common;camera3_device_ops_t *ops;void *priv;
} camera3_device_t;

三，provider进程操作hal层metadata实例

hal层通常不直接操作camera_metadata_t结构，一般使用类封装结构CameraMetadata。

/frameworks/av/camera/include/camera/CameraMetadata.h

/frameworks/av/camera/CameraMetadata.cpp

1，Static（characteristic）注册

int initStaticMetadata(int32_t cameraId, camera_metadata_t **static_metadata) {
...CameraMetadata staticInfo;/*FLASH_INFO*/staticInfo.update(ANDROID_FLASH_INFO_AVAILABLE,&(flash_InfoInfo.available), 1);*static_metadata = staticInfo.release();
...
}

2，metdata读操作

int updataWorkMetadata(const CameraMetadata &frame_settings, uint32_t frame_number) {
...if (frame_settings.exists(ANDROID_CONTROL_AE_ANTIBANDING_MODE)) {uint8_t valueU8 =frame_settings.find(ANDROID_CONTROL_AE_ANTIBANDING_MODE).data.u8[0];}
...
}

3，metdata写操作

camera_metadata_t *translateLocalToFwMetadata(uint32_t frame_number) {CameraMetadata camMetadata;camera_metadata_t *resultMetadata;
...uint8_t enable_zsl = false;camMetadata.update(ANDROID_CONTROL_ENABLE_ZSL,&(enable_zsl), 1);
...resultMetadata = camMetadata.release();return resultMetadata;
}

四，小结

从Android hidl和hal设计来看，总体都是为了解耦，metadata同样如此，解耦同时增加兼容性。