ais_server 学习笔记

一前序
二、ais init
- 1、时序图如下
- 2. 初始化一共分为以下几个重要步骤：
- - 2.1.1、在ais_server中启动main函数，然后创建AisEngine，接着初始化AisEngine
  - 2.1.2、解析/var/camera_config.xml 文件，获取相关配置参数。这个camera_config.xml 后面会细聊
  - 2.1.3、启动EventHandler线程，会启动四个EventHandler，来承接外面调回来的通知，比如FRAME_DONE、FATAL_ERROR等等, 至于为什么是四个EventHandler，我猜测可能是一个CSIPHY 对应一个 Thread handler吧。
  - 2.1.4、"加载" sensor so、CSIPHY 、IFE
  - 2.1.5、接着会根据前面解析的camera_config.xml 里面的配置来给input、ife、csiphy 里面的参数进行配置，在csiphy里面会进行mipi协议的一些配置, 配置好mipiconfig 后就会去 start mipi
  - 2.1.6、启动平台图像处理等功能
三、amera_client -> qcarcam_initialize
四、camera_client -> qcarcam_open
五、camera_client -> qcarcam_s_param
六、camera_client -> qcarcam_start
七、camera_client -> qcarcam_get_frame

一前序

ais server 的主要功能，简单理解就是高通平台用于使能相机模块，处理camera 数据，并在camera client连接的时候将camera数据顺利给到client。

我们还是从ais的init，然后client调用接口，init、open、s_param、start、get_frame、release_frame、stop、close、deinit聊起, 每个调用都会有对应的时序图。

二、ais init

1、时序图如下

在这里插入图片描述

2. 初始化一共分为以下几个重要步骤：

2.1.1、在ais_server中启动main函数，然后创建AisEngine，接着初始化AisEngine

2.1.2、解析/var/camera_config.xml 文件，获取相关配置参数。这个camera_config.xml 后面会细聊

<?xml version="1.0" encoding="UTF-8"?><!--Copyright (c) 2020 Qualcomm Technologies, Inc.All Rights Reserved.Confidential and Proprietary - Qualcomm Technologies, Inc.--><!-- THIS IS A SAMPLE CAMERA CONFIG XML TO ILLUSTRATE HOW IT CAN BE USED --><!--SA8155 SAMPLE
=========================================================================== */
/*              CSI     lanes   IFEs   I2C Port    Sensors* max96716    2        4      2    /dev/cci1    2 x 96717 YUV SENSORS
*/
--><CameraConfig version="0x403"><board name="SA8155_ADP" boardType="CAMERA_HW_BOARD_ADPAIR_V2_PL195" multiSocEnable="0"><i2cDevs><i2cDev name="cci0"><properties i2ctype = "CAMERA_I2C_TYPE_CCI" device_id = "0" port_id = "0"/><sda_pin gpio = "17" function = "1"/><scl_pin gpio = "18" function = "1"/></i2cDev><i2cDev name="cci1"><properties i2ctype = "CAMERA_I2C_TYPE_CCI" device_id = "0" port_id = "1"/><sda_pin gpio = "19" function = "1"/><scl_pin gpio = "20" function = "1"/></i2cDev><i2cDev name="cci2"><properties i2ctype = "CAMERA_I2C_TYPE_CCI" device_id = "1" port_id = "0"/><sda_pin gpio = "31" function = "1"/><scl_pin gpio = "32" function = "1"/></i2cDev></i2cDevs><inputDevs><inputDev devId = "0"><properties><!-- 2 AR0231 Bayer Sensor --><config subdevId = "0" type="1" numSensors="1" pmasterSocId="0" socMap="1" opMode="0" /><sensor link="0" type="1" /></properties><driverInfo devCategory = "CAMERA_DEVICE_CATEGORY_SENSOR" libName="ais_nio_max96716f" openFcn="CameraSensorDevice_Open_max96716"/><i2cPort soc_id = "0" i2ctype = "CAMERA_I2C_TYPE_CCI" device_id = "0" port_id = "1" /><csiInfo csiId = "0" numLanes = "2" laneAssign = "0x10" numIfeMap="1" ifeMap="0x0" /><gpio id = "CAMERA_GPIO_RESET" num = "22" config = "0x31"/><intr id = "CAMERA_GPIO_INTR" pin_id = "106" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_FALLING" gpio_cfg = "0x30"/></inputDev><inputDev devId = "1"><properties><!-- 2 AR0231 Bayer Sensor --><config subdevId = "1" type="1" numSensors="4" pmasterSocId="0" socMap="1" opMode="0" /><sensor link="0" type="1" /><sensor link="1" type="1" /><sensor link="2" type="1" /><sensor link="3" type="1" /></properties><driverInfo devCategory = "CAMERA_DEVICE_CATEGORY_SENSOR" libName="ais_nio_max96722" openFcn="CameraSensorDevice_Open_max96722"/><i2cPort soc_id = "0" i2ctype = "CAMERA_I2C_TYPE_CCI" device_id = "1" port_id = "0" /><csiInfo csiId = "3" numLanes = "4" laneAssign = "0x3210" ifeMap="0x32" numIfeMap="2"/><gpio id = "CAMERA_GPIO_RESET" num = "25" config = "0x31"/><intr id = "CAMERA_GPIO_INTR" pin_id = "13" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_FALLING" gpio_cfg = "0x30"/><intr id = "CAMERA_GPIO_LOCK" pin_id = "15" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_FALLING" gpio_cfg = "0x30"/><intr id = "CAMERA_GPIO_PASS" pin_id = "90" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_EDGE" gpio_cfg = "0x10"/><intr id = "CAMERA_GPIO_CUSTOM1" pin_id = "149" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_FALLING" gpio_cfg = "0x30"/><intr id = "CAMERA_GPIO_CUSTOM2" pin_id = "150" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_FALLING" gpio_cfg = "0x30"/><intr id = "CAMERA_GPIO_CUSTOM_REG1" pin_id = "151" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_FALLING" gpio_cfg = "0x30"/><intr id = "CAMERA_GPIO_CUSTOM_REG2" pin_id = "152" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_FALLING" gpio_cfg = "0x30"/></inputDev><inputDev devId = "2"><properties><!-- 2 AR0231 Bayer Sensor --><config subdevId = "0" type="1" numSensors="3" pmasterSocId="0" socMap="1" opMode="0" /><sensor link="0" type="1" /><sensor link="1" type="1" /><sensor link="2" type="1" /></properties><driverInfo devCategory = "CAMERA_DEVICE_CATEGORY_SENSOR" libName="ais_nio_svc_fy" openFcn="CameraSensorDevice_Open_max96712"/><i2cPort soc_id = "0" i2ctype = "CAMERA_I2C_TYPE_CCI" device_id = "0" port_id = "0" /><csiInfo csiId = "2" numLanes = "4" laneAssign = "0x3210" ifeMap="0x1" numIfeMap="1"/></inputDev></inputDevs><engineSettings numBufMax="12" powerManagementPolicy="CAMERA_PM_POLICY_LPM_EVENT_ALL" latencyMeasurementMode="CAMERA_LM_MODE_DISABLE"recoveryRestartDelay="33"  recoveryTimeoutAfterUsrCtxtRestart="500" recoveryRetryDelay="300" recoveryMaxNumAttempts="1"><inputFatalError type="MATCH_INPUT_DEVICE" severity="0"/><csidFatalError type="MATCH_IFE_DEVICE" severity="0"/><ifeOutputError type="MATCH_IFE_DEVICE" severity="0"/></engineSettings></board><inputMapping><inputMap qcarcamId = "1" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "0" srcId = "0"/></inputMap><inputMap qcarcamId = "2" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "2" srcId = "2"/></inputMap><inputMap qcarcamId = "3" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "2" srcId = "0"/></inputMap><inputMap qcarcamId = "4" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "2" srcId = "1"/></inputMap><inputMap qcarcamId = "5" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "1" srcId = "0"/></inputMap><inputMap qcarcamId = "6" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "1" srcId = "1"/></inputMap><inputMap qcarcamId = "7" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "1" srcId = "2"/></inputMap><inputMap qcarcamId = "8" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "1" srcId = "3"/></inputMap></inputMapping>
</CameraConfig>

2.1.3、启动EventHandler线程，会启动四个EventHandler，来承接外面调回来的通知，比如FRAME_DONE、FATAL_ERROR等等, 至于为什么是四个EventHandler，我猜测可能是一个CSIPHY 对应一个 Thread handler吧。

在这里插入图片描述

2.1.4、“加载” sensor so、CSIPHY 、IFE

分别会调用RegisterDeviceFromLib()、RegisterDevice()将libais_nio_*.so 和 IFE 、CSIPHY open

static CameraDeviceManagerRegisteredType sStaticDevicesSA8155[] =
{
#ifndef AIS_WITH_HAL_CAMERA{{CAMERADEVICEID_IFE_0, CAMERA_DEVICE_CATEGORY_IFE}, ife_device_open},{{CAMERADEVICEID_IFELITE_1, CAMERA_DEVICE_CATEGORY_IFE}, ifelite_device_open},
#endif{{CAMERADEVICEID_IFE_1, CAMERA_DEVICE_CATEGORY_IFE}, ife_device_open},{{CAMERADEVICEID_IFELITE_0, CAMERA_DEVICE_CATEGORY_IFE}, ifelite_device_open},{{CAMERADEVICEID_CSIPHY_0, CAMERA_DEVICE_CATEGORY_CSIPHY}, csiphy_device_open},{{CAMERADEVICEID_CSIPHY_1, CAMERA_DEVICE_CATEGORY_CSIPHY}, csiphy_device_open},{{CAMERADEVICEID_CSIPHY_2, CAMERA_DEVICE_CATEGORY_CSIPHY}, csiphy_device_open},{{CAMERADEVICEID_CSIPHY_3, CAMERA_DEVICE_CATEGORY_CSIPHY}, csiphy_device_open},
};

这里继续调用到RegisterDevice，将
libais_nio_max96722.so -> CameraSensorDevice_Open_max96722
libais_nio_svc_fy.so -> CameraSensorDevice_Open_max96712
libais_nio_max96716f.so -> CameraSensorDevice_Open_max96716
保存在registeredDevices中

CameraSensorDevice_Open_max96722 -> max96722_sensor_open_lib()
CameraSensorDevice_Open_max96712 -> max96712_sensor_open_lib()
CameraSensorDevice_Open_max96716 -> max96716_sensor_open_lib()

创建AisInputConfigurer、AisIFEConfigurer、AisCSIConfigurer，然后init他们三个，在init中分别调用DeviceOpen方法，最终会调用到CameraSensorDevice_Open_max967*、 ife_device_open、csiphy_device_open。
ife_device_open 和 ifelite_device_open 分别会被调用两次，每次都会创建一个VFE/CSID(IFE持有VFE和CSID) 对象，ife_device_open 和 ifelite_device_open的取别主要在于RDI的个数，前者会有四个RDI通道，后者会有三个RDI通道。
在这里插入图片描述
CSID和IFE分别在创建的时候对应一个线程，分别open四次就会有四个线程，这个四个线程用于监听中断和其他消息通知，后面会用到

2.1.5、接着会根据前面解析的camera_config.xml 里面的配置来给input、ife、csiphy 里面的参数进行配置，在csiphy里面会进行mipi协议的一些配置, 配置好mipiconfig 后就会去 start mipi

在这里插入图片描述

2.1.6、启动平台图像处理等功能

在这里插入图片描述当ais完全启动后，后续将从client端使用的角度来分析，ais_server对应的每一步处理

三、amera_client -> qcarcam_initialize

Camera client和ais_server是通过libais_client进行通信的，如果是client是在qnx 则通过socket，如果client是在

android侧，则通过hab（也是基于共享内存封装的类socket机制）
Camera client 开始使用的时候首先要调用接口qcarcam_initialize, 首先会调用到ais_client中，会为这个camera分配一个s_ais_client变量，然后再建立该camera client和ais_server(ais_client 是camera client和ais_server之间的桥梁)之间的心跳机制

这里注意sgs_ais_client数组最大值是64，意味着最多同时可以连接64个client，但是这里有个奇怪的点，就是我们可以open的camera个数其实和rdi有关系？

–> 所以答案是:
### 三级目录
每个camera client qcarcam_initialize之后都会有个心跳线程
在这里插入图片描述

四、camera_client -> qcarcam_open

camera_client 调用qcarcam_open到AisEngine 中这里并没有对IFE、INPUT、CSI做什么操作，只是为这个open的camera分配了一个属于，camera_client和ais_server 识别这个camera的唯一标识句柄qcarcam_hndl
在这里插入图片描述

五、camera_client -> qcarcam_s_param

这里client会为每个camera申请5个buffer（MIN_USR_NUMBER_OF_BUFFERS 3 <buffer < QCARCAM_MAX_NUM_BUFFERS 12)

在这里插入图片描述
将client传过来的buffer地址和ais_server中的buffer做内存映射

smmu_map_pt_v2 函数可能用于将物理地址（DA, Device Address）映射到虚拟地址空间，具体是通过系统的 SMMU（System Memory Management Unit，系统内存管理单元）进行的，这里就简单理解camera_client buffer和ais_server的buffer做了映射，然后ais_server的buffer又绑定了一块专门用于camera data传输的内存。
在这里插入图片描述

如上log分析，handle 就是camera_client buffer的地址，DA就是做完映射后，ais_server使用的buffer地址，我们打开了四个camera，每个camera申请了5个buffer。

根据环视的camera_config.xml 配置，当有camera_client连接ais_server的时候，会优先给该camera分配IFE2，然后当IFE2的四个RDI都占用后，再连接的时候就分配IFE3的RDI

在这里插入图片描述如下图，CSID2 对应的 IFE就可以理解为IFE2、CSID3 对应的IFE就可以理解为IFE3，他们IFE都是Lite模式四路数据传输线。

在这里插入图片描述
这个后面camera data传递的时候会深入讨论

六、camera_client -> qcarcam_start

在这里插入图片描述
如上时序图，camera_client的start操作，其实主要针对的就是ais_input_configurer, CISPHY已经在ais_server初始化的时候已经start过了，然后根据先后顺序会start IFE 中的VFE/CISD，然后会start camera sensor, 这里我们用的摄像头加串器是96716/解串器是96722。

在96722sensor中，max96722_sensor_open_lib的时候，

    memcpy(&pCtxt->sensor_lib, &sensor_lib_ptr, sizeof(pCtxt->sensor_lib));memcpy(&pCtxt->max96722_sensors, max96722_sensors_init_table, sizeof(pCtxt->max96722_sensors));

static max96722_sensor_info_t max96722_sensors_init_table[] =
{{.state = SENSOR_STATE_INVALID,.serializer_addr = MSM_DES_0_ADDR_CAM_SER_0,.sensor_addr = MSM_DES_0_ADDR_CAM_SNSR_0,},{.state = SENSOR_STATE_INVALID,.serializer_addr = MSM_DES_0_ADDR_CAM_SER_1,.sensor_addr = MSM_DES_0_ADDR_CAM_SNSR_1,},{.state = SENSOR_STATE_INVALID,.serializer_addr = MSM_DES_0_ADDR_CAM_SER_2,.sensor_addr = MSM_DES_0_ADDR_CAM_SNSR_2,},{.state = SENSOR_STATE_INVALID,.serializer_addr = MSM_DES_0_ADDR_CAM_SER_3,.sensor_addr = MSM_DES_0_ADDR_CAM_SNSR_3,},
};

static sensor_lib_t sensor_lib_ptr =
{.sensor_slave_info ={.sensor_name = SENSOR_MODEL,.slave_addr = MSM_DES_0_SLAVEADDR,.i2c_freq_mode = SENSOR_I2C_MODE_CUSTOM,.addr_type = CAMERA_I2C_WORD_ADDR,.data_type = CAMERA_I2C_BYTE_DATA,.sensor_id_info ={.sensor_id_reg_addr = 0x00,.sensor_id = MSM_DES_0_SLAVEADDR,.sensor_id_mask = 0xff00,},.power_setting_array ={.power_up_setting_a ={{.seq_type = CAMERA_POW_SEQ_VREG,.seq_val = CAMERA_VDIG,.config_val = 0,.delay = 1,},{.seq_type = CAMERA_POW_SEQ_GPIO,.seq_val = CAMERA_GPIO_RESET,.config_val = GPIO_OUT_LOW,.delay = 10,},{.seq_type = CAMERA_POW_SEQ_GPIO,.seq_val = CAMERA_GPIO_RESET,.config_val = GPIO_OUT_HIGH,.delay = 100,},},.size_up = 3,.power_down_setting_a ={{.seq_type = CAMERA_POW_SEQ_GPIO,.seq_val = CAMERA_GPIO_RESET,.config_val = GPIO_OUT_LOW,.delay = 0,},},.size_down = 1,},.is_init_params_valid = 1,},.csi_params ={{.lane_cnt = 4, /*4 lane mode is selected */.settle_cnt = 0xE,.lane_mask = 0x1F,.combo_mode = 0,.is_csi_3phase = 0,}},.sensor_close_lib = max96722_sensor_close_lib,.sensor_capability = 0,/*custom functions that were defined in the driver */.sensor_custom_func ={.sensor_set_platform_func_table = &max96722_sensor_set_platform_func_table,.sensor_power_suspend = &max96722_sensor_power_suspend,.sensor_power_resume = &max96722_sensor_power_resume,.sensor_detect_device = &max96722_sensor_detect_device,.sensor_detect_device_channels = &max96722_sensor_detect_device_channels,.sensor_init_setting = &max96722_sensor_init_setting,.sensor_set_channel_mode = &max96722_sensor_set_channel_mode,.sensor_start_stream = &max96722_sensor_start_stream,.sensor_stop_stream = &max96722_sensor_stop_stream,.sensor_s_param = &max96722_des_sensor_s_param,.sensor_g_param = &max96722_des_sensor_g_param,//.sensor_power_on = &max96722_sensor_power_on,},.use_sensor_custom_func = TRUE,
};

如上代码所示，在open的时候就将sensor_lib_ptr和max96722_sensors_init_table copy给了sensor_lib和max96722_sensors，后面对于camera寄存器的读写都操作sensor_lib_ptr，对于加/解串操作max96722_sensors，
max96722_sensor_start_stream() -> ser96717_start_link() -> i2c_slave_write_array(max96722_sensors->serializer_addr, cam_ser_reg_setting), 这里96722/96716对应的解串器都是96717，这里做了统一化处理，包括来自J5的摄像头也是定义了一个96717的解串器名字
在这里插入图片描述

七、camera_client -> qcarcam_get_frame

当ais_server 准备好camera data后就会通知camera_client去使用camera data，大致时序如下：
在这里插入图片描述

如上时序图，VFE中的camera data流转过程：
1、在VEF模块初始化的时候，会注册函数ifeIST()到CameraPlatformISTInit中，在CameraPlatformISTInit中会启动一个线程CameraIST，会专门用于监听中断, 然后回调ifeIST().
2、ifeIST() -> ProcessIST() -> CameraSetSignal(m_ifeIstProcThreadContext.evIfeIstNotification) -> ISTWorkerThread(), 回调到 ifeIST() 之后会间接的唤醒等待的函数ISTWorkerThread ->
CameraWaitOnSignal(pIfeCtx->m_ifeIstProcThreadContext.evIfeIstNotification), 然后会紧接着调用
ProcessISTQueueItem(), 将填充好sQueueItem的根据中断类型执行相应的操作：
ProcessResetIRQ(): 触发重启VFE中断
ProcessSofIRQ(): camera data从硬件缓冲区到内存，将client申请的buffer映射出来的软件缓冲区提交到硬件缓冲区，去填充camera data
ProcessWrBusIRQ(): 软件缓冲区数据填满后就会触发中断，通知去消费数据
ProcessOverflowIRQ(): 将IFE、MIPI相关的error, waring 通知到camera client端。