【Qcom Camera】DumpDebugInfo分析

DumpDebugInfo：

DumpDebugInfo主要包括Session::DumpDebugInfo、Pipeline::Dumpdebuginfo、Node::Dumpdebuginfo、DRQ::Dumpdebuginfo、Usecase::DumpDebugInfo

log：Hit SOF threshold of [xx] consecutive frames
      CamX: [ERROR][CORE ] camxpipeline.cpp:3248 CheckForRecovery() Hit SOF threshold of [17] consecutive frames with invalidrequestId; triggering watchdog recovery for pipeline RealTimeFeatureZSLPreviewRawYUV_0
      以上log表明kernel丢失连续的17帧，也意味着UMD 长时间没有向kernel发送sensor或 IFE request
ProcessCaptureRequest reset UMD
      CamX : [CONFIG][CORE ] camxsession.cpp:1689 ProcessCaptureRequest() Lets do a Reset:UMD
      当request queue已满时，Camx session需要等待至少一个live pending requests(待处理请求)完成。如果等待超时，就会出现以上log
      意味着所有live pending requests的result都没有完成。应该检查reslut是否卡在某个地方
Kernel requests recovery
       CamX : [ERROR][CSL ] camxcslhwinternal.cpp:1232 CSLHwInternalSendRequestManagerEvent() frame error: type 4, requestID 0, device hdl 0 04-10 00:12:39.463 11687 11689 E CamX : [ERROR][CORE ] camxpipeline.cpp:3435 CSLMessageHandler() request id 0, error type = 4, device handle = 0, resource index = 0
       首先check kernel问题
Flush timeout
      CamX: [ERROR][CORE ] camxsession.cpp:151 WaitTillFlushResultsAvailable() TimedWait for results timed out at 255 ms with error CamxResultETimeout! Pending results: 1249511 03-25 05:28:08:118452867 939 939 I CamX: [ DUMP][CORE ] camxsession.cpp:6108 DumpDebugInfo() + Stuck on Sequence Id: 201 Request Id: 87
      以上log表明flush 卡住，后面的session dump可能会有帮助

CamX : [ DUMP][CORE ] camxnode.cpp:8417 DumpDebugInfo() + Request: 1 Unsuccessful. metaComplete: 1, reqComplete: 0, numUnsigFences: 1, numUnprocFences: 1, status: SUBMIT , currTime 03:45:42:764 statusTransitionTime: 03:45:37:659
request:该node中未处理完成的request id
metaComplete:该request的meta是否处理完成。0表示没有处理完成，1表示处理完成
reqComplete: 该request的buffer是否处理完成。0表示没有处理完成，1表示处理完成
numUnsigFences: 该request中unsigned fences的个数，这个值随着 fence在node中被触发而减少。该值为0时，表示所有output都是可用的。
numUnprocFences：该request中的还没有处理的fence的个数
status：该request的状态

camxsession.cpp DumpDebugInfo
                   Stuck on Sequence Id: 0 Request Id: 1     //卡在了Request Id:
camxpipeline.cpp DumpDebugInfo
                   isSofdispatched: 0                        //sensor没有出帧（如果DRQ处于DEFERRED状态，也就是request尚未满足dependency，所以request还没有发到HW，需要再往下分析sensor node还有哪些dependency尚未满足）
                                                                              （如果DRQ处于SUBMIT状态，说明request已经提交给硬件，这时候就需要看sensor driver（kernel）的log，看是不是有硬件操作失败的相关log了）
                   numNodes: 23                              //request 要经过23个node处理
                   numNodesRequestIdDone: 22                 //request 已经在22个node处理完，还有一个没处理完
camxnode.cpp   DumpDebugInfo
                   status: SUBMIT    //request已经提交给硬件，说明这个request的相关依赖已经得到满足，并且执行了该node 的ExecuteProcessRequest，等待结果的返回
                   status: DEFERRED //说明当前node还缺少依赖的property或者buffer，相关request还没有发到HW
camxdeferredrequestqueue.cpp DumpDebugInfo()
                   Property[0] = 0x30000024 PropertyIDSensorProperties offset 18446744073709551615 contextType 0 on Pipeline = 0 is not published
                                      //在node上还没满足的是PropertyIDSensorProperties这个tag没有被publish出来。需要分析代码为什么这个property没有被published

---------------------------------------------

-----------------------------------
发现连续丢帧 --> check session dump确定 first sutck的request --> Check pipeline dump确定出问题的pipeline以及first sutck的request和它的pending node
--> Check node dump看sensor 和IFE 分别stuck在哪里（sensor request n ready则IFE request n-1也必须ready 否则report invalid）确定丢帧是否因为UMD stuck,找到 deferred的request
--> Check 相应request的DRQ dump看它未满足的prop，确定这些prop由哪个node publish --> 继续追踪相应request 会publish prop的DRQ dump -->...

1.3.3 ISP apply fail Issue
Issue description
   Take a video with 4K@60fps, move the phone to keep the preview moving , and sometimes the preview freezes.(4K video 预览卡住)
Log analysis:
   1. Still recovery triggered by kernel consecutive invalid frames
   CamX : [ERROR][CORE ] camxpipeline.cpp:2698 CheckForRecovery() Hit SOF threshold of [17] consecutive frames with invalidrequestId; triggering watchdog recovery for pipeline PreviewVideo_0

   2. Always check sensor and IFE’s node dump for ‘Hit SOF threshold’ first
   CamX : [ DUMP][CORE ] camxsession.cpp:4960 DumpDebugInfo() + Stuck on Sequence Id: 1240 Request Id: 1241
   CamX : [ DUMP][CORE ] camxpipeline.cpp:3429 DumpDebugInfo() + Pipeline Name: PreviewVideo_0, Pipeline ID: 0, 0x7604a31800, CurrentRequestId: 1246

   //Pipeline PreviewVideo_0 is stuck on request 1241
   CamX : [ DUMP][CORE ] camxnode.cpp:4410 DumpDebugInfo() + NODE:[PreviewVideo_Sensor0]:Type:0 0x75da2ff6c0
   CamX : [ DUMP][CORE ] camxnode.cpp:4425 DumpDebugInfo() + Request: 1246 Unsuccessful. metadataComplete: 0, requestComplete: 0, numUnsignaledFences: 0, status: Deferred

   //Sensor:Request 1246 status deferred
   CamX : [ DUMP][CORE ] camxnode.cpp:4410 DumpDebugInfo() + NODE:[PreviewVideo_IFE0]:Type:65536 0x75f50195c0
   CamX : [ DUMP][CORE ] camxnode.cpp:4425 DumpDebugInfo() + Request: 1241 Unsuccessful. metadataComplete: 1, requestComplete: 0, numUnsignaledFences: 4, status: Submit

   //IFE:Request 1241 status Submit, but it still has 4 fences un-signaled
   3. 需要check IFE request 1241的fence为什么没有从kernel signal
   CAM_INFO: CAM-ISP: __cam_isp_ctx_epoch_in_applied: 1145 ctx:1 Report Bubble flag 1 req id:1238
   CAM_WARN: CAM-CRM: cam_req_mgr_process_trigger: 2290 Error recovery idx 1 status 1
   CAM_WARN: CAM-CRM: __cam_req_mgr_process_req: 1282 Err recovery done idx 1
   CAM_ERR : CAM-CRM: __cam_req_mgr_send_req: 565 APPLY FAILED pd 1 req_id 1241
   CAM_ERR : CAM-ISP: __cam_isp_ctx_apply_req: 4120 Apply failed in active substate 3
   //request 1238 Report Bubble,recovery done //ISP apply request 1241失败在它之后，并且没有为request 1241生成buffer,所以UMD不能获取request 1241的 IFE fence callbacks
Solution：
   Bubble and ISP apply 失败通常是因为performance低或者system schedule问题