8.稳定性专题

1. anr

https://code84.com/303466.html

一句话，规定的时间没有干完要干的事，就会发生anr
system_anr
场景
- input 5s
- service 前台20s 后台60s
- contentprivider超市比较少见
原因
- 主线程耗时复杂layout io
- binder对端block
- 子线程同步锁block
- binder被占满导致主线程无法和systemServer通信
- 得不到系统资源（cpu ram io）
进程角度分析
- 问题出在当前进程
  - 主线程耗时
  - 主线程消息队列存在耗时操作
  - 主线程被本进程的其他子线程block
- 问题出现远端进程，binder call 或者socket
分析log

怎么分析

anr trace adb pull /data/anr/traces.txt xxx，有可能没有记录到。不是第一案发现场
mainlog

//anr 发生时间
06-16 16:16:28.590  1853  2073 E ActivityManager: ANR in com.android.camera (com.android.camera/.Camera)
// anr 进程pid  如果为0，找个进程就被lmk杀死或者出现了出现了crash
06-16 16:16:28.590  1853  2073 E ActivityManager: PID: 27661
//原因是input timeout
06-16 16:16:28.590  1853  2073 E ActivityManager: Reason: Input dispatching timed out (com.android.camera/com.android.camera.Camera, Waiting to send non-key event because the touched window has not finished processing certain input events that were delivered to it over 500.0ms ago.  Wait queue length: 24.  Wait queue head age: 5511.1ms.)
//cpu负载  Linux系统中  uptime可以得到一段时间负载
//Load后面的三个数字的意思分别是1、5、15 分钟内正在使用和等待使用CPU 的活动进程的平均数。当CPU完全空闲的时候，平均负荷为0
//大于0.5 需要调查  1.0必须动手解决   5.0  系统有很严重的问题
06-16 16:16:28.590  1853  2073 E ActivityManager: Load: 16.25 / 29.48 / 38.33
06-16 16:16:28.590  1853  2073 E ActivityManager: CPU usage from 0ms to 8058ms later:
//cpu占用率
06-16 16:16:28.590  1853  2073 E ActivityManager:   58% 291/mediaserver: 51% user + 6.7% kernel / faults: 2457 minor 4 major
06-16 16:16:28.590  1853  2073 E ActivityManager:   27% 317/mm-qcamera-daemon: 21% user + 5.8% kernel / faults: 15965 minor
06-16 16:16:28.590  1853  2073 E ActivityManager:   0.4% 288/debuggerd: 0% user + 0.3% kernel / faults: 21615 minor 87 major
06-16 16:16:28.590  1853  2073 E ActivityManager:   17% 27661/com.android.camera: 10% user + 6.8% kernel / faults: 2412 minor 34 major
06-16 16:16:28.590  1853  2073 E ActivityManager:   16% 1853/system_server: 10% user + 6.4% kernel / faults: 1754 minor 87 major
06-16 16:16:28.590  1853  2073 E ActivityManager:   10% 539/sensors.qcom: 7.8% user + 2.6% kernel / faults: 16 minor
06-16 16:16:28.590  1853  2073 E ActivityManager:   4.4% 277/surfaceflinger: 1.8% user + 2.6% kernel / faults: 14 minor
06-16 16:16:28.590  1853  2073 E ActivityManager:   4% 203/mmcqd/0: 0% user + 4% kernel
....
06-16 16:16:28.590  1853  2073 E ActivityManager:   0.1% 29330/ksoftirqd/1: 0% user + 0.1% kernel
06-16 16:16:28.590  1853  2073 E ActivityManager: 96% TOTAL: 56% user + 29% kernel + 6.3% iowait + 4.1% softirq

可以从log中看到，发生anr的时候，Top进程的Cpu占用情况，user代表是用户空间，kernel是内核空间，一般的有如下的规

kswapd0 cpu占用率偏高，系统整体运行会缓慢，从而引起各种ANR。把问题转给"内存优化"，请他们进行优化。
logd CPU占用率偏高，也会引起系统卡顿和ANR，因为各个进程输出LOG的操作被阻塞从而执行的极为缓慢。
Vold占用CPU过高，会引起系统卡顿和ANR，请负责存储的同学先调查
qcom.sensor CPU占用率过高，会引起卡顿，请系统同学调查
应用自身CPU占用率较高，高概率应用自身问题
系统CPU占用率不高，但主线程在等待一个锁，高概率应用自身问题
应用处于D状态，发生ANR，如果最后的操作是refriger，那么是应用被冻结了，正常情况下是功耗优化引起的。
ps:如果ANR时间点前后，日志里有打印onTrimMemory，也可以作为内存紧张的一个参考判断

trace文件

main log可以得到anr基本信息，要得到阻塞的地方，还得靠trace文件，一般在/data/anr目录下，可在找个trace文件中搜索主线程堆栈

----- pid 27661 at 2017-06-16 16:16:20 -----
Cmd line: com.android.camera
"main" prio=5 tid=1 Waiting| group="main" sCount=1 dsCount=0 obj=0x75a4b5c8 self=0xb4cf6500| sysTid=27661 nice=-10 cgrp=default sched=0/0 handle=0xb6f6cb34| state=S schedstat=( 11242036155 8689191757 38520 ) utm=895 stm=229 core=0 HZ=100| stack=0xbe4ea000-0xbe4ec000 stackSize=8MB| held mutexes=at java.lang.Object.wait!(Native method)- waiting on <0x09e6a059> (a java.lang.Object)at java.lang.Thread.parkFor$(Thread.java:1220)- locked <0x09e6a059> (a java.lang.Object)at sun.misc.Unsafe.park(Unsafe.java:299)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:810)at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:970)at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1278)at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:203)// block点at android.app.SharedPreferencesImpl$EditorImpl$1.run(SharedPreferencesImpl.java:366)at android.app.QueuedWork.waitToFinish(QueuedWork.java:88)at android.app.ActivityThread.handleStopActivity(ActivityThread.java:3605)at android.app.ActivityThread.access$1300(ActivityThread.java:153)at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1399)at android.os.Handler.dispatchMessage(Handler.java:102)at android.os.Looper.loop(Looper.java:154)at android.app.ActivityThread.main(ActivityThread.java:5528)at java.lang.reflect.Method.invoke!(Native method)at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:740)at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:630)

字段	含义
tid=1	线程号
sysTid=27661	主线程的线程号和进程号相同
Waiting	线程状态，其中state也是线程状态，如果state=D代表底层被blocked了。
nice	nice值越小，则优先级越高。因为是主线程此处nice=-10, 可以看到优先级很高了
schedstat	括号中的3个数字，依次是Running, Runable, Switch，Running时间。Running时间：CPU运行的时间，单位ns。 Runable时间：RQ队列的等待时间，单位ns。 Switch次数：CPU调度切换次数
utm	该线程在用户态所执行的时间，单位是jiffies
stm	该线程在内核态所执行的时间，单位是jiffies
sCount	此线程被挂起的次数
dsCount	线程被调试器挂起的次数，当一个进程被调试后，sCount会重置为0，调试完毕后sCount会根据是否被正常挂起增长，但是dsCount不会被重置为0，所以dsCount也可以用来判断这个线程是否被调试过
self	线程本身的地址

线程状态

状态	值	说明
THREAD_ZOMBIE	0	TERMINATED
THREAD_RUNNING	1	RUNNABLE or running now
THREAD_TIMED_WAIT	2	TIMED_WAITING in Object.wait()
THREAD_MONITOR	3	BLOCKED on a monitor
THREAD_INITIALIZING	5	allocated not yet running
THREAD_STARTING	6	started not yet on thread list
THREAD_NATIVE	7	off in a JNI native method
THREAD_VMWAIT	8	waiting on a VM resource
THREAD_SUSPENDED	9	suspended usually by GC or debugger

线程状态

java线程状态

trace文件中是cpp的线程状态，对应关系：

![在这里插入图片描述](https://img-blog.csdnimg.cn/bb4e96be00fe4cdc85f6fd891fdbc9e2.png#pic_center

典型案例分析

主线程无卡顿，正常堆栈

"main" prio=5 tid=1 Native| group="main" sCount=1 dsCount=0 flags=1 obj=0x74b38080 self=0x7ad9014c00| sysTid=23081 nice=0 cgrp=default sched=0/0 handle=0x7b5fdc5548| state=S schedstat=( 284838633 166738594 505 ) utm=21 stm=7 core=1 HZ=100| stack=0x7fc95da000-0x7fc95dc000 stackSize=8MB| held mutexes=kernel: __switch_to+0xb0/0xbckernel: SyS_epoll_wait+0x288/0x364kernel: SyS_epoll_pwait+0xb0/0x124kernel: cpu_switch_to+0x38c/0x2258native: #00 pc 000000000007cd8c  /system/lib64/libc.so (__epoll_pwait+8)native: #01 pc 0000000000014d48  /system/lib64/libutils.so (android::Looper::pollInner(int)+148)native: #02 pc 0000000000014c18  /system/lib64/libutils.so (android::Looper::pollOnce(int, int*, int*, void**)+60)native: #03 pc 00000000001275f4  /system/lib64/libandroid_runtime.so (android::android_os_MessageQueue_nativePollOnce(_JNIEnv*, _jobject*, long, int)+44)at android.os.MessageQueue.nativePollOnce(Native method)at android.os.MessageQueue.next(MessageQueue.java:330)at android.os.Looper.loop(Looper.java:169)at android.app.ActivityThread.main(ActivityThread.java:7073)at java.lang.reflect.Method.invoke(Native method)at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:536)at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:876)

android_os_MessageQueue_nativePollOnce 很正常的空闲堆栈，表明主线程正在等待新的消息
可能有两个原因
- 该ANR是CPU抢占或内存紧张等其他因素引起，可以去分析cpu 内存情况
- 这份ANR日志抓取的时候，主线程已经恢复正常，可以关注抓取日志的时间和ANR发生的时间是否相隔过久，时间过久这个堆栈就没有分析意义了

主线程耗时

"main" prio=5 tid=1 Runnable| group="main" sCount=0 dsCount=0 flags=0 obj=0x72deb848 self=0x7748c10800| sysTid=8968 nice=-10 cgrp=default sched=0/0 handle=0x77cfa75ed0| state=R schedstat=( 24783612979 48520902 756 ) utm=2473 stm=5 core=5 HZ=100| stack=0x7fce68b000-0x7fce68d000 stackSize=8192KB| held mutexes= "mutator lock"(shared held)at com.example.test.MainActivity$onCreate$2.onClick(MainActivity.kt:20)——关键行！！！at android.view.View.performClick(View.java:7187)at android.view.View.performClickInternal(View.java:7164)at android.view.View.access$3500(View.java:813)at android.view.View$PerformClick.run(View.java:27640)at android.os.Handler.handleCallback(Handler.java:883)at android.os.Handler.dispatchMessage(Handler.java:100)at android.os.Looper.loop(Looper.java:230)at android.app.ActivityThread.main(ActivityThread.java:7725)at java.lang.reflect.Method.invoke(Native method)at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:526)at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1034)

主线程正处于执行状态，看堆栈信息可知不是处于空闲状态，发生ANR是因为一处click监听函数里执行了耗时操作

主线程被锁阻塞

"main" prio=5 tid=1 Blocked| group="main" sCount=1 dsCount=0 flags=1 obj=0x72deb848 self=0x7748c10800| sysTid=22838 nice=-10 cgrp=default sched=0/0 handle=0x77cfa75ed0| state=S schedstat=( 390366023 28399376 279 ) utm=34 stm=5 core=1 HZ=100| stack=0x7fce68b000-0x7fce68d000 stackSize=8192KB| held mutexes=at com.example.test.MainActivity$onCreate$1.onClick(MainActivity.kt:15)等待的锁是<0x01aed1da>，这个锁的持有者是线程 3。进一步搜索 “tid=3” 找到线程3， 发现它正在TimeWating。- waiting to lock <0x01aed1da> (a java.lang.Object) held by thread 3 ——————关键行！！！at android.view.View.performClick(View.java:7187)at android.view.View.performClickInternal(View.java:7164)at android.view.View.access$3500(View.java:813)at android.view.View$PerformClick.run(View.java:27640)at android.os.Handler.handleCallback(Handler.java:883)at android.os.Handler.dispatchMessage(Handler.java:100)at android.os.Looper.loop(Looper.java:230)at android.app.ActivityThread.main(ActivityThread.java:7725)at java.lang.reflect.Method.invoke(Native method)at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:526)at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1034)........省略N行....."WQW TEST" prio=5 tid=3 TimeWating| group="main" sCount=1 dsCount=0 flags=1 obj=0x12c44230 self=0x772f0ec000| sysTid=22938 nice=0 cgrp=default sched=0/0 handle=0x77391fbd50| state=S schedstat=( 274896 0 1 ) utm=0 stm=0 core=1 HZ=100| stack=0x77390f9000-0x77390fb000 stackSize=1039KB| held mutexes=at java.lang.Thread.sleep(Native method)- sleeping on <0x043831a6> (a java.lang.Object)at java.lang.Thread.sleep(Thread.java:440)- locked <0x043831a6> (a java.lang.Object)at java.lang.Thread.sleep(Thread.java:356)at com.example.test.MainActivity$onCreate$2$thread$1.run(MainActivity.kt:22)- locked <0x01aed1da> (a java.lang.Object)————————————————————关键行！！！at java.lang.Thread.run(Thread.java:919)

其中等待的锁是<0x01aed1da>，这个锁的持有者是线程 3。进一步搜索 “tid=3” 找到线程3，发现它正在TimeWating。

那么ANR的原因找到了：线程3持有了一把锁，并且自身长时间不释放，主线程等待这把锁发生超时。在线上环境中，常见因锁而ANR的场景是SharePreference写入。

cpu被抢占

CPU usage from 0ms to 10625ms later (2020-03-09 14:38:31.633 to 2020-03-09 14:38:42.257):543% 2045/com.alibaba.android.rimet: 54% user + 89% kernel / faults: 4608 minor 1 major ————关键行！！！99% 674/android.hardware.camera.provider@2.4-service: 81% user + 18% kernel / faults: 403 minor24% 32589/com.wang.test: 22% user + 1.4% kernel / faults: 7432 minor 1 major........省略N行.....

钉钉的进程，占据CPU高达543%，抢占了大部分CPU资源，因而导致发生ANR。

内存紧张

CPU和堆栈都很正常（不贴出来了），仍旧发生ANR，考虑是内存紧张。

从CPU第一行信息可以发现，ANR的时间点是2020-10-31 22:38:58.468—CPU usage from 0ms to 21752ms later (2020-10-31 22:38:58.468 to 2020-10-31 22:39:20.220)

接着去系统日志里搜索am_meminfo，这个没有搜索到。再次搜索onTrimMemory，果然发现了很多条记录；

10-31 22:37:19.749 20733 20733 E Runtime : onTrimMemory level:80,pid:com.xxx.xxx:Launcher0
10-31 22:37:33.458 20733 20733 E Runtime : onTrimMemory level:80,pid:com.xxx.xxx:Launcher0
10-31 22:38:00.153 20733 20733 E Runtime : onTrimMemory level:80,pid:com.xxx.xxx:Launcher0
10-31 22:38:58.731 20733 20733 E Runtime : onTrimMemory level:80,pid:com.xxx.xxx:Launcher0
10-31 22:39:02.816 20733 20733 E Runtime : onTrimMemory level:80,pid:com.xxx.xxx:Launcher0

可以看出，在发生ANR的时间点前后，内存都处于紧张状态，level等级是80，查看Android API 文档；

/*** Level for {@link #onTrimMemory(int)}: the process is nearing the end* of the background LRU list, and if more memory isn't found soon it will* be killed.*/static final int TRIM_MEMORY_COMPLETE = 80;

可知80这个等级是很严重的，应用马上就要被杀死，被杀死的这个应用从名字可以看出来是桌面，连桌面都快要被杀死，那普通应用能好到哪里去呢？

一般来说，发生内存紧张，会导致多个应用发生ANR，所以在日志中如果发现有多个应用一起ANR了，可以初步判定，此ANR与你的应用无关

系统服务超时

系统服务超时一般会包含BinderProxy.transactNative关键字，请看如下日志：

"main" prio=5 tid=1 Native| group="main" sCount=1 dsCount=0 flags=1 obj=0x727851e8 self=0x78d7060e00| sysTid=4894 nice=0 cgrp=default sched=0/0 handle=0x795cc1e9a8| state=S schedstat=( 8292806752 1621087524 7167 ) utm=707 stm=122 core=5 HZ=100| stack=0x7febb64000-0x7febb66000 stackSize=8MB| held mutexes=kernel: __switch_to+0x90/0xc4kernel: binder_thread_read+0xbd8/0x144ckernel: binder_ioctl_write_read.constprop.58+0x20c/0x348kernel: binder_ioctl+0x5d4/0x88ckernel: do_vfs_ioctl+0xb8/0xb1ckernel: SyS_ioctl+0x84/0x98kernel: cpu_switch_to+0x34c/0x22c0native: #00 pc 000000000007a2ac  /system/lib64/libc.so (__ioctl+4)native: #01 pc 00000000000276ec  /system/lib64/libc.so (ioctl+132)native: #02 pc 00000000000557d4  /system/lib64/libbinder.so (android::IPCThreadState::talkWithDriver(bool)+252)native: #03 pc 0000000000056494  /system/lib64/libbinder.so (android::IPCThreadState::waitForResponse(android::Parcel*, int*)+60)native: #04 pc 00000000000562d0  /system/lib64/libbinder.so (android::IPCThreadState::transact(int, unsigned int, android::Parcel const&, android::Parcel*, unsigned int)+216)native: #05 pc 000000000004ce1c  /system/lib64/libbinder.so (android::BpBinder::transact(unsigned int, android::Parcel const&, android::Parcel*, unsigned int)+72)native: #06 pc 00000000001281c8  /system/lib64/libandroid_runtime.so (???)native: #07 pc 0000000000947ed4  /system/framework/arm64/boot-framework.oat (Java_android_os_BinderProxy_transactNative__ILandroid_os_Parcel_2Landroid_os_Parcel_2I+196)at android.os.BinderProxy.transactNative(Native method) ————————————————关键行！！！at android.os.BinderProxy.transact(Binder.java:804)at android.net.IConnectivityManager$Stub$Proxy.getActiveNetworkInfo(IConnectivityManager.java:1204)—关键行！at android.net.ConnectivityManager.getActiveNetworkInfo(ConnectivityManager.java:800)at com.xiaomi.NetworkUtils.getNetworkInfo(NetworkUtils.java:2)at com.xiaomi.frameworkbase.utils.NetworkUtils.getNetWorkType(NetworkUtils.java:1)at com.xiaomi.frameworkbase.utils.NetworkUtils.isWifiConnected(NetworkUtils.java:1

从堆栈可以看出获取网络信息发生了ANR：getActiveNetworkInfo。

前文有讲过：系统的服务都是Binder机制（16个线程），服务能力也是有限的，有可能系统服务长时间不响应导致ANR。如果其他应用占用了所有Binder线程，那么当前应用只能等待。

可进一步搜索：blockUntilThreadAvailable关键字：

at android.os.Binder.blockUntilThreadAvailable(Native method)

如果有发现某个线程的堆栈，包含此字样，可进一步看其堆栈，确定是调用了什么系统服务。此类ANR也是属于系统环境的问题，如果某类型机器上频繁发生此问题，应用层可以考虑规避策略

对端block

 首先需要去看的就是traces.txt 中ANR进程的主线程的栈。以settting的这次ANR为例："main" prio=5 tid=1 Native| group="main" sCount=1 dsCount=0 obj=0x73ee6470 self=0xb4d76500| sysTid=22831 nice=0 cgrp=default sched=0/0 handle=0xb6f4bc00| state=S schedstat=( 0 0 0 ) utm=22 stm=22 core=0 HZ=100| stack=0xbe283000-0xbe285000 stackSize=8MB| held mutexes=native: #00 pc 000410ac  /system/lib/libc.so (__ioctl+8)native: #01 pc 000477e5  /system/lib/libc.so (ioctl+14)native: #02 pc 0001e7c5  /system/lib/libbinder.so (android::IPCThreadState::talkWithDriver(bool)+132)native: #03 pc 0001ee17  /system/lib/libbinder.so (android::IPCThreadState::waitForResponse(android::Parcel*, int*)+38)native: #04 pc 0001efcd  /system/lib/libbinder.so (android::IPCThreadState::transact(int, unsigned int, android::Parcel const&, android::Parcel*, unsigned int)+124)native: #05 pc 00019fb7  /system/lib/libbinder.so (android::BpBinder::transact(unsigned int, android::Parcel const&, android::Parcel*, unsigned int)+30)native: #06 pc 00086de9  /system/lib/libandroid_runtime.so (???)native: #07 pc 00d94629  /data/dalvik-cache/arm/system@framework@boot.oat (Java_android_os_BinderProxy_transactNative__ILandroid_os_Parcel_2Landroid_os_Parcel_2I+140)at android.os.BinderProxy.transactNative(Native method)at android.os.BinderProxy.transact(Binder.java:503)//此处blockat android.net.INetworkPolicyManager$Stub$Proxy.getNetworkPolicies(INetworkPolicyManager.java:410)at android.net.NetworkPolicyManager.getNetworkPolicies(NetworkPolicyManager.java:174)at com.android.settings.net.NetworkPolicyEditor.read(NetworkPolicyEditor.java:57)at com.android.settings.DataUsageSummary.onCreate(DataUsageSummary.java:361)at android.app.Fragment.performCreate(Fragment.java:2202)at android.app.FragmentManagerImpl.moveToState(FragmentManager.java:942)at android.app.FragmentManagerImpl.moveToState(FragmentManager.java:1148)at android.app.BackStackRecord.run(BackStackRecord.java:793)at android.app.FragmentManagerImpl.execPendingActions(FragmentManager.java:1535)at android.app.FragmentManagerImpl.executePendingTransactions(FragmentManager.java:562)at com.android.settings.SettingsActivity.switchToFragment(SettingsActivity.java:1084)at com.android.settings.SettingsActivity.onCreate(SettingsActivity.java:657)at android.app.Activity.performCreate(Activity.java:6251)at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1110)at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2370)at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2477)at android.app.ActivityThread.-wrap11(ActivityThread.java:-1)at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1345)at android.os.Handler.dispatchMessage(Handler.java:102)at android.os.Looper.loop(Looper.java:148)at android.app.ActivityThread.main(ActivityThread.java:5438)at java.lang.reflect.Method.invoke!(Native method)at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:762)at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:652)主线程被对端block，而对端是在systemserver中的NetworkPolicyManager。接下来继续去查NetworkPolicyManager为何会被block。通常情况下，可以搜索getNetworkPolicies，一般的对端的函数并不会修改函数名：看到我们的对端："Binder_4" prio=5 tid=56 Blocked| group="main" sCount=1 dsCount=0 obj=0x1321a0a0 self=0xad31e200| sysTid=2491 nice=0 cgrp=default sched=0/0 handle=0x9fd00930| state=S schedstat=( 0 0 0 ) utm=46441 stm=46582 core=1 HZ=100| stack=0x9fc04000-0x9fc06000 stackSize=1014KB| held mutexes=at com.android.server.net.NetworkPolicyManagerService.getNetworkPolicies(NetworkPolicyManagerService.java:1696)- waiting to lock <0x07439315> (a java.lang.Object) held by thread 35at android.net.INetworkPolicyManager$Stub.onTransact(INetworkPolicyManager.java:145)at android.os.Binder.execTransact(Binder.java:453被Tid=35的人拿住了一把锁（0x07439315），那么继续看tid=35是谁，有两种方法：1. 搜索tid=352. 搜索0x07439315，找到  - locked <0x07439315> (a java.lang.Object)"NetworkPolicy" prio=5 tid=35 TimedWaiting| group="main" sCount=1 dsCount=0 obj=0x12d98940 self=0x9f91f700| sysTid=2415 nice=0 cgrp=default sched=0/0 handle=0xa0f33930| state=S schedstat=( 0 0 0 ) utm=7681 stm=7783 core=0 HZ=100| stack=0xa0e31000-0xa0e33000 stackSize=1038KB| held mutexes=at java.lang.Object.wait!(Native method)- waiting on <0x02580c1b> (a java.lang.Object)at java.lang.Thread.parkFor$(Thread.java:1220)- locked <0x02580c1b> (a java.lang.Object)at sun.misc.Unsafe.park(Unsafe.java:299)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2053)at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:372)at com.android.server.NativeDaemonConnector$ResponseQueue.remove(NativeDaemonConnector.java:634)at com.android.server.NativeDaemonConnector.executeForList(NativeDaemonConnector.java:426)at com.android.server.NativeDaemonConnector.execute(NativeDaemonConnector.java:345)at com.android.server.NativeDaemonConnector.execute(NativeDaemonConnector.java:340)at com.android.server.NetworkManagementService.setInterfaceQuota(NetworkManagementService.java:1712)- locked <0x0b0f91b8> (a java.lang.Object)at com.android.server.net.NetworkPolicyManagerService.setInterfaceQuota(NetworkPolicyManagerService.java:2421)at com.android.server.net.NetworkPolicyManagerService.updateNetworkRulesLocked(NetworkPolicyManagerService.java:1232)at com.android.server.net.NetworkPolicyManagerService$14.onReceive(NetworkPolicyManagerService.java:1060)- locked <0x07439315> (a java.lang.Object)at android.app.LoadedApk$ReceiverDispatcher$Args.run(LoadedApk.java:881)at android.os.Handler.handleCallback(Handler.java:739)at android.os.Handler.dispatchMessage(Handler.java:95)at android.os.Looper.loop(Looper.java:148)at android.os.HandlerThread.run(HandlerThread.java:61)可以看到，NetworkPolicy在通过NativeDaemonConnector和netd通信（setInterfaceQuota）我们结合log来看下是否有有用信息，按照之前的经验，netd在执行完cmd的时候，会打印出slow operation。在发生ANR的前后，查找netd相关的打印：06-19 15:29:00.997  1235  1270 I am_anr  : [0,22831,com.android.settings,818429509,Input dispatching timed out (Waiting because no window has focus but there is a focused application that may eventually add a window when it finishes starting up.)]06-19 15:29:05.683  1235  2415 E NetdConnector: NDC Command {55445 bandwidth setiquota seth_w0 9223372036854775807} took too long (4755ms06-19 15:29:05.723  1235  2491 I dvm_lock_sample: [system_server,1,Binder_4,4919,NetworkPolicyManagerService.java,1696,-,1056,100]从eventslog中还可以发现，setting出现了两次连续的ANR，而上下文中都有类似上面的dvm_lock_sample NetworkPolicyManagerService相关的告警。  因此虽然15:28的这次ANR并没有打出有用的栈，但是我们还是可以猜测出这两次ANR的原因都是netd的cmd耗时太久导致的（在主线程的调用路径上存在必然的block）那么，在netd可能无法修改的情况下，我们应该如何去resolve这个问题呢。将可能存在block的操作放到非UI线程中去做。

再分析一个

  打开dropbox中对应的system_app_anrxxxx:查看主线程的栈："main" prio=5 tid=1 Native| group="main" sCount=1 dsCount=0 obj=0x752b0000 self=0xb4276500| sysTid=25390 nice=-1 cgrp=default sched=3/0 handle=0xb6f18b34| state=S schedstat=( 0 0 0 ) utm=81 stm=12 core=2 HZ=100| stack=0xbe78b000-0xbe78d000 stackSize=8MB| held mutexes=kernel: (couldn't read /proc/self/task/25390/stack)native: #00 pc 000422d0  /system/lib/libc.so (__ioctl+8)native: #01 pc 00047825  /system/lib/libc.so (ioctl+14)native: #02 pc 0001e835  /system/lib/libbinder.so (_ZN7android14IPCThreadState14talkWithDriverEb+132)native: #03 pc 0001ee93  /system/lib/libbinder.so (_ZN7android14IPCThreadState15waitForResponseEPNS_6ParcelEPi+38)native: #04 pc 0001f049  /system/lib/libbinder.so (_ZN7android14IPCThreadState8transactEijRKNS_6ParcelEPS1_j+124)native: #05 pc 00019fe3  /system/lib/libbinder.so (_ZN7android8BpBinder8transactEjRKNS_6ParcelEPS1_j+30)native: #06 pc 0008a035  /system/lib/libandroid_runtime.so (???)native: #07 pc 00d78869  /data/dalvik-cache/arm/system@framework@boot.oat (Java_android_os_BinderProxy_transactNative__ILandroid_os_Parcel_2Landroid_os_Parcel_2I+140)at android.os.BinderProxy.transactNative(Native method)at android.os.BinderProxy.transact(Binder.java:510)at android.os.storage.IMountService$Stub$Proxy.getVolumeList(IMountService.java:771)at android.os.storage.StorageManager.getVolumeList(StorageManager.java:883)at android.os.Environment$UserEnvironment.getExternalDirs(Environment.java:95)at android.os.Environment.getExternalStorageDirectory(Environment.java:354)at com.huawei.common.utils.PathUtils.<clinit>(PathUtils.java:51)at com.huawei.common.utils.PathUtils.getWorkspacePath(PathUtils.java:80)at com.huawei.common.components.log.Logger.<clinit>(Logger.java:37)at com.huawei.common.components.log.Logger.i(Logger.java:162)at com.huawei.hwvplayer.data.db.DbProvider.attachInfo(DbProvider.java:89)at android.app.ActivityThread.installProvider(ActivityThread.java:5279)at android.app.ActivityThread.installContentProviders(ActivityThread.java:4868)at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4799)at android.app.ActivityThread.access$1600(ActivityThread.java:165)at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1436)at android.os.Handler.dispatchMessage(Handler.java:102)at android.os.Looper.loop(Looper.java:188)at android.app.ActivityThread.main(ActivityThread.java:5578)at java.lang.reflect.Method.invoke!(Native method)at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:794)at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:684)主线程尝试去调用MountService的 getVolumeList接口，可能没有返回。查看system_server中相关的，搜索getVolumeList发现systemserver中有三个Binder线程和主线程被block，我们的对端是哪个Binder线程暂时无法确认，但这并不影响我们继续分析，因为他们被blockd的路径是一致的："main" prio=5 tid=1 Blocked| group="main" sCount=1 dsCount=0 obj=0x752b0000 self=0xb4276500| sysTid=22735 nice=-2 cgrp=default sched=0/0 handle=0xb6f18b34| state=S schedstat=( 0 0 0 ) utm=432 stm=85 core=1 HZ=100| stack=0xbe78b000-0xbe78d000 stackSize=8MB| held mutexes=at com.android.server.MountService.getVolumeList(MountService.java:2759)- waiting to lock <0x0eeb54f1> (a java.lang.Object) held by thread 40at android.os.storage.StorageManager.getVolumeList(StorageManager.java:883)at android.os.storage.StorageManager.getVolumeList(StorageManager.java:858)at android.os.storage.StorageManager.getPrimaryVolume(StorageManager.java:906)at com.android.server.usb.UsbDeviceManager.systemReady(UsbDeviceManager.java:327)at com.android.server.usb.UsbService.systemReady(UsbService.java:181)at com.android.server.usb.UsbService$Lifecycle.onBootPhase(UsbService.java:78)at com.android.server.SystemServiceManager.startBootPhase(SystemServiceManager.java:135)at com.android.server.SystemServer$3.run(SystemServer.java:1489)at com.android.server.am.ActivityManagerService.systemReady(ActivityManagerService.java:12417)at com.android.server.am.HwActivityManagerService.systemReady(HwActivityManagerService.java:960)at com.android.server.SystemServer.startOtherServices(SystemServer.java:1485)at com.android.server.SystemServer.run(SystemServer.java:381)at com.android.server.SystemServer.main(SystemServer.java:272)at java.lang.reflect.Method.invoke!(Native method)at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:794)at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:684)"Binder_8" prio=5 tid=76 Blocked| group="main" sCount=1 dsCount=0 obj=0x13bd60a0 self=0x9c1abe00| sysTid=25191 nice=-1 cgrp=default sched=0/0 handle=0x97158930| state=S schedstat=( 0 0 0 ) utm=9 stm=7 core=2 HZ=100| stack=0x9705c000-0x9705e000 stackSize=1014KB| held mutexes=at com.android.server.MountService.getVolumeList(MountService.java:2759)- waiting to lock <0x0eeb54f1> (a java.lang.Object) held by thread 40at android.os.storage.IMountService$Stub.onTransact(IMountService.java:1634)at android.os.Binder.execTransact(Binder.java:453)"Binder_2" prio=5 tid=8 Blocked| group="main" sCount=1 dsCount=0 obj=0x12cac0a0 self=0xaebf0300| sysTid=22761 nice=-1 cgrp=default sched=0/0 handle=0xaef7d930| state=S schedstat=( 0 0 0 ) utm=41 stm=25 core=0 HZ=100| stack=0xaee81000-0xaee83000 stackSize=1014KB| held mutexes=at com.android.server.MountService.getVolumeList(MountService.java:2759)- waiting to lock <0x0eeb54f1> (a java.lang.Object) held by thread 40at android.os.storage.IMountService$Stub.onTransact(IMountService.java:1634)at android.os.Binder.execTransact(Binder.java:453)他们均是被tid=40的人block，按照上面的方法搜索tid=40或者 0x0eeb54f1得到block的人："MountService" prio=5 tid=40 TimedWaiting| group="main" sCount=1 dsCount=0 obj=0x132c1160 self=0x9ce57400| sysTid=23512 nice=0 cgrp=default sched=0/0 handle=0x9a239930| state=S schedstat=( 0 0 0 ) utm=29 stm=2 core=2 HZ=100| stack=0x9a137000-0x9a139000 stackSize=1038KB| held mutexes=at java.lang.Object.wait!(Native method)- waiting on <0x00fea1f3> (a java.lang.Object)at java.lang.Thread.parkFor$(Thread.java:1235)- locked <0x00fea1f3> (a java.lang.Object)at sun.misc.Unsafe.park(Unsafe.java:299)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2053)at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:372)at com.android.server.NativeDaemonConnector$ResponseQueue.remove(NativeDaemonConnector.java:777)at com.android.server.NativeDaemonConnector.executeForList(NativeDaemonConnector.java:489)at com.android.server.NativeDaemonConnector.execute(NativeDaemonConnector.java:386)at com.android.server.NativeDaemonConnector.execute(NativeDaemonConnector.java:381)at com.android.server.MountService.resetIfReadyAndConnectedLocked(MountService.java:827)at com.android.server.MountService.handleSystemReady(MountService.java:776)- locked <0x0eeb54f1> (a java.lang.Object)at com.android.server.MountService.access$500(MountService.java:152)at com.android.server.MountService$MountServiceHandler.handleMessage(MountService.java:596)at android.os.Handler.dispatchMessage(Handler.java:102)at android.os.Looper.loop(Looper.java:150)at android.os.HandlerThread.run(HandlerThread.java:61)和上面netd类似的，mountservice也是通过ndc和vold通信，这里我们需要继续查看是否vold存在异常。之前提到过，类似这种同步锁block的，dvm_lock_sample一定会有打印，于是先去找eventslog，不过这个是华为的log，是没有eventlog的。而华为实现了一个blockMonitor的功能，和dvm_lock_sample类似，当某个操作特别耗时的时候，会将其打印出来：在ANR的附近找到如下：07-19 10:17:50.739 25271 25271 W BlockMonitor: The binder calling took 55209ms.
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.os.BlockMonitor.checkBinderTime(BlockMonitor.java:141)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.os.BinderProxy.transact(Binder.java:511)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.os.storage.IMountService$Stub$Proxy.getVolumeList(IMountService.java:771)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.os.storage.StorageManager.getVolumeList(StorageManager.java:883)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.os.Environment$UserEnvironment.getExternalDirs(Environment.java:95)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.os.Environment.getExternalStorageDirectory(Environment.java:354)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.support.v4.content.FileProvider.parsePathStrategy(FileProvider.java:583)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.support.v4.content.FileProvider.getPathStrategy(FileProvider.java:534)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.support.v4.content.FileProvider.attachInfo(FileProvider.java:352)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.app.ActivityThread.installProvider(ActivityThread.java:5279)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.app.ActivityThread.installContentProviders(ActivityThread.java:4868)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.app.ActivityThread.handleBindApplication(ActivityThread.java:4799)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.app.ActivityThread.access$1600(ActivityThread.java:165)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.app.ActivityThread$H.handleMessage(ActivityThread.java:1436)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.os.Handler.dispatchMessage(Handler.java:102)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.os.Looper.loop(Looper.java:188)
07-19 10:17:50.739 25271 25271 W BlockMonitor: android.app.ActivityThread.main(ActivityThread.java:5578)
07-19 10:17:50.739 25271 25271 W BlockMonitor: java.lang.reflect.Method.invoke(Native Method)
07-19 10:17:50.739 25271 25271 W BlockMonitor: com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:794)
07-19 10:17:50.739 25271 25271 W BlockMonitor: com.android.internal.os.ZygoteInit.main(ZygoteInit.java:684)
再加上之前的：
07-19 10:17:50.729 22735 23512 E NativeDaemonConnector.ResponseQueue: Timeout waiting for response
07-19 10:17:50.729 22735 23512 E VoldConnector: timed-out waiting for response to 4 volume reset
07-19 10:17:50.729 22735 23512 E VoldConnector: timed-out waiting for response mOutputStream = android.net.LocalSocketImpl$SocketOutputStream@922644, mSocket = vold
07-19 10:17:50.731 22735 23512 W MountService: Failed to reset vold
07-19 10:17:50.731 22735 23512 W MountService: com.android.server.NativeDaemonTimeoutException: command '4 volume reset' failed with 'null'
07-19 10:17:50.731 22735 23512 W MountService: 	at com.android.server.NativeDaemonConnector.executeForList(NativeDaemonConnector.java:501)
07-19 10:17:50.731 22735 23512 W MountService: 	at com.android.server.NativeDaemonConnector.execute(NativeDaemonConnector.java:386)
07-19 10:17:50.731 22735 23512 W MountService: 	at com.android.server.NativeDaemonConnector.execute(NativeDaemonConnector.java:381)
07-19 10:17:50.731 22735 23512 W MountService: 	at com.android.server.MountService.resetIfReadyAndConnectedLocked(MountService.java:827)
07-19 10:17:50.731 22735 23512 W MountService: 	at com.android.server.MountService.handleSystemReady(MountService.java:776)
07-19 10:17:50.731 22735 23512 W MountService: 	at com.android.server.MountService.access$500(MountService.java:152)
07-19 10:17:50.731 22735 23512 W MountService: 	at com.android.server.MountService$MountServiceHandler.handleMessage(MountService.java:596)
07-19 10:17:50.731 22735 23512 W MountService: 	at android.os.Handler.dispatchMessage(Handler.java:102)
07-19 10:17:50.731 22735 23512 W MountService: 	at android.os.Looper.loop(Looper.java:150)
07-19 10:17:50.731 22735 23512 W MountService: 	at android.os.HandlerThread.run(HandlerThread.java:61)
我们有理由去推断vold的状态是不对的，但是又没有vold的栈。
那么全局grep vold试试能不能找到线索，在kernellog中发现了vold的异常：
07-19 14:22:24.669 <3>[10772.492156] c0 Freezing of tasks failed after 20.008 seconds (1 tasks refusing to freeze, wq_busy=0):
07-19 14:22:24.669 <6>[10772.492217] c0 vold            R running      0   224      1 0x00000001
07-19 14:22:24.669 <4>[10772.492278] c0 [<c05ebecc>] (__schedule+0x38c/0x5bc) from [<c05ea478>] (schedule_timeout+0x18/0x1e8)
07-19 14:22:24.669 <4>[10772.492309] c0 [<c05ea478>] (schedule_timeout+0x18/0x1e8) from [<c05eb90c>] (wait_for_common+0x11c/0x164)
07-19 14:22:24.669 <4>[10772.492309] c0 [<c05eb90c>] (wait_for_common+0x11c/0x164) from [<c03cd8c8>] (mmc_wait_for_req+0xb4/0xe4)
07-19 14:22:24.669 <4>[10772.492339] c0 [<c03cd8c8>] (mmc_wait_for_req+0xb4/0xe4) from [<c03cd95c>] (mmc_wait_for_cmd+0x64/0x74)
07-19 14:22:24.669 <4>[10772.492370] c0 [<c03cd95c>] (mmc_wait_for_cmd+0x64/0x74) from [<c03d41f0>] (mmc_send_status+0x6c/0x8c)
07-19 14:22:24.670 <4>[10772.492400] c0 [<c03d41f0>] (mmc_send_status+0x6c/0x8c) from [<c03d4504>] (sd_send_status+0x14/0x44)
07-19 14:22:24.670 <4>[10772.492431] c0 [<c03d4504>] (sd_send_status+0x14/0x44) from [<c03d491c>] (mmc_lock_unlock_by_buf+0xac/0x168)
07-19 14:22:24.670 <4>[10772.492431] c0 [<c03d491c>] (mmc_lock_unlock_by_buf+0xac/0x168) from [<c03dabd8>] (mmc_lockable_store+0x594/0x75c)
07-19 14:22:24.670 <4>[10772.492461] c0 [<c03dabd8>] (mmc_lockable_store+0x594/0x75c) from [<c029d560>] (dev_attr_store+0x18/0x24)
07-19 14:22:32.070 <4>[10772.492492] c0 [<c029d560>] (dev_attr_store+0x18/0x24) from [<c013b370>] (sysfs_write_file+0x104/0x148)
07-19 14:22:32.070 <4>[10772.492522] c0 [<c013b370>] (sysfs_write_file+0x104/0x148) from [<c00eabb4>] (vfs_write+0xd0/0x180)
07-19 14:22:32.070 <4>[10772.492553] c0 [<c00eabb4>] (vfs_write+0xd0/0x180) from [<c00eb070>] (SyS_write+0x38/0x68)
07-19 14:22:32.071 <4>[10772.492583] c0 [<c00eb070>] (SyS_write+0x38/0x68) from [<c000e840>] (ret_fast_syscall+0x0/0x30)
vold一直在这个操作中没有退出来，所以不能响应客户端的请求，从而导致了ANR。
这个问题需要mmc的同事进一步去分析，目前怀疑是SD卡发生了错误。

系统资源紧张造成的anr

通常情况下，一般这种问题的表现的可能形式是
1. traces.txt中的主线程的栈在一个非常common的操作中，如new一个变量，读取某个文件等
2. ANR的trace中IOW相当高（IOW+CPU sys+usr=100）
3. Kswapd非常活跃
4. Low Ram 配置
5. kernel log中 D状态的用户进程比较多，且都block在内存相关，且内存比较紧张.
6. 连续出现ANR，slog和ams一直在dumptrace，造成比较大的IO压力
7. 前台正在玩一个很大的游戏
8. 某个应用正在安装做dex2oat.
出现这种情况下，我们需要做的是：
1. 针对性的调整lmk参数，减少后台的数量
2. 检查是否有不应该常驻的进程
3. 是否需要修改伪前台的adj，减少常驻内存
4. 是否需要关闭dex2oat
此类问题比较难解决。

] (SyS_write+0x38/0x68) from [] (ret_fast_syscall+0x0/0x30)
vold一直在这个操作中没有退出来，所以不能响应客户端的请求，从而导致了ANR。
这个问题需要mmc的同事进一步去分析，目前怀疑是SD卡发生了错误。


## 系统资源紧张造成的anr通常情况下，一般这种问题的表现的可能形式是1. traces.txt中的主线程的栈在一个非常common的操作中，如new一个变量，读取某个文件等2. ANR的trace中IOW相当高（IOW+CPU sys+usr=100）3. Kswapd非常活跃4. Low Ram 配置5. kernel log中 D状态的用户进程比较多，且都block在内存相关，且内存比较紧张.6. 连续出现ANR，slog和ams一直在dumptrace，造成比较大的IO压力7. 前台正在玩一个很大的游戏8. 某个应用正在安装做dex2oat.出现这种情况下，我们需要做的是：1. 针对性的调整lmk参数，减少后台的数量2. 检查是否有不应该常驻的进程3. 是否需要修改伪前台的adj，减少常驻内存4. 是否需要关闭dex2oat此类问题比较难解决。