Watchdog 工作机制解析

一、Watchdog 的初始化

1.1 startOtherServices()

SystemServer.java

private void startOtherServices() {    ...    // 创建 watchdog【1.2节】    final Watchdog watchdog = Watchdog.getInstance();    // init watchdog【1.3节】    watchdog.init(context, mActivityManagerService);    ...    mSystemServiceManager.startBootPhase(SystemService.PHASE_LOCK_SETTINGS_READY); // 480    ...    mActivityManagerService.systemReady(new Runnable() {       public void run() {           mSystemServiceManager.startBootPhase(                   SystemService.PHASE_ACTIVITY_MANAGER_READY);           ...           // watchdog 启动【1.4节】           Watchdog.getInstance().start();           mSystemServiceManager.startBootPhase(                   SystemService.PHASE_THIRD_PARTY_APPS_CAN_START);        }    }}

从上面可以看到 watchdog 初始化的过程主要分为三步：

• create watchdog
• init watchdog
• start watchdog

下面我们分这三步来分别看一下

1.2 Watchdog.getInstance()

Watchdog.java

    public static Watchdog getInstance() {        if (sWatchdog == null) {            sWatchdog = new Watchdog();        }        return sWatchdog;    }

可以看到这就是一个单例模式，下面看一下 Watchdog 的构造函数

1.2.1 Watchdog()

Watchdog.java

    private Watchdog() {        super("watchdog");        // 初始化各 handler checker        // fg 线程是最主要的 check 对象，同时各个 MonitorChecker 也会被添加到这个 HandlerChecker        mMonitorChecker = new HandlerChecker(FgThread.getHandler(),                "foreground thread", DEFAULT_TIMEOUT);        mHandlerCheckers.add(mMonitorChecker);        // Add checker for main thread.  We only do a quick check since there        // can be UI running on the thread.        mHandlerCheckers.add(new HandlerChecker(new Handler(Looper.getMainLooper()),                "main thread", DEFAULT_TIMEOUT));        // Add checker for shared UI thread.        mHandlerCheckers.add(new HandlerChecker(UiThread.getHandler(),                "ui thread", DEFAULT_TIMEOUT));        // And also check IO thread.        mHandlerCheckers.add(new HandlerChecker(IoThread.getHandler(),                "i/o thread", DEFAULT_TIMEOUT));        // And the display thread.        mHandlerCheckers.add(new HandlerChecker(DisplayThread.getHandler(),                "display thread", DEFAULT_TIMEOUT));        // Initialize monitor for Binder threads.        addMonitor(new BinderThreadMonitor());    }

可以看到这里的作用是初始化各个 HandlerChecker，并将他们添加到 mHandlerCheckers 这个 ArrayList 中

1.2.2 HandlerChecker(…)

Watchdog.java

    public final class HandlerChecker implements Runnable {        private final Handler mHandler;        private final String mName; // 线程名        private final long mWaitMax; // 最大等待时间        private final ArrayList<Monitor> mMonitors = new ArrayList<Monitor>(); // 包含的 Monitor        private boolean mCompleted; // 本轮 check 是否完成        private Monitor mCurrentMonitor; // 当前 check 的 Monitor        private long mStartTime; // 开始 check 的系统时间        HandlerChecker(Handler handler, String name, long waitMaxMillis) {            mHandler = handler;            mName = name;            mWaitMax = waitMaxMillis;            mCompleted = true;        }    }

可以看到仅仅是初始化了一些成员变量，各个成员的含义见注释

1.2.3 addMonitor()

Watchdog.java

    public void addMonitor(Monitor monitor) {        synchronized (this) {            if (isAlive()) {                throw new RuntimeException("Monitors can't be added once the Watchdog is running");            }            // 将 monitor 添加到 mMonitorChecker            mMonitorChecker.addMonitor(monitor);        }    }

这里的作用是将 new BinderThreadMonitor() 添加到 mMonitorChecker 中，也就是 fg 线程的 HandlerChecker 中，这个 Monitor 是用来 check binder 线程的，用来确保其他进程可以与 system_server 进程通信

1.3 watchdog.init(…)

Watchdog.java

    public void init(Context context, ActivityManagerService activity) {        mResolver = context.getContentResolver();        // AMS        mActivity = activity;        context.registerReceiver(new RebootRequestReceiver(),                new IntentFilter(Intent.ACTION_REBOOT),                android.Manifest.permission.REBOOT, null);    }

可以看到这里的作用主要是对 mResolver、mActivity 进行赋值，并且注册了一个 RebootRequestReceiver 来监听 ACTION_REBOOT 的广播

1.3.1 RebootRequestReceiver

Watchdog.java

    final class RebootRequestReceiver extends BroadcastReceiver {        @Override        public void onReceive(Context c, Intent intent) {            if (intent.getIntExtra("nowait", 0) != 0) {                rebootSystem("Received ACTION_REBOOT broadcast");                return;            }            Slog.w(TAG, "Unsupported ACTION_REBOOT broadcast: " + intent);        }    }    void rebootSystem(String reason) {        Slog.i(TAG, "Rebooting system because: " + reason);        IPowerManager pms = (IPowerManager)ServiceManager.getService(Context.POWER_SERVICE);        try {            pms.reboot(false, reason, false);        } catch (RemoteException ex) {        }    }

可以看到其是监听到广播并且条件符合的情况下通过 PMS 重启手机

1.4 Watchdog.getInstance().start()

这一步会调用 “watchdog” 线程的 run() 方法，下面我们来具体看一下 “watchdog” 线程是如何检测的

二、Watchdog 的运行

2.1 run()

Watchdog.java

    public void run() {        boolean waitedHalf = false;        while (true) {            final ArrayList<HandlerChecker> blockedCheckers; // 用于记录被 block 的 Checkers            final String subject;            final boolean allowRestart;            int debuggerWasConnected = 0;            synchronized (this) {                long timeout = CHECK_INTERVAL; // 正常模式下为 30s                for (int i=0; i<mHandlerCheckers.size(); i++) {                    HandlerChecker hc = mHandlerCheckers.get(i);                    // 第一步，对每个 HandlerChecker 执行 scheduleCheckLocked() 方法                    hc.scheduleCheckLocked();                }                if (debuggerWasConnected > 0) {                    debuggerWasConnected--;                }                // 第二步，等待 30s                long start = SystemClock.uptimeMillis();                while (timeout > 0) {                    if (Debug.isDebuggerConnected()) {                        debuggerWasConnected = 2;                    }                    try {                        wait(timeout);                    } catch (InterruptedException e) {                        Log.wtf(TAG, e);                    }                    if (Debug.isDebuggerConnected()) {                        debuggerWasConnected = 2;                    }                    timeout = CHECK_INTERVAL - (SystemClock.uptimeMillis() - start);                }                // 第三步，得悉 check 的结果                final int waitState = evaluateCheckerCompletionLocked();                if (waitState == COMPLETED) {                    // check 通过，reset                    waitedHalf = false;                    continue;                } else if (waitState == WAITING) {                    // ？没有搞清这里存在的意义                    continue;                } else if (waitState == WAITED_HALF) { //                     if (!waitedHalf) {                        // We've waited half the deadlock-detection interval.  Pull a stack                        // trace and wait another half.                        ArrayList<Integer> pids = new ArrayList<Integer>();                        pids.add(Process.myPid());                        ActivityManagerService.dumpStackTraces(true, pids, null, null,                                NATIVE_STACKS_OF_INTEREST);                        waitedHalf = true;                    }                    continue;                }                // block 超过 60s，获得被 block 的 Checkers 信息等                blockedCheckers = getBlockedCheckersLocked();                subject = describeCheckersLocked(blockedCheckers);                allowRestart = mAllowRestart;            }            // 第四步，走到这里意谓着系统很有可能 hung 住了            // First collect stack traces from all threads of the system process.            // Then kill this process so that the system will restart.            EventLog.writeEvent(EventLogTags.WATCHDOG, subject);            ArrayList<Integer> pids = new ArrayList<Integer>();            pids.add(Process.myPid());            if (mPhonePid > 0) pids.add(mPhonePid);            // Pass !waitedHalf so that just in case we somehow wind up here without having            // dumped the halfway stacks, we properly re-initialize the trace file.            final File stack = ActivityManagerService.dumpStackTraces(                    !waitedHalf, pids, null, null, NATIVE_STACKS_OF_INTEREST);            // Give some extra time to make sure the stack traces get written.            // The system's been hanging for a minute, another second or two won't hurt much.            SystemClock.sleep(2000);            // Pull our own kernel thread stacks as well if we're configured for that            if (RECORD_KERNEL_THREADS) {                dumpKernelStackTraces();            }            String tracesPath = SystemProperties.get("dalvik.vm.stack-trace-file", null);            String traceFileNameAmendment = "_SystemServer_WDT" + mTraceDateFormat.format(new Date());            if (tracesPath != null && tracesPath.length() != 0) {                File traceRenameFile = new File(tracesPath);                String newTracesPath;                int lpos = tracesPath.lastIndexOf (".");                if (-1 != lpos)                    newTracesPath = tracesPath.substring (0, lpos) + traceFileNameAmendment + tracesPath.substring (lpos);                else                    newTracesPath = tracesPath + traceFileNameAmendment;                traceRenameFile.renameTo(new File(newTracesPath));                tracesPath = newTracesPath;            }            final File newFd = new File(tracesPath);            // Try to add the error to the dropbox, but assuming that the ActivityManager            // itself may be deadlocked.  (which has happened, causing this statement to            // deadlock and the watchdog as a whole to be ineffective)            Thread dropboxThread = new Thread("watchdogWriteToDropbox") {                    public void run() {                        mActivity.addErrorToDropBox(                                "watchdog", null, "system_server", null, null,                                subject, null, newFd, null);                    }                };            dropboxThread.start();            try {                dropboxThread.join(2000);  // wait up to 2 seconds for it to return.            } catch (InterruptedException ignored) {}            // Trigger the kernel to dump all blocked threads, and backtraces on all CPUs to the kernel log            Slog.e(TAG, "Triggering SysRq for system_server watchdog");            doSysRq('w');            doSysRq('l');            // At times, when user space watchdog traces don't give an indication on            // which component held a lock, because of which other threads are blocked,            // (thereby causing Watchdog), crash the device to analyze RAM dumps            boolean crashOnWatchdog = SystemProperties                                        .getBoolean("persist.sys.crashOnWatchdog", false);            if (crashOnWatchdog) {                // wait until the above blocked threads be dumped into kernel log                SystemClock.sleep(3000);                // now try to crash the target                doSysRq('c');            }            IActivityController controller;            ...            // Only kill the process if the debugger is not attached.            if (...) {            } else if (!allowRestart) {                Slog.w(TAG, "Restart not allowed: Watchdog is *not* killing the system process");            } else {                Slog.w(TAG, "*** WATCHDOG KILLING SYSTEM PROCESS: " + subject);                for (int i=0; i<blockedCheckers.size(); i++) {                    Slog.w(TAG, blockedCheckers.get(i).getName() + " stack trace:");                    StackTraceElement[] stackTrace                            = blockedCheckers.get(i).getThread().getStackTrace();                    for (StackTraceElement element: stackTrace) {                        Slog.w(TAG, "    at " + element);                    }                }                Slog.w(TAG, "*** GOODBYE!");                Process.killProcess(Process.myPid());                System.exit(10);            }            waitedHalf = false;        }    }

可以看到其主要分为四步：

• 第一步，对每个 HandlerChecker 执行 scheduleCheckLocked() 方法
• 第二步，等待 30s
• 第三步，得悉 check 的结果，并分为 COMPLETED、WAITING、WAITED_HALF、OVERDUE 四种情况分别处理
• 第四步，对于 OVERDUE 情况的后续处理

下面，我们从一、三、四步来分别进行分析

2.2 scheduleCheckLocked()

Watchdog.java#HandlerChecker

        public void scheduleCheckLocked() {            if (mMonitors.size() == 0 && mHandler.getLooper().getQueue().isPolling()) {                // 如果 target looper 为 polling 状态，并且其 mMonitors 的 size 为 0                mCompleted = true;                return;            }            if (!mCompleted) {                // 正在进行 check 不需要重新安排                return;            }            // 1. 初始化变量            mCompleted = false; // 标志本轮 check 开始            mCurrentMonitor = null; // 正在 check 的 Monitor            mStartTime = SystemClock.uptimeMillis(); // 设置开始时间            // 2. 将 msg 放到 mHandler 的 msg 队列首            mHandler.postAtFrontOfQueue(this);        }

可以看到第二步将 msg 放到 mHandler 的 msg 队列首，这样 mHandler 在处理完当前的 msg 后，就会处理到这个 msg，会调用到 HandlerChecker 的 run() 方法。
如果当前线程中存在耗时较长的操作，就会导致在某次 Handler Check 的时候 msg 不能立刻执行，这就是对 Handler check 的原理。

2.2.1 HandlerChecker.run()

Watchdog.java#HandlerChecker

        public void run() {            final int size = mMonitors.size();            for (int i = 0 ; i < size ; i++) {                synchronized (Watchdog.this) {                    mCurrentMonitor = mMonitors.get(i);                }                mCurrentMonitor.monitor();            }            synchronized (Watchdog.this) {                mCompleted = true;                mCurrentMonitor = null;            }        }

可以看到，这里是对 HandlerChecker 中的每个 Monitor 执行 monitor() 方法，monitor() 实际上是一个拿锁操作，如果有其他线程一直持锁，譬如 “ActivityManager” 线程一直持着 AMS 的 this 锁（Monitor 要事先添加到 mMonitors 中，见后面），那么 monitor() 将一直被 block 无法返回，导致超时，这就是 Monitor Check 的原理。

2.3 获得 check 结果并处理

2.3.1 evaluateCheckerCompletionLocked()

Watchdog.java

    private int evaluateCheckerCompletionLocked() {        int state = COMPLETED;        for (int i=0; i<mHandlerCheckers.size(); i++) {            HandlerChecker hc = mHandlerCheckers.get(i);            state = Math.max(state, hc.getCompletionStateLocked());        }        return state;    }

可以看到其是遍历所有的 HandlerChecker，并取出它们数值最大的状态，状态包含四种，分别是：COMPLETED = 0、WAITING = 1、WAITED_HALF = 2、OVERDUE = 3

2.3.2 getCompletionStateLocked()

Watchdog.java#HandlerChecker

        public int getCompletionStateLocked() {            if (mCompleted) { // 已经完成 check 则返回 COMPLETED                return COMPLETED;            } else {                 long latency = SystemClock.uptimeMillis() - mStartTime;                if (latency < mWaitMax/2) {                    return WAITING;                } else if (latency < mWaitMax) {                    return WAITED_HALF;                }            }            return OVERDUE;        }

分两种情况返回状态：

• 已经完成：返回 COMPLETED
• 未完成：根据开始时长返回状态，WAITING、WAITED_HALF、OVERDUE 分别对应小于 30s、小于 60s、大于等于 60s
  

2.3.3 结果的处理

Watchdog.run()

                if (waitState == COMPLETED) {                    // 1. COMPLETED 则恢复 waitedHalf 初始值，开始下轮检测                    waitedHalf = false;                    continue;                } else if (waitState == WAITING) {                    // 2. WAITING 直接再次检测                    continue;                } else if (waitState == WAITED_HALF) {                    if (!waitedHalf) {                        // 3. 如果第一次 WAITED_HALF 状态，则 dump traces 并且再次经历一轮检测查看状态                        ArrayList<Integer> pids = new ArrayList<Integer>();                        pids.add(Process.myPid());                        ActivityManagerService.dumpStackTraces(true, pids, null, null,                                NATIVE_STACKS_OF_INTEREST);                        waitedHalf = true;                    }                    continue;                }                // 4. OVERDUE，一般相当于连续的第二次 WAITED_HALF                blockedCheckers = getBlockedCheckersLocked();                subject = describeCheckersLocked(blockedCheckers);                allowRestart = mAllowRestart;

上面的注释列出了对于四种状态分别是如何处理的，注意第四种状态 OVERDUE 就是通常所说的 Watchdog 超时了，后面还需对其进行更多处理

2.4 对于 OVERDUE 情况的后续处理

2.4.1 getBlockedCheckersLocked()

Watchdog.java

    private ArrayList<HandlerChecker> getBlockedCheckersLocked() {        ArrayList<HandlerChecker> checkers = new ArrayList<HandlerChecker>();        for (int i=0; i<mHandlerCheckers.size(); i++) {            HandlerChecker hc = mHandlerCheckers.get(i);            if (hc.isOverdueLocked()) {                checkers.add(hc);            }        }        return checkers;    }

返回处于 OVERDUE 状态的 HandlerChecker 的 ArrayList

2.4.2 describeCheckersLocked(…)

Watchdog.java

    private String describeCheckersLocked(ArrayList<HandlerChecker> checkers) {        StringBuilder builder = new StringBuilder(128);        for (int i=0; i<checkers.size(); i++) {            if (builder.length() > 0) {                builder.append(", ");            }            builder.append(checkers.get(i).describeBlockedStateLocked());        }        return builder.toString();    }

Watchdog.java#HandlerChecker

        public String describeBlockedStateLocked() {            if (mCurrentMonitor == null) {                return "Blocked in handler on " + mName + " (" + getThread().getName() + ")";            } else {                return "Blocked in monitor " + mCurrentMonitor.getClass().getName()                        + " on " + mName + " (" + getThread().getName() + ")";            }        }

可以看到这里是将每个 Blocked 的 checker 的信息拼在一起，每个 Blocked 的 checker 的信息是由 describeBlockedStateLocked() 方法来获得的，主要分为两种情况：

• 没有 Monitor，那就是 Blocked 在 handler 中，为 “Blocked in handler on ThreadXXX”
• 有 Monitor，说明 Blocked 在 Monitor 的 check 中，为 “Blocked in monitor MonitorXXX on ThreadXXX”

并且这些信息会用于后面 log 的打印

2.4.3 AMS.dumpStackTraces(…)

Watchdog.java

    public static File dumpStackTraces(boolean clearTraces, ArrayList<Integer> firstPids,            ProcessCpuTracker processCpuTracker, SparseArray<Boolean> lastPids, String[] nativeProcs) {        // 默认情况下为 /data/anr/traces.txt        String tracesPath = SystemProperties.get("dalvik.vm.stack-trace-file", null);        if (tracesPath == null || tracesPath.length() == 0) {            return null;        }        File tracesFile = new File(tracesPath);        try {            File tracesDir = tracesFile.getParentFile();            if (!tracesDir.exists()) {                tracesDir.mkdirs();                if (!SELinux.restorecon(tracesDir)) {                    return null;                }            }            FileUtils.setPermissions(tracesDir.getPath(), 0775, -1, -1);  // drwxrwxr-x            // 如果需要清理并且文件存在则删除存在文件            if (clearTraces && tracesFile.exists()) tracesFile.delete();            tracesFile.createNewFile();            FileUtils.setPermissions(tracesFile.getPath(), 0666, -1, -1); // -rw-rw-rw-        } catch (IOException e) {            Slog.w(TAG, "Unable to prepare ANR traces file: " + tracesPath, e);            return null;        }        // 写入 traces 信息        dumpStackTraces(tracesPath, firstPids, processCpuTracker, lastPids, nativeProcs);        return tracesFile;    }

可以看到这里主要是初始化目录和文件等，可以通过设置 clearTraces 来决定是否清除之前的 traces，这就是 Watchdog traces 文件中可以有两个时间点的 traces 的原因。

2.4.4 dumpKernelStackTraces()

Watchdog.java

    private File dumpKernelStackTraces() {        String tracesPath = SystemProperties.get("dalvik.vm.stack-trace-file", null);        if (tracesPath == null || tracesPath.length() == 0) {            return null;        }        native_dumpKernelStacks(tracesPath);        return new File(tracesPath);    }

通过调用 native_dumpKernelStacks(tracesPath) 来 dump kernel traces，即下面的方法
android_server_Watchdog.cpp

static void dumpKernelStacks(JNIEnv* env, jobject clazz, jstring pathStr) {    ...    int outFd = open(path, O_WRONLY | O_APPEND | O_CREAT,        S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP|S_IROTH|S_IWOTH);    if (outFd < 0) {        ALOGE("Unable to open stack dump file: %d (%s)", errno, strerror(errno));        goto done;    }    snprintf(buf, sizeof(buf), "\n----- begin pid %d kernel stacks -----\n", getpid());    write(outFd, buf, strlen(buf));    // look up the list of all threads in this process    snprintf(buf, sizeof(buf), "/proc/%d/task", getpid());    taskdir = opendir(buf);    if (taskdir != NULL) {        struct dirent * ent;        while ((ent = readdir(taskdir)) != NULL) {            int tid = atoi(ent->d_name);            if (tid > 0 && tid <= 65535) {                // dump each stack trace                dumpOneStack(tid, outFd);            }        }        closedir(taskdir);    }    ...}

可以看出这里是通过 /proc/%d/task 节点获取进程的所有线程信息，然后再通过 dumpOneStack 方法 dump 每个线程的 stack
android_server_Watchdog.cpp

static void dumpOneStack(int tid, int outFd) {    char buf[64];    snprintf(buf, sizeof(buf), "/proc/%d/stack", tid);    int stackFd = open(buf, O_RDONLY);    if (stackFd >= 0) {        // header for readability        strncat(buf, ":\n", sizeof(buf) - strlen(buf) - 1);        write(outFd, buf, strlen(buf));        // copy the stack dump text        int nBytes;        while ((nBytes = read(stackFd, buf, sizeof(buf))) > 0) {            write(outFd, buf, nBytes);        }        write(outFd, "\n", 1);        close(stackFd);    } else {        ALOGE("Unable to open stack of tid %d : %d (%s)", tid, errno, strerror(errno));    }}

通过读取 /proc/%d/stack 节点来 dump 每个线程的 kernel stack

2.4.5 doSysRq(char c)

Watchdog.java

    private void doSysRq(char c) {        try {            FileWriter sysrq_trigger = new FileWriter("/proc/sysrq-trigger");            sysrq_trigger.write(c);            sysrq_trigger.close();        } catch (IOException e) {            Slog.w(TAG, "Failed to write to /proc/sysrq-trigger", e);        }    }

通过向节点 /proc/sysrq-trigger 写入字符，触发 kernel 操作

2.4.6 Process.killProcess()

不详细介绍了

三、总结

Watchdog 是一个运行在 system_server 进程的名为 “watchdog” 的线程，可以看到：

• Watchdog 运行过程中，当阻塞时间超过1分钟则触发一次 watchdog，会杀死 system_server，触发上层重启
• Watchdog 的检测主要分为两类：对于 Handler 处理时长的检测和对于 Monitor 的检测（即线程持锁时间过长）
• 默认情况下，发生 Watchdog 后，会在 /data/anr 目录下生成 “traces__SystemServer_WDT时间戳XXX.txt” 的 traces 文件，其是由 dump traces 的文件 “traces.txt” 重命名而来的
• 通过 AMS.dumpStackTraces 输出 system_server 进程和 native 进程的 traces
• 通过 dumpKernelStackTraces 输出system_server 进程中所有线程的 kernel stack：
  
  • 通过 /proc/%d/task 节点获取进程的所有线程信息
  • 通过读取 "/proc/%d/stack" 节点来 dump 每个线程的 kernel stack
• 杀掉 system_server，进而触发 zygote 进程自杀，从而重启上层 framework