Domino Calendar REST API性能调优 之 系统崩溃篇

    我们对Domino Calendar REST API性能测试进行加压，当用户数增加到3000的时候，Domino Server频繁崩溃，Server崩溃后的错误NSD片段见下面Log片段。从历史经验来看一定是我们的REST代码的不当操作。可为什么当用户数是1000,2000的时候不会崩溃，而当用户数上3000后Server就会崩溃？这个原因一定是和线程数，内存使用等相关。

    将此日志抛给了Domino JAVA API专家，专家的判断是一定是内存泄露导致的Server崩溃，那就是我们的REST JAVA代码导致的内存泄露。等等，JAVA代码导致的内存泄露？JVM不是有垃圾自动回收机制吗？为什么你的JAVA代码会导致内存泄露？读者有这个疑问是正常的，我们的Domino JAVA API用的是JNI调用底层C， C++代码，Domino JAVA API的对象用完后需要程序员主动进行recycle操作来通知JVM或系统进行该对象的内存回收，详情请见我的另外一篇博客《Java JNI介绍及JNI在Domino中的使用》。找到问题的根源以后就要扫描我们的代码了，终于发现在某个地方我们没有调用Domino JAVA API对象的recycle方法，这样当用户量上来后内存持续地泄露，最终导致内存耗尽，Server崩溃。

    代码中典型的recycle代码如下：

        try {            Session session = _database.getParent();            calendar = session.getCalendar(_database);            if (start != null) {                dtStart = session.createDateTime(start);            }                            if ( since != null ) {                dtSince = session.createDateTime(since);            }            // Get a list of new invitations            List<String> iCalendarList = null;            Vector<Object> list = calendar.getNewInvitations(dtStart, dtSince);            .......        }        catch (NotesException e) {            throw new StoreException("Error getting new invitations", mapError(e.id), e); // $NLX-NotesEventStore.Errorgettingnewinvitations-1$        }        finally {            // recycle            BackendUtil.safeRecycle(dtStart);            BackendUtil.safeRecycle(dtSince);            BackendUtil.safeRecycle(calendar);        }

BackendUtil类如下所示：

public class BackendUtil {    /**     * Safely recycle a backend object.     *      * @param obj     */    public static void safeRecycle(Base obj) {                if ( obj != null ) {            try {                obj.recycle();            }            catch (NotesException e) {                // Ignore exceptions inside recycle            }        }    }        /**     * Safely recycle a vector of backend objects.     *      * @param list     */    public static void safeRecycle(Vector list) {        if ( list != null ) {            Iterator iterator = list.iterator();            while ( iterator.hasNext() ) {                Object obj = iterator.next();                if ( obj instanceof Base ) {                    safeRecycle((Base)obj);                }            }        }    }    /**     * Gets a trusted session.     *      * @param session     * @return     */    public static Session getTrustedSession(Session session) {        Session trustedSession = null;                try {            ITrustProvider provider = ProviderFactory.getTrustProvider();            if ( provider != null ) {                trustedSession = provider.getTrustedSession(session);            }        }        catch (Throwable e) {            // Ignore all exceptions        }                return trustedSession;    }}

Host Name       : User Name       : AdministratorDate            : Tue Sep 24 00:37:45 2013Windows Dir     : C:\WindowsArguments       : "C:\Program Files\IBM\Domino\nsd.exe" -dumpandkill -termstatus 1 -nomemcheck -shutdownhang -crashpid 3724 -crashtid 3644 -runtime 600NSD Version     : 9.0.10.3261 (Build V901_09192013)OS Version      : Windows/2008 R2 6.1 [64-bit] (Build 7601), PlatID=2, Service Pack 1 (8 Processors)Build time      : Fri Sep 20 01:17:55 2013Latest file mod : Thu Sep 19 18:06:30 2013Domino Version   : Build V901_09192013 (64-bit server)Keyview Version : 10.16.0.0 <@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>Section: System Data -> OS Process Table (Time 00:37:46)<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@><@@ ------ System Data -> Active Users (Time 00:37:46) ------ @@> UID     Name   500  Administrator   18  SYSTEM   20  NETWORK SERVICE   19  LOCAL SERVICE<@@ ------ System Data -> Processes (Time 00:37:46) ------ @@>  INFO       PID     PPID      UID   STIME          COMMAND               0        0       18              ??? [[System Process]:  0000]             120        4       18   08/29 03:07:11 [    smss:  0120]             174      16c       18   08/29 03:07:12 [   csrss:  0174]             1a0      16c       18   08/29 03:07:12 [ wininit:  01a0]             20c      1a0       18   08/29 03:07:13   [services:  020c]             280      20c       18   08/29 03:07:14     [ svchost:  0280]             2d0      20c       20   08/29 03:07:14     [ svchost:  02d0]             328      20c       19   08/29 03:07:14     [ svchost:  0328]             350      20c       18   08/29 03:07:14     [ svchost:  0350]             38c      20c       19   08/29 03:07:14     [ svchost:  038c]             3bc      20c       18   08/29 03:07:15     [ svchost:  03bc]             8a4      3bc      500   08/29 04:56:51       [     Dwm:  08a4]             3e8      20c       20   08/29 03:07:15     [ svchost:  03e8]             188      20c       19   08/29 03:07:15     [ svchost:  0188]             408      20c       18   08/29 03:07:15     [ spoolsv:  0408]             444      20c       19   08/29 03:07:15     [ svchost:  0444]             498      20c       18   08/29 03:07:16     [vmtoolsd:  0498]             5d8      20c       20   08/29 03:07:16     [ svchost:  05d8]             168      5d8      500   08/29 04:56:48       [ rdpclip:  0168]             630      20c       20   08/29 03:07:16     [ svchost:  0630]             750      20c       18   08/29 03:07:16     [ dllhost:  0750]             7a8      20c       20   08/29 03:07:17     [   msdtc:  07a8]             31c      20c       19   08/29 03:09:16     [ svchost:  031c]             288      20c       20   08/29 03:09:16     [  sppsvc:  0288]             728      20c      500   08/29 04:56:48     [taskhost:  0728]             214      1a0       18   08/29 03:07:13   [   lsass:  0214]             21c      1a0       18   08/29 03:07:13   [     lsm:  021c]             1b4      1ac       18   08/29 03:07:12 [   csrss:  01b4]             1d0      1ac       18   08/29 03:07:12 [winlogon:  01d0]             320      1d0       18   08/29 03:07:14   [ LogonUI:  0320]             2c0      66c       18   08/29 04:56:47 [   csrss:  02c0]             be0      2c0      500   09/06 01:21:36   [ conhost:  0be0]            133c      2c0      500   09/23 09:02:30   [ conhost:  133c]             ec0      2c0      500   09/24 00:37:45   [ conhost:  0ec0]             714      66c       18   08/29 04:56:47 [winlogon:  0714]             8f0      89c      500   08/29 04:56:51 [Explorer:  08f0]             968      8f0      500   08/29 04:56:52   [VMwareTray:  0968]             970      8f0      500   08/29 04:56:52   [vmtoolsd:  0970]             ce8      8f0      500   09/06 01:21:36   [     cmd:  0ce8]     ->      e8c      8f0      500   09/23 09:02:30   [ nserver:  0e8c]             d9c      e8c      500   09/23 09:02:30     [nsrvwrap:  0d9c]     ->     1378      e8c      500   09/23 09:02:37     [   nHTTP:  1378]             a88      e8c      500   09/24 00:37:45     [     nsd:  0a88]<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>Section: System Data -> Notes Process Summary (Time 00:37:46)<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@><@@ ------ Instances -> Instance1 (current) (Time 00:37:46) ------ @@>                Ini File   : C:\Program Files\IBM\Domino\notes.ini                Data Dir   : C:\Program Files\IBM\Domino\data                Prog Dir   : C:\Program Files\IBM\Domino                Notes Build: Build V901_09192013 September 19, 2013                         Proc List:                  C:\Program Files\IBM\Domino\nserver.exe (e8c)                  C:\Program Files\IBM\Domino\nHTTP.EXE (1378)<@@ ------ System Data -> Memory Usage (Time 00:37:46) ------ @@>Total Physical Memory: 16.0GAvail Physical Memory: 12.3GMemory Usage        : 23%Total Paging File   : 32.0GAvail Paging File   : 28.0GTotal Virtual Memory:  8.0TAvail Virtual Memory:  8.0TAvail Extended Virtual Memory:  0.0KMin. File Cache Size: 1024 KBMax. File Cache Size: 5032856 KBMin. Size Limit: DisabledMax. Size Limit: Enabled