Oracle EBS R12 - opmn.xml少写一个M导致oacore启动失败

Oracle EBS R12 - opmn.xml少写一个M导致oacore启动失败

VER:
Application: EBS R12.0.6
Database: Oracle 10gR2


打完patch启动应用时发现oacore服务启动失败.

bash-3.2$ adstrtal.sh apps/apps
...
****************************************************


Executing service control script:
/dabcd/inst/apps/DABCD_vmohsabcd013/admin/scripts/adoacorectl.sh start
script returned:
****************************************************

You are running adoacorectl.sh version 120.11.12000000.3

Starting OPMN managed OACORE OC4J instance  ...

adoacorectl.sh: exiting with status 204

adoacorectl.sh: check the logfile /dabcd/mtlog/DABCD_vmohsabcd013/logs/appl/admin/log/adoacorectl.txt for more informa  tion ...


.end std out.

.end err out.

****************************************************
...


尝试单独重启opmn来启动oacore还是不行,检查oacore日志后发现了"Too small initial heap"的信息.
bash-3.2$ adopmnctl.sh stopall

You are running adopmnctl.sh version 120.4.12000000.3

opmnctl: stopping opmn and all managed processes...
================================================================================
opmn id=vmohsabcd013:13725
    1 of 2 processes stopped.

ias-instance id=DABCD_vmohsabcd013.vmohsabcd013.oracleoutsourcing.com
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ias-component/process-type/process-set:
    default_group/oafm/default_group/

Error
--> Process (index=1,uid=1362972063,pid=12284)
    time out while waiting for a managed process to stop
    Log:
    /dabcd/mtlog/DABCD_vmohsabcd013/logs/ora/10.1.3/opmn/default_group~oafm~default_group~1.log

opmnctl: graceful stop of processes failed, trying forceful shutdown...

adopmnctl.sh: exiting with status 0

adopmnctl.sh: check the logfile /dabcd/mtlog/DABCD_vmohsabcd013/logs/appl/admin/log/adopmnctl.txt for more information   ...

bash-3.2$
bash-3.2$ adopmnctl.sh startall

You are running adopmnctl.sh version 120.4.12000000.3

opmnctl: starting opmn and all managed processes...
================================================================================
opmn id=vmohsabcd013:13725
    2 of 3 processes started.

ias-instance id=DABCD_vmohsabcd013.vmohsabcd013.oracleoutsourcing.com
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ias-component/process-type/process-set:
    default_group/oacore/default_group/

Error
--> Process (index=1,uid=1362972066,pid=14654)
    failed to start a managed process after the maximum retry limit
    Log:
    /dabcd/mtlog/DABCD_vmohsabcd013/logs/ora/10.1.3/opmn/default_group~oacore~default_group~1.log


adopmnctl.sh: exiting with status 0

adopmnctl.sh: check the logfile /dabcd/mtlog/DABCDI_vmohsabcd013/logs/appl/admin/log/adopmnctl.txt for more information   ...

bash-3.2$ adoacorectl.sh status

You are running adoacorectl.sh version 120.11.12000000.3

Checking status of OPMN managed OACORE OC4J instance ...

Processes in Instance: DABCD_vmohsabcd013.vmohsabcd013.oracleoutsourcing.com
---------------------------------+--------------------+---------+---------
ias-component                    | process-type       |     pid | status
---------------------------------+--------------------+---------+---------
OC4JGroup:default_group          | OC4J:oafm          |   14586 | Alive
OC4JGroup:default_group          | OC4J:oacore        |     N/A | Down
HTTP_Server                      | HTTP_Server        |   14584 | Alive


adoacorectl.sh: exiting with status 0

adoacorectl.sh: check the logfile /dabcd/mtlog/DABCD_vmohsabcd013/logs/appl/admin/log/adoacorectl.txt for more informa  tion ...

bash-3.2$ view /dabcd/mtlog/DABCD_vmohsabcd013/logs/ora/10.1.3/opmn/default_group~oacore~default_group~1.log
...
--------
13/11/07 03:22:54 Start process
--------
Error occurred during initialization of VM
Too small initial heap

--------
13/11/07 03:25:37 Start process
--------
Error occurred during initialization of VM
Too small initial heap

...


根据oacore日志检查opmn参数发现有一项java参数(-Xms1024)少了一个M(第114行).
由于本机是开发环境,物理内存小于生产环境,可以看到当初克隆的人为适应内存减少还修改过jvm参数
(-Xmx2048M=>-Xmx1024M,-Xms1538M=>-Xms1024M),但是改的过程中却漏写了一个M.
bash-3.2$ head -n 30 $ADMIN_SCRIPTS_HOME/adopmnctl.sh
#!/bin/sh
# $Header: adopmnctl_sh_1013.tmp 120.4.12000000.3 2007/07/02 04:48:41 sbandla ship $
# *===========================================================================+
# |  Copyright (c) 1996 Oracle Corporation, Redwood Shores, California, USA   |
# |                        All rights reserved                                |
# |                       Applications  Division                              |
# +===========================================================================+
# |
# | FILENAME
# |   adopmnctl.sh
# |
# |   This script is used to start, stop and check the status of OPMN.
# | DESCRIPTION
# |
# | USAGE
# |   adopmnctl.sh {start|stop|status}
# |
# | PLATFORM
# |   Unix Generic
# |
# | NOTES
# |   This script is provided as a wrapper on top of the opmnctl script shipped
# |   with Oracle Application Server. To use additional functionalities provided
# |   by opmnctl, use the script from /dabcd/inst/apps/DABCD_vmohsabcd013/ora/10.1.3/opmn/bin
# |
# | HISTORY
# |
# +===========================================================================+
#
# ###############################################################
bash-3.2$
bash-3.2$ cd /dabcd/inst/apps/DABCDI_vmohsabcd013/ora/10.1.3/opmn/bin
bash-3.2$ pwd
/dabcd/inst/apps/DABCD_vmohsabcd013/ora/10.1.3/opmn/bin
bash-3.2$ ls -l
total 39
-rwx------ 1 apdabcd aadabcd 38939 Nov  1 04:47 opmnctl
bash-3.2$ cd ..
bash-3.2$ ls -l
total 5
drwxr-xr-x 2 apdabcd aadabcd  3 Apr 13  2012 bin
drwxr-xr-x 2 apdabcd aadabcd 10 Nov  7 03:40 conf
drwxr-xr-x 5 apdabcd aadabcd  5 Dec 24  2012 logs
bash-3.2$ cd conf/
bash-3.2$ ls -l
total 65
-rw-r--r-- 1 apdabcd aadabcd     0 Nov  7 03:40 devnull
-rw-r--r-- 1 apdabcd aadabcd  1409 Nov  7 03:40 javacache.log
-rw-r----- 1 apdabcd aadabcd 11836 Nov  7 03:34 opmn.xml
-rw-r----- 1 apdabcd aadabcd 11835 Nov  7 03:34 opmn.xml-07nov2013
-rw-r----- 1 apdabcd aadabcd 11835 Nov  1 04:14 opmn.xml-1nov
-rw-r----- 1 apdabcd aadabcd 11896 Jan 12  2013 opmn.xml_jan_12_13
-rw-r----- 1 apdabcd aadabcd 11880 Oct 29 08:09 opmn.xml-old
bash-3.2$
bash-3.2$ diff opmn.xml opmn.xml-old
114c114
<                      <data id="java-options" value="-server -verbose:gc -Xmx1024M -Xms1024 -XX:MaxPermSize=256M -XX:NeTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Dcom.sun.management.jmxremote -Djava.security.policy=$ORACLE_HOME/j2ee/adless=true -Dhttp.webdir.enable=false -Doracle.security.jazn.config=/dabcd/inst/apps/DABCD_vmohsabcd013/ora/10.1.3/j2ignoreCommaInCookiesNamed=X_NoMatchingCookies"/>
---
>                      <data id="java-options" value="-server -verbose:gc -Xmx2048M -Xms1538M -XX:MaxPermSize=256M -XX:NeTLAB -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=4 -Dcom.sun.management.jmxremote -Djava.security.poa2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Doracle.security.jazn.config=/dabcd/inst/apps/DABCD_vmojazn.xml -Dhttp.cookie.ignoreCommaInCookiesNamed=X_NoMatchingCookies"/>
119c119
<                      <data id="java-options" value="-server -verbose:gc -Xmx1024M -Xms1024M -XX:MaxPermSize=256M -XX:NeTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy -Djle=false"/>
---
>                      <data id="java-options" value="-server -verbose:gc -Xmx2048M -Xms1538M -XX:MaxPermSize=256M -XX:NeTLAB -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=4 -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/crue -Dhttp.webdir.enable=false"/>
131c131
<                <process-set id="default_group" numprocs="1"/>
---
>                <process-set id="default_group" numprocs="4"/>
bash-3.2$

也可采用如下命令重现上述错误,参见Note: Too Small Initial Heap For New Size Specified (Doc ID 885093.1).
java -server -XX:+AggressiveHeap -ms1280m -mx1280m -version
java -server -XX:+AggressiveHeap -ms1280 -mx1280m -version

bash-3.2$ java -server -XX:+AggressiveHeap -ms1280m -mx1280m -version
java version "1.6.0_10"
Java(TM) SE Runtime Environment (build 1.6.0_10-b33)
Java HotSpot(TM) Server VM (build 11.0-b15, mixed mode)
bash-3.2$$ java -server -XX:+AggressiveHeap -ms1280 -mx1280m -version
Error occurred during initialization of VM
Too small initial heap
bash-3.2$



把M加上后发现单独启动oacore服务已经没有问题.
bash-3.2$ cp opmn.xml opmn.xml-07nov2013
bash-3.2$ vi opmn.xml
bash-3.2$ diff opmn.xml opmn.xml-07nov2013
114c114
<                      <data id="java-options" value="-server -verbose:gc -Xmx1024M -Xms1024M -XX:MaxPermSize=256M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Dcom.sun.management.jmxremote -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Doracle.security.jazn.config=/dabcd/inst/apps/DABCD_vmohsabcd013/ora/10.1.3/j2ee/oacore/config/jazn.xml -Dhttp.cookie.ignoreCommaInCookiesNamed=X_NoMatchingCookies"/>
---
>                      <data id="java-options" value="-server -verbose:gc -Xmx1024M -Xms1024 -XX:MaxPermSize=256M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Dcom.sun.management.jmxremote -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Doracle.security.jazn.config=/dabcd/inst/apps/DABCD_vmohsabcd013/ora/10.1.3/j2ee/oacore/config/jazn.xml -Dhttp.cookie.ignoreCommaInCookiesNamed=X_NoMatchingCookies"/>
bash-3.2$
bash-3.2$ adoacorectl.sh status

You are running adoacorectl.sh version 120.11.12000000.3

Checking status of OPMN managed OACORE OC4J instance ...
Unable to connect to opmn.
Opmn may not be up.

adoacorectl.sh: exiting with status 0

adoacorectl.sh: check the logfile /dabcd/mtlog/DABCD_vmohsabcd013/logs/appl/admin/log/adoacorectl.txt for more informa

bash-3.2$ adoacorectl.sh start

You are running adoacorectl.sh version 120.11.12000000.3

Starting OPMN managed OACORE OC4J instance  ...

adoacorectl.sh: exiting with status 0

adoacorectl.sh: check the logfile /dabcd/mtlog/DABCD_vmohsabcd013/logs/appl/admin/log/adoacorectl.txt for more informa

bash-3.2$ adoacorectl.sh status

You are running adoacorectl.sh version 120.11.12000000.3

Checking status of OPMN managed OACORE OC4J instance ...

Processes in Instance: DABCD_vmohsabcd013.vmohsabcd013.oracleoutsourcing.com
---------------------------------+--------------------+---------+---------
ias-component                    | process-type       |     pid | status
---------------------------------+--------------------+---------+---------
OC4JGroup:default_group          | OC4J:oafm          |     N/A | Down
OC4JGroup:default_group          | OC4J:oacore        |    9260 | Alive
HTTP_Server                      | HTTP_Server        |     N/A | Down


adoacorectl.sh: exiting with status 0

adoacorectl.sh: check the logfile /dabcd/mtlog/DABCD_vmohsabcd013/logs/appl/admin/log/adoacorectl.txt for more informa

bash-3.2$


最后停止MT再重启MT也没有报任何错误了.
bash-3.2$ adstrtal.sh apps/apps