Server(Admin) start is failing in WLST as outofmemory error permgen space for weblogic 12c

OutOfMemory Causes and First Aid Steps?

In response to the Comment: http://middlewaremagic.com/weblogic/?p=4456#comment-2026 We developed a Post on Java Heap. Thanks to “Swordfish” for querying us very important topic.

.

In most of the environments we usually see some kind of Memory related issues like OutOfMemory/Server Slowless kind of things. most of the times it is related to the in accurate tuning of the JVM and some times it happens due to the Application Frameworks Bug/ In accurate tuning configuration or it may happen due to the Application Code as well ..like Object Leakin in the Application code.

.

NOTE: The pre requisit for this Post is that you are already aware of different Memory Spaces available as part of the Java Process..If Not then Please quickly review: http://middlewaremagic.com/weblogic/?p=4456

.

Here we are going to see that what causes the OutOfMemory issues  and Why it happens along with some basic First Aid Steps to debug this kind of  issues.

What is OutOfMemory?

An OutOfMemory is a condition in which there is not enough space left for allocating required space for the new objects or libraries or native codes. OutOfMemory can be divided in tow main categories:

1). OutOfMemory in Java Heap:

This happens when the JVM is not able to allocate the required memory space for a Java Object. There may be many reasons behind this…like

Point-1). Very Less Heap Size allocation. Means setting the MaxHeapSize (-Xmx) parameter to a very less value.

.

Point-2). The Leaking of Objects. Either the Application is not unreferencing the unused Objects or the Third part frameworks (Hibernate/Spring/Seam…etc) might not be releasing the references of the objects due to some inaccurate configurations.

.

Point-3). In Many cases it may be the reason that Application codes are getting the JDBC connections objects from the DataSource are not being released back to the Connection Pool.

.

Point-4). Garbage Collection strategy may be in correct according to the environmental/application requirements.

.

Point-5). In-accurate setting of Application/Frameworks Cache.

Example:

Exception in thread "Thread-10" java.lang.OutOfMemoryError: Java heap spaceat java.util.Arrays.copyOf(Arrays.java:2882)at java.lang.AbstractStringBuilder.expandCapacity(Abs tractStringBuilder.java:100)at java.lang.AbstractStringBuilder.append(AbstractStr ingBuilder.java:390)at java.lang.StringBuilder.append(StringBuilder.java: 119)at java.lang.Throwable.toString(Throwable.java:344)

2). Native OutOfMemory:

Native OutOfMemory is a scenario when the JVM is not able to allocate the required Native Libraries and JNI Codes in the memory.

Native Memory is an area which is usually used by the JVM for it’s internal operations and to execute the JNI codes. The JVM Uses Native Memory for Code Optimization and for loading the classes and libraries along with the intermediate code generation.

The Size of the Native Memory depends on the Architecture of the Operating System and the amount of memory which is already commited to the Java Heap. Native memory is an Process Area where the JNI codes gets loaded or JVM Libraries gets loaded or the native Performance packs and the Proxy Modules gets loaded…

Native OutOfMemory can happen due to the following main reasons:

.

Point-1). Setting very small StackSize (-Xss). StackSize is a memory area which is allocated to individual threads where they can place their thread local objects/variables.

.

Point-2). Usually it may be seen because of Tuxedos incorrect setting. WebLogic Tuxedo Connectors allows the interoperability between the Java Applications deployed on WebLogic Server and the Native Services deployed on Tuxedo Servers. Because Tuxedos uses JNI code intensively.

.

Point-3). Less RAM or Swap Space.

Example: For details on this kind of error Please refer to: http://middlewaremagic.com/weblogic/?p=422

.

Point-4). Usually it may occur is our Application is using a very large number of JSPs in our application. The JSPs need to be converted into the Java Code and then need to be compiled. Which reqires DTD and Custom Tag Library resolution as well. Which usually consumes more native memory.

Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native threadat java.lang.Thread.start0(Native Method)at java.lang.Thread.start(Thread.java:574)at TestXss.main(TestXss.java:18)

3).  OutOfMemory in PermGen Space:

Permanent Generation is a Non-Heap Memory Area inside the JVM Space. Manytimes we see OutOfMemory in this Area. PermGen Area is NOT present in JRockit JVMs. For more details on this area please refer to:http://middlewaremagic.com/weblogic/?p=4456.

.

The PermGen Area is measured independently from the other generations because this is the place where the JVM allocates Classes, Class Structures, Methods and Reflection Objects. PermGen is a Non-Heap Area.It means we DO NOT count the PermGen Area as part of Java Heap.

The OutOfMemory in PermGen Area can be seen because of the following main reasons:

Point-1).Deploying and Redeploying a very Large Application which has many Classes inside it.

.

Point-2). If an Application is getting deployed/Updated/redeployed repeatedly using the Auto Deployment feature of the Containers. In that case the Classes belonging to the application stays un cleaned and remains in the PermGen Area without Class Garbage Collection.

.

Point-3). If  ”-noclassgc” Java Option is added while starting the Server. In that case the Classes instances which are not required will not be Garbage collected.

.

Point-4). Very Less Space for allocated the “=XX:MaxPermGen”

Example: you can see following kind of Trace in the Server/Stdout Logs:

<Notice> <Security> <BEA-090171> <Loading the identity certificate and private key stored under the alias DemoIdentity from the jks keystore file D:\ORACLE\MIDDLE~1\WLSERV~1.3\server\lib\DemoIdentity.jks.>Exception in thread "[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen spaceat java.lang.ClassLoader.defineClass1(Native Method)at java.lang.ClassLoader.defineClass(ClassLoader.java:621)at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)at java.net.URLClassLoader.access$000(URLClassLoader.java:56)at java.net.URLClassLoader$1.run(URLClassLoader.java:195)at java.security.AccessController.doPrivileged(Native Method)at java.net.URLClassLoader.findClass(URLClassLoader.java:188)at java.lang.ClassLoader.loadClass(ClassLoader.java:307)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)at java.lang.ClassLoader.loadClass(ClassLoader.java:252)at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

What to do in case of OutOfMemory In JavaHeap?

Whenever we see an OutOfMemory in the server log or in the stdout of the server. We must try to do the following things as first aid steps:

.

Point-1). If possible enable the following JAVA_OPTIONS in the server start Scripts to get the informations of the Garbage Collection status.

-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails  -Xloggc:/opt/app/GCLogsDirectory/gc.log

.

Point-2). It is always needed to see what all objects were present when the OutOfMemory error occured to identify whether those objects belongs to the Application Code/ Application Framework Codes/ The Application Server APIs. Sothat we can isolate the issue. In order to get the details of the Heap Objects collect “HeapDump” either using JHat (not a better tool) or JMap (Much Better compared to the Jhat tool). Plese refer to the post to know how to do it : http://middlewaremagic.com/weblogic/?p=2241

.

Point-3). Once we collected the Heap Dump we can easily monitor the Heap Details using best GUI toold like “Jhat Web Browser” or using “Eclipse Memory Analyzer”.

What to do in case of Native OutOfMemory?

Point-1). Usually Native OutOfMemory causes Server/JVM Crash. So it is always recommended to apply the following JAVA_OPTIONS flags in the Server Start Script to instruct the JVM to generate the HeapDump  ”-XX:+HeapDumpOnOutOfMemoryError“

By default the heap dump is created in a file called java_pidpid.hprof in the working directory of the VM, as in the example above. You can specify an alternative file name or directory with the “-XX:HeapDumpPath=C:/someLocation/“

.

Note: Above Flags are also suitable to collect HeapDump in case of JavaHeap OutOfMemory as well. But these flags never gurantees that the JVM will always generate the Heap Dump in case of any OutOfMemory Situation.

.

Point-2). Usually in case of Native OutOfMemory a “hs_err_pid.log” file is created in case of Sun JDK and “xxxx.dump” file is created in case of JRockit JDK. These log files are usually Text Files and tells about the Libraries which caused the Crash. These files need to be collected and analyzed to find out the root cause.

.

Point-3). Make Sure that the-XX:MaxHeapSize is not set to a Very Large Space…because it will cause a very less Native Space allocation. Because as soon as we increase the HeapSize, the Native Area decreases. Please see the Post:http://middlewaremagic.com/weblogic/?p=4456

.

Point-4). Keep Monitoring the process’s memory using the Unix utility ‘ps’ like following:

ps -p <PID> -o vsz

Here you need to pass the WebLogic Server’s PID (Process ID) to get it’s Threading Details with respect to the Virtual Memory Space.

.

Point-5). If the Heap Usages is less Or if you see that Your Application usages less Heap Memory then it is always better to reduls the MaxHeapSize  so that the Native Area will automatically gets increased.

.

Point-6). Sometimes the JVMs code optimization causes Native OutOfMemory or the Crash…So in this case we can disable the Code Optimization feature of JVM.

(Note: disabling the Code Optimization of JVM will decrease the Performance of JVM)

For JRockit JVM Code Optimization can be disabled using JAVA_OPTION  -Xnoopt

For Sun JDK Code Optimization can be disabled using   JAVA_OPTION  -Xint

What to do in case of OutOfMemory In PermGen?

Point-1). Make Sure that the PermGen Area is not set to a very less value.

.

Point-2). Usually if an Application has Many JSP Pages in that case every JSP will be converted to a *.class file before JSP Request Process. So a large number of JSPs causes generation of a Large number of *.class files all these classes gets loaded in the PermGen area.

.

Point-3). While allocating the -XX:MaxPermSize make sure that you follow a rough Formula… which works in most of the Application Servers.

MaxPermSize = (Xmx/3) —- Very Special Cases (One Third of maximum Heap Size)

MaxPermSize = (Xmx/4) —- Recommended (One Fourth Of maximum Heap Size)

Point-4). If you are repeatedly getting the OutOfMemory in PermGen space then it could be a Classloader leak….

May be some of the classes are not being unloaded from the permgen area of JVM . So please try to increase the -XX:MaxPermSize=512M  or little more and see if it goes away.

If not then add the following JAVA_OPTIONS to trace the classloading and unloading to find out the root cause :

-XX:+TraceClassloading and -XX:+TraceClassUnloading

.

.

Thanks

Jay SenSharma

debugging,JVM Tuning,Monitoring,Native outOfMemory,OutOfMemory,PermGen45 Commentsmore...

30

Dec/09

JVM Crash And Native OutOfMemory

by JaySenSharma under Common Category

Hi,

Jay SenSharma

*******JVM Crash Investigation*********

JVM Core Dump is the most important File to investigate the JVM Crash. By default the Core Dump will be genarated. But Just in case if JVM is not able to generate the Core Dump then there may be the following reasons:

If there is not enough disk space or quota to write the file in your File System.
If JVM is not having to create or write a file in the directory.
if another file exists in the same directory with that is read-only or write-protected.

Unix/Linux-specific: Use the limit or ulimit commands to determine if core dumps are disabled.
Example, on Linux, the command “ulimit -c unlimited” enables core dumps to be written, no matter what their size. Core dump sizes can be restricted if disk space limitations are a concern.

It may be possible to get a thread dump before the process exits.  HotSpot supports the Java_Option-XX:+ShowMessageBoxOnError; the corresponding JRockit option is -Djrockit.waitonerror.  When the JVM is crashing, it may prompt the user :::Do you want to debug the problem?::: This pauses the process, thereby creating an opportunity to generate a thread dump (a stack trace of every thread in the JVM), attach a debugger, or perform some other debugging activity.  However, this does not work in all cases (for eg., in case of stack overflow).

=========================

*******Crash Because of OutOfMemory:*******

Please apply the Following Flag in your JAVA_OPTIONS in the start Script of your Server: -XX:+HeapDumpOnOutOfMemoryError

Details:
1) -XX:+HeapDumpOnOutOfMemoryError option available in 1.5.0_07 and 1.4.2_12, producing an hprof binary format heap dump (By default the DUMP will be generated in your Systems TEMP directory….In Window machine we can easily findout the TEMP directory by running the command “echo %TEMP%”)

2) Analyse hprof heap dumps using HAT, or jhat or YourKit (has an hprof import option)
hprof heap dumps are platform independent and so you don’t need to analyze the dump on the same system that produced it

3) running with -XX:+HeapDumpOnOutOfMemoryError does not impact performance – it is simply a flag to indicate that a heap dump should be generated when the first thread throws OutOfMemoryError.

4) NOTE: -XX:+HeapDumpOnOutOfMemoryError does not work with -XX:+UseConcMarkSweepGC in 1.5.0_07

=========================
TestCase:

public class TestXss {public static void main(final String[] args) throws Exception {// start the given number of threadsfor (int i = 1; i <= Integer.parseInt(args[0]); i++) {System.out.println("Starting Thread " + i);final Thread t = new Thread("T[" + i + "]") {public void run() {try {while (true) {Thread.sleep(1000);}} catch (Exception e) {e.printStackTrace();}}};t.setDaemon(true);t.start();Thread.sleep(5);}// waitThread.sleep(1000000);}}

———————————OUTPUT————————————
NOTE:   java -Xmx1496m -Xss1m TestXss 5000
Starting Thread 411
Starting Thread 412
Starting Thread 413
Starting Thread 414
Starting Thread 415
Starting Thread 416
Starting Thread 417
Exception in thread “main” java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at TestXss.main(TestXss.java:18)

————-

Note: if you see OOM in the Native Space  ( unable to create new native thread) Then Please first of all Try to decrease the-Xss size. It’s default value is -Xss512K   ———>Reduce it to ——–>  -Xss256K

Like this you will see that JVm is able to create more Native Threads…But beware that if u will decrease it more than a certain limit …u may start getting “java.lang.StackOverflowError”.

———————————————-

Thread Related Definitions

• CompletedRequestCount. Number of completed requests in the priority queue.
• ExecuteThreadIdleCount. Number of idle threads in the pool. This count does not include standby threads and stuck threads. The count indicates threads that are ready to pick up new work when it arrives.
• ExecuteThreadTotalCount. Total number of threads in the pool.
• HoggingThreadCount. Number of threads that are being hogged by a request right now. These threads will either be declared as stuck after the configured timeout or will return to the pool before that. The self-tuning mechanism will backfill if necessary.
• MinThreadsConstraintsCompleted. Number of requests with min threads constraint picked up out of order for execution immediately since their min threads requirement was not met. This does not include the case where threads are idle during schedule.
• MinThreadsConstraintsPending. Number of requests that should be executed now to satisfy the min threads requirement.
• PendingUserRequestCount. Number of pending user requests in the priority queue. The priority queue contains requests from internal subsystems and users. This is just the count of all user requests.
• QueueLength. Number of pending requests in the priority queue. This is the total of internal system requests and user requests.
• StandbyThreadCount. Number of threads in the standby pool. Surplus threads that are not needed to handle the present work load are designated as standby and added to the standby pool. These threads are activated when more threads are needed.
• Throughput. Mean number of requests completed per second

How to rotate .out ( stdout) log file in weblogic 12c (solaris/linux)

By default Weblogic wont have options to rotate the *.out file, If you are running in unix or Linux boxes, you can add the following snippet

under /etc/logrotate.conf file append a function to handle *.out files

<Location of logs directory>/*.out

{
copytruncate
rotate
size=5Mb
}