Deploy WebLogic12c to Multiple Machines

In this post, we set-up a WebLogic cluster that spans multiple machines. We start with installing the WebLogic software on the machines involved. Next, we create the WebLogic domain by using the WebLogic Scripting Tool (WLST). Usepack and unpack to create managed server directories on remote machines. Create start and stop scripts for the environment. Deploy an application and tune it by using deployment override descriptors. Set-up load balancing by using the Apache HTTP Server and show the steps involved in scaling the environment. Finally, we perform a load test by using The Grinder and use JRockit Mission Control to obtain insight in the performance of the cluster.

Install WebLogic

First, we choose an installation directory, for example /home/oracle/weblogic12.1.1. This will be our middleware home in the installation of WebLogic. To keep things simple use the same directory structure on both machines.

To install JRockit we follow these steps:

• Run the file jrrt-4.0.1-1.6.0-linux-x64.bin (JRockit can be downloaded here).
• Click next on the welcome screen.
• Define the install directory /home/oracle/jrrt-4.0.1-1.6.0 and click next.
• Optionally select extra components and click next.
• When the installation is finished click done.

To WebLogic we follow these steps:

• Navigate to the JDK’s bin directory (/home/oracle/jrrt-4.0.1-1.6.0/bin).
• Enter the following command: ./java -d64 -Xms1024m -Xmx1024m -jar wls1211_generic.jar (WebLogic can be downloadedhere).
• Click next on the welcome screen.
• Define the Middleware-Home directory and click next.
• Fill in support credentials if applicable and click next.
• Select custom as installation type and click next.
• De-select the evaluation database and click next.
• Select the installed JDK (if the JDK is not shown, click browse and browse to the root of the JDK) and click next.
• Accept the defaults in the directories screen and click next.
• Click next to start the installation.
• De-select run quickstart and click done.

Create domain

To create our domain, we are going to use WLST. The following script, creates the domain in production mode, disables hostname verification, creates two managed server that are clustered and are present on different machines and creates the artifacts needed to run the application, such as JMS and a data source.

beahome = '/home/oracle/weblogic12.1.1';pathseparator = '/';adminusername = 'weblogic';adminpassword = 'magic12c';adminservername='AdminServer';adminserverurl='t3://172.31.0.113:7001';domainname = 'base_domain';domaindirectory = beahome + pathseparator + 'user_projects' + pathseparator + 'domains' + pathseparator + domainname;domaintemplate = beahome + pathseparator + 'wlserver_12.1' + pathseparator + 'common' + pathseparator + 'templates' + pathseparator + 'domains' + pathseparator + 'wls.jar';jvmdirectory = '/home/oracle/jrrt-4.0.1-1.6.0';print 'CREATE DOMAIN';readTemplate(domaintemplate);setOption('DomainName', domainname);setOption('OverwriteDomain', 'true');setOption('ServerStartMode', 'prod');cd('/Security/base_domain/User/weblogic');cmo.setName(adminusername);cmo.setUserPassword(adminpassword);cd('/');writeDomain(domaindirectory);# When using startServer command on WebLogic12c, we have to perform the following steps (1 and 2):# Starting weblogic server ...# WLST-WLS-1326447719560: Exception in thread "Main Thread" java.lang.AssertionError: JAX-WS 2.2 API is required, but an older version was found in the JDK.# WLST-WLS-1326447719560: Use the endorsed standards override mechanism (http://java.sun.com/javase/6/docs/technotes/guides/standards/).# WLST-WLS-1326447719560: 1) locate the bundled Java EE 6 endorsed directory in $WL_HOME/endorsed.# WLST-WLS-1326447719560: 2) copy those JAR files to $JAVA_HOME/jre/lib/endorsed OR add the endorsed directory to the value specified by system property java.endorsed.dirs.print 'START ADMIN SERVER';startServer(adminservername, domainname, adminserverurl, adminusername, adminpassword, domaindirectory);print 'CONNECT TO ADMIN SERVER';connect(adminusername, adminpassword, adminserverurl);print 'START EDIT MODE';edit();startEdit();#print 'CHANGE NODEMANAGER USERNAME AND PASSWORD';#cd('/SecurityConfiguration/' + domainname);#cmo.setNodeManagerUsername(adminusername);#set('NodeManagerPasswordEncrypted', encrypt(adminpassword, domaindirectory));#cd('/');print 'DISABLE HOSTNAME VERIFICATION';cd('/Servers/' + adminservername + '/SSL/' + adminservername);cmo.setHostnameVerificationIgnored(true);cmo.setHostnameVerifier(None);cmo.setTwoWaySSLEnabled(false);cmo.setClientCertificateEnforced(false);cd('/');print 'SAVE AND ACTIVATE CHANGES';save();activate(block='true');print 'SHUTDOWN THE ADMIN SERVER';shutdown(block='true');print 'START ADMIN SERVER';startServer(adminservername, domainname, adminserverurl, adminusername, adminpassword, domaindirectory);print 'CONNECT TO ADMIN SERVER';connect(adminusername, adminpassword, adminserverurl);print 'START EDIT MODE';edit();startEdit();print 'CREATE MACHINE: machine1';machine1 = cmo.createUnixMachine('machine1');machine1.setPostBindUIDEnabled(true);machine1.setPostBindUID('oracle');machine1.getNodeManager().setListenAddress('172.31.0.175');machine1.getNodeManager().setNMType('ssl');print 'CREATE MACHINE: machine2';machine2 = cmo.createUnixMachine('machine2');machine2.setPostBindUIDEnabled(true);machine2.setPostBindUID('oracle');machine2.getNodeManager().setListenAddress('172.31.0.113');machine2.getNodeManager().setNMType('ssl');print 'CREATE CLUSTER: CLUSTER';cluster = cmo.createCluster('cluster');cluster.setClusterMessagingMode('unicast');print 'CREATE MANAGED SERVER: server1';server1 = cmo.createServer('server1');server1.setListenPort(7002);server1.setListenAddress('172.31.0.175');server1.setAutoRestart(true);server1.setAutoKillIfFailed(true);server1.setRestartMax(2);server1.setRestartDelaySeconds(10);server1.getServerStart().setJavaHome(jvmdirectory);server1.getServerStart().setJavaVendor('Oracle');server1.getServerStart().setArguments('-jrockit -Xms1024m -Xmx1024m -Xns256m -Xgc:throughput');print 'CREATE MANAGED SERVER: server2';server2 = cmo.createServer('server2');server2.setListenPort(7003);server2.setListenAddress('172.31.0.113');server2.setAutoRestart(true);server2.setAutoKillIfFailed(true);server2.setRestartMax(2);server2.setRestartDelaySeconds(10);server2.getServerStart().setJavaHome(jvmdirectory);server2.getServerStart().setJavaVendor('Oracle');server2.getServerStart().setArguments('-jrockit -Xms1024m -Xmx1024m -Xns256m -Xgc:throughput');print 'ADD MANAGED SERVERS TO CLUSTER';server1.setCluster(cluster);server2.setCluster(cluster);print 'ADD MANAGED SERVERS TO MACHINE';server1.setMachine(machine1);server2.setMachine(machine2);print 'SAVE AND ACTIVATE CHANGES';save();activate(block='true');print 'START EDIT MODE';startEdit();print 'CREATE FILESTORE FOR SERVER1';filestore1 = cmo.createFileStore('FileStore1');filestore1.setDirectory(beahome + pathseparator + 'deploy');targets = filestore1.getTargets();targets.append(server1);filestore1.setTargets(targets);print 'CREATE JMS SERVER FOR SERVER1';jmsserver1 = cmo.createJMSServer('JMSServer1');jmsserver1.setPersistentStore(filestore1);jmsserver1.setTargets(targets);targets.remove(server1);targets.append(server2);print 'CREATE FILESTORE FOR SERVER2';filestore2 = cmo.createFileStore('FileStore2');filestore2.setDirectory(beahome + pathseparator + 'deploy');filestore2.setTargets(targets);print 'CREATE JMS SERVER FOR SERVER2';jmsserver2 = cmo.createJMSServer('JMSServer2');jmsserver2.setPersistentStore(filestore2);jmsserver2.setTargets(targets);targets.remove(server2);targets.append(cluster);print 'CREATE JMS SYSTEM MODULE';module = cmo.createJMSSystemResource('SystemModule');module.setTargets(targets);print 'CREATE SUBDEPLOYMENT';module.createSubDeployment('SubDeployment');cd('/JMSSystemResources/SystemModule/SubDeployments/SubDeployment');set('Targets',jarray.array([ObjectName('com.bea:Name=JMSServer1,Type=JMSServer'), ObjectName('com.bea:Name=JMSServer2,Type=JMSServer')], ObjectName));cd('/');resource = module.getJMSResource();print 'CREATE CONNECTION FACTORY';resource.createConnectionFactory('ConnectionFactory');connectionfactory = resource.lookupConnectionFactory('ConnectionFactory');connectionfactory.setJNDIName('jms/ConnectionFactory');connectionfactory.setDefaultTargetingEnabled(true);connectionfactory.getTransactionParams().setTransactionTimeout(3600);connectionfactory.getTransactionParams().setXAConnectionFactoryEnabled(true);print 'CREATE UNIFORM DISTRIBUTED QUEUE';resource.createUniformDistributedQueue('DistributedQueue');distributedqueue = resource.lookupUniformDistributedQueue('DistributedQueue');distributedqueue.setJNDIName('jms/CompanyQueue');distributedqueue.setLoadBalancingPolicy('Round-Robin');distributedqueue.setSubDeploymentName('SubDeployment');print 'CREATE DATA SOURCE';datasource = cmo.createJDBCSystemResource('DataSource');datasource.setTargets(targets);jdbcResource = datasource.getJDBCResource();jdbcResource.setName('DataSource');names = ['jdbc/exampleDS'];dataSourceParams = jdbcResource.getJDBCDataSourceParams();dataSourceParams.setJNDINames(names);dataSourceParams.setGlobalTransactionsProtocol('LoggingLastResource');driverParams = jdbcResource.getJDBCDriverParams();driverParams.setUrl('jdbc:oracle:thin:@hostname:1521:sid');driverParams.setDriverName('oracle.jdbc.OracleDriver');driverParams.setPassword('password');driverProperties = driverParams.getProperties();driverProperties.createProperty('user');userProperty = driverProperties.lookupProperty('user');userProperty.setValue('username');connectionPoolParams = jdbcResource.getJDBCConnectionPoolParams();connectionPoolParams.setTestTableName('SQL SELECT 1 FROM DUAL');connectionPoolParams.setConnectionCreationRetryFrequencySeconds(100);print 'SAVE AND ACTIVATE CHANGES';save();activate(block='true');print 'SHUTDOWN THE ADMIN SERVER';shutdown(block='true');

To run the script we can use the following (which also shows the output logging):

[oracle@edu-wls-rh ~]$ cd weblogic12.1.1/wlserver_12.1/common/bin/[oracle@edu-wls-rh bin]$ ./wlst.sh /home/oracle/weblogic12.1.1/deploy/scripts/create_environment.py CLASSPATH=/home/oracle/weblogic12.1.1/patch_wls1211/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/weblogic12.1.1/patch_ocp371/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrrt-4.0.1-1.6.0/lib/tools.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic_sp.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic.jar:/home/oracle/weblogic12.1.1/modules/features/weblogic.server.modules_12.1.1.0.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/webservices.jar:/home/oracle/weblogic12.1.1/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/weblogic12.1.1/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/home/oracle/weblogic12.1.1/utils/config/10.3/config-launch.jar::/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbynet.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbyclient.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbytools.jar::Initializing WebLogic Scripting Tool (WLST) ...Welcome to WebLogic Server Administration Scripting ShellType help() for help on available commandsCREATE DOMAINSTART ADMIN SERVERStarting weblogic server ...WLST-WLS-1326701511109: <Jan 16, 2012 9:11:51 AM CET> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>WLST-WLS-1326701511109: <Jan 16, 2012 9:11:51 AM CET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>WLST-WLS-1326701511109: <Jan 16, 2012 9:11:52 AM CET> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Oracle JRockit(R) Version R28.0.1-21-133393-1.6.0_20-20100512-2126-linux-x86_64 from Oracle Corporation.>WLST-WLS-1326701511109: <Jan 16, 2012 9:11:52 AM CET> <Info> <Management> <BEA-141107> <Version: WebLogic Server 12.1.1.0  Wed Dec 7 08:40:57 PST 2011 1445491 >WLST-WLS-1326701511109: <Jan 16, 2012 9:11:54 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>WLST-WLS-1326701511109: <Jan 16, 2012 9:11:54 AM CET> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>WLST-WLS-1326701511109: <Jan 16, 2012 9:11:54 AM CET> <Notice> <Log Management> <BEA-170019> <The server log file /home/oracle/weblogic12.1.1/user_projects/domains/base_domain/servers/AdminServer/logs/AdminServer.log is opened. All server side log events will be written to this file.>WLST-WLS-1326701511109: <Jan 16, 2012 9:11:57 AM CET> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:00 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:00 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <Log Management> <BEA-170027> <The server has successfully established a connection with the Domain level Diagnostic Service.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RESUMING.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[1]" is now listening on fe80:0:0:0:20c:29ff:feb9:e2cc:7001 for protocols iiop, t3, ldap, snmp, http.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 172.31.0.113:7001 for protocols iiop, t3, ldap, snmp, http.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[2]" is now listening on 127.0.0.1:7001 for protocols iiop, t3, ldap, snmp, http.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[3]" is now listening on 0:0:0:0:0:0:0:1:7001 for protocols iiop, t3, ldap, snmp, http.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <WebLogicServer> <BEA-000329> <Started the WebLogic Server Administration Server "AdminServer" for domain "base_domain" running in production mode.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <WebLogicServer> <BEA-000360> <The server started in RUNNING mode.>Server started successfully.CONNECT TO ADMIN SERVERConnecting to t3://172.31.0.113:7001 with userid weblogic ...Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_domain'.Warning: An insecure protocol was used to connect to theserver. To ensure on-the-wire security, the SSL port orAdmin port should be used instead.START EDIT MODELocation changed to edit tree. This is a writable tree withDomainMBean as the root. To make changes you will need to startan edit session via startEdit(). For more help, use help(edit)Starting an edit session ...Started edit session, please be sure to save and activate yourchanges once you are done.DISABLE HOSTNAME VERIFICATIONSAVE AND ACTIVATE CHANGESSaving all your changes ...Saved all your changes successfully.Activating all your changes, this may take a while ...The edit lock associated with this edit session is releasedonce the activation is completed.WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Warning> <Management> <BEA-141239> <The non-dynamic attribute HostnameVerificationIgnored on weblogic.management.configuration.SSLMBeanImpl@4c9421a2([base_domain]/Servers[AdminServer]/SSL[AdminServer]) has been changed. This may require redeploying or rebooting configured entities.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Warning> <Management> <BEA-141238> <A non-dynamic change has been made which affects the server AdminServer. This server must be rebooted in order to consume this change.> The following non-dynamic attribute(s) have been changed on MBeansthat require server re-start:MBean Changed : com.bea:Name=AdminServer,Type=SSL,Server=AdminServerAttributes changed : HostnameVerificationIgnoredActivation completedSHUTDOWN THE ADMIN SERVERShutting down the server AdminServer with force=false while connected to AdminServer ...WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Alert> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by weblogic.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to SUSPENDING.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <HTTP> <BEA-101278> <There are no active sessions. The Web service is ready to suspend.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to SHUTTING_DOWN.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[3]", listening on 0:0:0:0:0:0:0:1:7001, was shut down.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default", listening on 172.31.0.113:7001, was shut down.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[2]", listening on 127.0.0.1:7001, was shut down.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[1]", listening on fe80:0:0:0:20c:29ff:feb9:e2cc:7001, was shut down.><Jan 16, 2012 9:12:11 AM CET> <Warning> <JNDI> <BEA-050001> <WLContext.close() was called in a different thread than the one in which it was created.>WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Warning> <Management> <BEA-141269> <The temporary bean tree updateDeploymentContext(Mon Jan 16 09:12:11 CET 2012) was allocated for an undo, get, or activate operation, but has not been garbage collected.>WLST-WLS-1326701511109: Stopped draining WLST-WLS-1326701511109WLST-WLS-1326701511109: Stopped draining WLST-WLS-1326701511109WLST lost connection to the WebLogic Server that you wereconnected to, this may happen if the server was shutdown orpartitioned. You will have to re-connect to the server once theserver is available.Disconnected from weblogic server: AdminServerDisconnected from weblogic server:START ADMIN SERVERStarting weblogic server ...WLST-WLS-1326701534609: <Jan 16, 2012 9:12:15 AM CET> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:15 AM CET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:15 AM CET> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Oracle JRockit(R) Version R28.0.1-21-133393-1.6.0_20-20100512-2126-linux-x86_64 from Oracle Corporation.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:16 AM CET> <Info> <Management> <BEA-141107> <Version: WebLogic Server 12.1.1.0  Wed Dec 7 08:40:57 PST 2011 1445491 >WLST-WLS-1326701534609: <Jan 16, 2012 9:12:18 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:18 AM CET> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:18 AM CET> <Notice> <Log Management> <BEA-170019> <The server log file /home/oracle/weblogic12.1.1/user_projects/domains/base_domain/servers/AdminServer/logs/AdminServer.log is opened. All server side log events will be written to this file.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:21 AM CET> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:24 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:24 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <Log Management> <BEA-170027> <The server has successfully established a connection with the Domain level Diagnostic Service.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RESUMING.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[1]" is now listening on fe80:0:0:0:20c:29ff:feb9:e2cc:7001 for protocols iiop, t3, ldap, snmp, http.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 172.31.0.113:7001 for protocols iiop, t3, ldap, snmp, http.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[2]" is now listening on 127.0.0.1:7001 for protocols iiop, t3, ldap, snmp, http.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[3]" is now listening on 0:0:0:0:0:0:0:1:7001 for protocols iiop, t3, ldap, snmp, http.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <WebLogicServer> <BEA-000329> <Started the WebLogic Server Administration Server "AdminServer" for domain "base_domain" running in production mode.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <WebLogicServer> <BEA-000360> <The server started in RUNNING mode.>Server started successfully.CONNECT TO ADMIN SERVERConnecting to t3://172.31.0.113:7001 with userid weblogic ...Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_domain'.Warning: An insecure protocol was used to connect to theserver. To ensure on-the-wire security, the SSL port orAdmin port should be used instead.START EDIT MODEStarting an edit session ...Started edit session, please be sure to save and activate yourchanges once you are done.CREATE MACHINE: machine1CREATE MACHINE: machine2CREATE CLUSTER: CLUSTERCREATE MANAGED SERVER: server1CREATE MANAGED SERVER: server2ADD MANAGED SERVERS TO CLUSTERADD MANAGED SERVERS TO MACHINESAVE AND ACTIVATE CHANGESSaving all your changes ...Saved all your changes successfully.Activating all your changes, this may take a while ...The edit lock associated with this edit session is releasedonce the activation is completed.Activation completedSTART EDIT MODEStarting an edit session ...Started edit session, please be sure to save and activate yourchanges once you are done.CREATE FILESTORE FOR SERVER1CREATE JMS SERVER FOR SERVER1CREATE FILESTORE FOR SERVER2CREATE JMS SERVER FOR SERVER2CREATE JMS SYSTEM MODULECREATE SUBDEPLOYMENTCREATE CONNECTION FACTORYCREATE UNIFORM DISTRIBUTED QUEUECREATE DATA SOURCESAVE AND ACTIVATE CHANGESSaving all your changes ...Saved all your changes successfully.Activating all your changes, this may take a while ...The edit lock associated with this edit session is releasedonce the activation is completed.Activation completedSHUTDOWN THE ADMIN SERVERShutting down the server AdminServer with force=false while connected to AdminServer ...WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Alert> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by weblogic.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to SUSPENDING.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <HTTP> <BEA-101278> <There are no active sessions. The Web service is ready to suspend.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to SHUTTING_DOWN.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default", listening on 172.31.0.113:7001, was shut down.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[2]", listening on 127.0.0.1:7001, was shut down.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[3]", listening on 0:0:0:0:0:0:0:1:7001, was shut down.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[1]", listening on fe80:0:0:0:20c:29ff:feb9:e2cc:7001, was shut down.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Warning> <Management> <BEA-141269> <The temporary bean tree updateDeploymentContext(Mon Jan 16 09:12:34 CET 2012) was allocated for an undo, get, or activate operation, but has not been garbage collected.>WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Warning> <Management> <BEA-141269> <The temporary bean tree updateDeploymentContext(Mon Jan 16 09:12:32 CET 2012) was allocated for an undo, get, or activate operation, but has not been garbage collected.>WLST-WLS-1326701534609: Stopped draining WLST-WLS-1326701534609WLST-WLS-1326701534609: Stopped draining WLST-WLS-1326701534609WLST lost connection to the WebLogic Server that you wereconnected to, this may happen if the server was shutdown orpartitioned. You will have to re-connect to the server once theserver is available.Disconnected from weblogic server: AdminServerDisconnected from weblogic server:

Next, we add a boot.properties file in the ${DOMAIN_HOME}/servers/AdminServer/security directory:

[oracle@edu-wls-rh ~]$ cd weblogic12.1.1/user_projects/domains/base_domain/servers/AdminServer/[oracle@edu-wls-rh AdminServer]$ mkdir security[oracle@edu-wls-rh AdminServer]$ vi boot.properties

Add the following name-value pairs to the boot.properties file:

username=weblogicpassword=magic12c

Note that when the admin server is started these values will be encrypted. The node manager requires authentication to start and stop managed servers. The first time the node manager is started it communicates with the admin server to obtain a username and password that will be used by the admin server to authenticate the node manager. As we created the domain in production mode, random node manager credits are created. When we want to use, for example,nmConnect we have to set the node manager username and password to known values. To this end start the admin server open the admin console:

• Click on base_domain, security, general and click on the advanced link.
• Edit the NodeManager Username and Password. In our case, we set these to respectivelyweblogic and magic12c.
• Start the node manager by using the startNodeManager.sh script.

WebLogic provides two command-line utilities, pack and unpack, that helps to create new managed server directories on remote machines, for example on the172.31.0.113 machine we do the following (first navigate to the ${WL_HOME}/common/bin directory):

[oracle@edu-wls-rh ~]$ cd weblogic12.1.1/wlserver_12.1/common/bin/[oracle@edu-wls-rh bin]$ ./pack.sh -managed=true -domain=/home/oracle/weblogic12.1.1/user_projects/domains/base_domain -template=/home/oracle/temp/base_domain.jar -template_name=base_domain<< read domain from "/home/oracle/weblogic12.1.1/user_projects/domains/base_domain">>  succeed: read domain from "/home/oracle/weblogic12.1.1/user_projects/domains/base_domain"<< set config option Managed to "true">>  succeed: set config option Managed to "true"<< write template to "/home/oracle/temp/base_domain.jar"....................................................................................................>>  succeed: write template to "/home/oracle/temp/base_domain.jar"<< close template>>  succeed: close template

Next, we copy the created base_domain.jar to the 172.31.0.175 machine, for example, by using:

[oracle@edu-wls-rh bin]$ scp oracle@172.31.0.113:/home/oracle/temp/base_domain.jar oracle@172.31.0.175:/home/oracle/temp/oracle@172.31.0.113's password:oracle@172.31.0.175's password:base_domain.jar                                                                                  100%   45KB  44.9KB/s   00:00Connection to 172.31.0.113 closed.

To use unpack on the 172.31.0.175 machine we can do the following (first navigate to the${WL_HOME}/common/bin directory):

[oracle@middleware-magic ~]$ cd weblogic12.1.1/wlserver_12.1/common/bin/[oracle@middleware-magic bin]$ ./unpack.sh -domain=/home/oracle/weblogic12.1.1/user_projects/domains/base_domain -template=/home/oracle/temp/base_domain.jar<< read template from "/home/oracle/temp/base_domain.jar">>  succeed: read template from "/home/oracle/temp/base_domain.jar"<< set config option DomainName to "base_domain">>  succeed: set config option DomainName to "base_domain"<< write Domain to "/home/oracle/weblogic12.1.1/user_projects/domains/base_domain"...............................................................................................>>  succeed: write Domain to "/home/oracle/weblogic12.1.1/user_projects/domains/base_domain"<< close template>>  succeed: close template

To test the set-up, start the admin server (by using ./startWebLogic.sh located in the${DOMAIN_HOME} directory). Subsequently, start the node managers on both machines (by using./startNodeManager.sh located in the ${WL_HOME}/server/bin directory). When the node manager is started for the first time anodemanager.properties file is created in the node manager home directory (which is${WL_HOME}/common/nodemanager by default). We edit the nodemanager.properties and set the propertyStartScriptEnabled=false (the default is true). When doing this, we need to copy the${WL_HOME}/endorsed directory to the jrrt-4.0.1-1.6.0/jre/lib/endorsed directory. When the changes are made the node manager needs to be restarted. Subsequently, use the admin console to the start the managed servers.

Start and stop scripts

In general, it is recommended to start the node manager when the machine boots. In this case, we need to know where to put our custom commands that will be called when the system boots. Note that RedHat Enterprise Linux has a/etc/rc.d/rc.local file, that can be used to put custom commands in. Let us do it in the recommended script-based way. Note that Unix-based systems specify so-called run levels, and that for each run level, scripts can be defined that start a certain service. These scripts are located in the /etc/rc.d/init.d directory. This allows for services to be started when the system boots or to be stopped on system shutdown. The different run levels are specified by a specific directory in the/etc/rc.d directory, i.e.,

• rc0.d – contains scripts that are executed on system shutdown.
• rc1.d – contains scripts for single-user mode.
• rc2.d – contains scripts for multi-user mode.
• rc3.d – contains scripts for multi-user mode and networking.
• rc4.d – not used.
• rc5.d – same as rc3.d plus some graphical stuff.
• rc6.d – contains scripts that are executed on system reboot.

The boot sequence is as follows: in the /etc/inittab file the starting runlevel is defined, the script/etc/rc.d/rc.sysinit is called and /etc/rc.d/rc is run. Therc script looks in the /etc/rc.d/rc<start-runlevel>.d to execute theK**<script-name> scripts with the stop option. After this theS**<script-name> scripts are executed with the start option. Note that scripts are started in numerical order, i.e., theS10network script is executed before the S80sendmail script.

To create a node manager ‘service’, we first create basic start and stop scripts for the node manager. To this end, we will use WLST. To start the node manager by using WLST we can use the following:

bea_home = '/home/oracle/weblogic12.1.1';pathseparator = '/';listen_address = '172.31.0.175';nodemanager_listen_port = '5556';domain_name='base_domain';node_manager_home = bea_home + pathseparator + 'wlserver_12.1' + pathseparator + 'common' + pathseparator + 'nodemanager';startNodeManager(verbose='true', NodeManagerHome=node_manager_home, ListenPort=nodemanager_listen_port, ListenAddress=listen_address);

To stop the node manager by using WLST we can use the following:

bea_home = '/home/oracle/weblogic12.1.1';pathseparator = '/';admin_username = 'weblogic';admin_password = 'magic12c';listen_address = '172.31.0.175';nodemanager_listen_port = '5556';admin_server_listen_port = '7001';domain_name = 'base_domain';domain_home = bea_home + pathseparator + 'user_projects' + pathseparator + 'domains' + pathseparator + domain_name;admin_server_url='t3://' + listen_address + ':' + admin_server_listen_port;nmConnect(admin_username, admin_password, listen_address, nodemanager_listen_port, domain_name, domain_home, 'ssl');stopNodeManager();

Create two shell scripts that respectively run the start and stop script, i.e.,

#!/bin/shBEA_HOME="/home/oracle/weblogic12.1.1"export BEA_HOMEWL_HOME="${BEA_HOME}/wlserver_12.1"export WL_HOME. ${WL_HOME}/common/bin/wlst.sh ${BEA_HOME}/deploy/scripts/startNodeManager.py

#!/bin/shBEA_HOME="/home/oracle/weblogic12.1.1"export BEA_HOMEWL_HOME="${BEA_HOME}/wlserver_12.1"export WL_HOME. ${WL_HOME}/common/bin/wlst.sh ${BEA_HOME}/deploy/scripts/stopNodeManager.py

Create a boot script (nodemanager) under the root user and place this script in the/etc/rc.d/init.d directory. An example boot script for the node manager looks as follows:

#!/bin/sh## chkconfig: 235 91 35# description: starts and stops the node manager##. /etc/rc.d/init.d/functionsRETVAL=0SERVICE="nodemanager"start() {echo "Starting Node Manager"su - oracle -c "/home/oracle/weblogic12.1.1/deploy/scripts/NodeManagerStartService.sh" >/dev/null 2>&1RETVAL=$?[ $RETVAL -eq 0 ] && success || failureecho[ $RETVAL -eq 0 ] && touch /var/lock/subsys/${SERVICE}return $RETVAL} stop() {echo "Stopping Node Manager"su - oracle -c "/home/oracle/weblogic12.1.1/deploy/scripts/NodeManagerStopService.sh" >/dev/null 2>&1RETVAL=$?[ $RETVAL -eq 0 ] && success || failureecho[ $RETVAL -eq 0 ] && rm -r /var/lock/subsys/${SERVICE}return $RETVAL} restart() {stopstart} case "$1" instart)start;;stop)stop;;restart)restart;;*)echo $"Usage: $0 {start|stop|restart}"exit 1esac exit $?

By using the chkconfig command we can update the runlevel information for system services, for example,chkconfig --add nodemanager. To test the set-up shut the system down and start it again. To check if the nodemanager is running we can use eitherps -ef|grep java or netstat -anp|grep :5556 which assumes the node manager is listening on port 5556.

By using WLST we can also create start and stop scripts for the admin server and the managed servers, for example,

bea_home = '/home/oracle/weblogic12.1.1';pathseparator = '/';admin_username = 'weblogic';admin_password = 'magic12c';listen_address_machine1 = '172.31.0.175';listen_address_machine2 = '172.31.0.113';nodemanager_listen_port = '5556';admin_server_listen_port = '7001';domain_name = 'base_domain';domain_home = bea_home + pathseparator + 'user_projects' + pathseparator + 'domains' + pathseparator + domain_name;admin_server_url='t3://' + listen_address_machine2 + ':' + admin_server_listen_port;print 'CONNECT TO NODE MANAGER ON MACHINE2';nmConnect(admin_username, admin_password, listen_address_machine2, nodemanager_listen_port, domain_name, domain_home, 'ssl');print 'START ADMIN SERVER ON MACHINE2';nmStart('AdminServer');print 'CONNECT TO ADMIN SERVER';connect(admin_username, admin_password, admin_server_url);print 'START MANAGED SERVERS ON MACHINE2';start('server2','Server');print 'DISCONNECT FROM NODE MANAGER ON MACHINE2';nmDisconnect();print 'CONNECT TO NODE MANAGER ON MACHINE1';nmConnect(admin_username, admin_password, listen_address_machine1, nodemanager_listen_port, domain_name, domain_home, 'ssl');print 'START MANAGED SERVERS ON MACHINE1';start('server1','Server');# server3 will be added in a later stadium when we scale the cluster#start('server3','Server');print 'DISCONNECT FROM THE ADMIN SERVER';disconnect();print 'DISCONNECT FROM NODE MANAGER ON MACHINE1';nmDisconnect();

bea_home = '/home/oracle/weblogic12.1.1';pathseparator = '/';admin_username = 'weblogic';admin_password = 'magic12c';listen_address_machine1 = '172.31.0.175';listen_address_machine2 = '172.31.0.113';nodemanager_listen_port = '5556';admin_server_listen_port = '7001';domain_name = 'base_domain';domain_home = bea_home + pathseparator + 'user_projects' + pathseparator + 'domains' + pathseparator + domain_name;admin_server_url='t3://' + listen_address_machine2 + ':' + admin_server_listen_port;print 'CONNECT TO NODE MANAGER ON MACHINE1';nmConnect(admin_username, admin_password, listen_address_machine1, nodemanager_listen_port, domain_name, domain_home, 'ssl');print 'CONNECT TO ADMIN SERVER';connect(admin_username, admin_password, admin_server_url);print 'STOPPING MANAGED SERVERS ON MACHINE1';shutdown('server1','Server','true',1000,'true');# server3 will be added in a later stadium when we scale the cluster#shutdown('server3','Server','true',1000,'true');print 'DISCONNECT FROM NODE MANAGER ON MACHINE1';nmDisconnect();print 'CONNECT TO NODE MANAGER ON MACHINE2';nmConnect(admin_username, admin_password, listen_address_machine1, nodemanager_listen_port, domain_name, domain_home, 'ssl');print 'STOPPING MANAGED SERVERS ON MACHINE2';shutdown('server2','Server','true',1000,'true');print 'STOPPING ADMIN SERVER ON MACHINE2';shutdown('AdminServer','Server','true',1000,'true');print 'DISCONNECT FROM NODE MANAGER ON MACHINE2';nmDisconnect();

For these WLST scripts, we create two shell scripts that respectively run the start and stop script, i.e.,

#!/bin/shBEA_HOME="/home/oracle/weblogic12.1.1"export BEA_HOMEWL_HOME="${BEA_HOME}/wlserver_12.1"export WL_HOME. ${WL_HOME}/common/bin/wlst.sh ${BEA_HOME}/deploy/scripts/startDomain.py

#!/bin/shBEA_HOME="/home/oracle/weblogic12.1.1"export BEA_HOMEWL_HOME="${BEA_HOME}/wlserver_12.1"export WL_HOME. ${WL_HOME}/common/bin/wlst.sh ${BEA_HOME}/deploy/scripts/stopDomain.py

To start the domain we can use:

[oracle@edu-wls-rh scripts]$ ./DomainStartService.sh CLASSPATH=/home/oracle/weblogic12.1.1/patch_wls1211/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/weblogic12.1.1/patch_ocp371/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrrt-4.0.1-1.6.0/lib/tools.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic_sp.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic.jar:/home/oracle/weblogic12.1.1/modules/features/weblogic.server.modules_12.1.1.0.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/webservices.jar:/home/oracle/weblogic12.1.1/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/weblogic12.1.1/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/home/oracle/weblogic12.1.1/utils/config/10.3/config-launch.jar::/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbynet.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbyclient.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbytools.jar::Initializing WebLogic Scripting Tool (WLST) ...Welcome to WebLogic Server Administration Scripting ShellType help() for help on available commandsCONNECT TO NODE MANAGER ON MACHINE2Connecting to Node Manager ...<Jan 16, 2012 1:59:54 PM CET> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true><Jan 16, 2012 1:59:54 PM CET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>Successfully Connected to Node Manager.START ADMIN SERVER ON MACHINE2Starting server AdminServer ...Successfully started server AdminServer ...CONNECT TO ADMIN SERVERConnecting to t3://172.31.0.113:7001 with userid weblogic ...Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_domain'.Warning: An insecure protocol was used to connect to theserver. To ensure on-the-wire security, the SSL port orAdmin port should be used instead.START MANAGED SERVERS ON MACHINE2Starting server server2 ...............................................................Server with name server2 started successfullyDISCONNECT FROM NODE MANAGER ON MACHINE2Successfully disconnected from Node Manager.CONNECT TO NODE MANAGER ON MACHINE1Connecting to Node Manager ...Successfully Connected to Node Manager.START MANAGED SERVERS ON MACHINE1Starting server server1 ..................................................................................Server with name server1 started successfullyDISCONNECT FROM THE ADMIN SERVERDisconnected from weblogic server: AdminServerDISCONNECT FROM NODE MANAGER ON MACHINE1Successfully disconnected from Node Manager.

To stop the domain we can use:

[oracle@edu-wls-rh scripts]$ ./DomainStopService.sh CLASSPATH=/home/oracle/weblogic12.1.1/patch_wls1211/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/weblogic12.1.1/patch_ocp371/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrrt-4.0.1-1.6.0/lib/tools.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic_sp.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic.jar:/home/oracle/weblogic12.1.1/modules/features/weblogic.server.modules_12.1.1.0.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/webservices.jar:/home/oracle/weblogic12.1.1/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/weblogic12.1.1/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/home/oracle/weblogic12.1.1/utils/config/10.3/config-launch.jar::/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbynet.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbyclient.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbytools.jar::Initializing WebLogic Scripting Tool (WLST) ...Welcome to WebLogic Server Administration Scripting ShellType help() for help on available commandsCONNECT TO NODE MANAGER ON MACHINE1Connecting to Node Manager ...<Jan 16, 2012 2:05:40 PM CET> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true><Jan 16, 2012 2:05:40 PM CET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>Successfully Connected to Node Manager.CONNECT TO ADMIN SERVERConnecting to t3://172.31.0.113:7001 with userid weblogic ...Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_domain'.Warning: An insecure protocol was used to connect to theserver. To ensure on-the-wire security, the SSL port orAdmin port should be used instead.STOPPING MANAGED SERVERS ON MACHINE1Shutting down the server server1 with force=true while connected to AdminServer ...DISCONNECT FROM NODE MANAGER ON MACHINE1Successfully disconnected from Node Manager.CONNECT TO NODE MANAGER ON MACHINE2Connecting to Node Manager ...Successfully Connected to Node Manager.STOPPING MANAGED SERVERS ON MACHINE2Shutting down the server server2 with force=true while connected to AdminServer ...STOPPING ADMIN SERVER ON MACHINE2Shutting down the server AdminServer with force=true while connected to AdminServer ...WLST lost connection to the WebLogic Server that you wereconnected to, this may happen if the server was shutdown orpartitioned. You will have to re-connect to the server once theserver is available.Disconnected from weblogic server: AdminServerDisconnected from weblogic server:DISCONNECT FROM NODE MANAGER ON MACHINE2Successfully disconnected from Node Manager.

Deployment

The application can be found in the post WebLogic12c in Action (in the ‘My first JavaEE6 application’ section). First, we create the following directory structure in/home/oracle/weblogic12.1.1/deploy:

/loadtest/appLoadTest6.ear/plan

By using weblogic.PlanGenerator, we generate a deployment plan:

[oracle@edu-wls-rh ~]$ cd weblogic12.1.1/wlserver_12.1/server/bin/[oracle@edu-wls-rh bin]$ . ./setWLSEnv.shCLASSPATH=/home/oracle/weblogic12.1.1/patch_wls1211/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/weblogic12.1.1/patch_ocp371/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrrt-4.0.1-1.6.0/lib/tools.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic_sp.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic.jar:/home/oracle/weblogic12.1.1/modules/features/weblogic.server.modules_12.1.1.0.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/webservices.jar:/home/oracle/weblogic12.1.1/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/weblogic12.1.1/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:PATH=/home/oracle/weblogic12.1.1/wlserver_12.1/server/bin:/home/oracle/weblogic12.1.1/modules/org.apache.ant_1.7.1/bin:/home/oracle/jrrt-4.0.1-1.6.0/jre/bin:/home/oracle/jrrt-4.0.1-1.6.0/bin:/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/home/oracle/binYour environment has been set.[oracle@edu-wls-rh bin]$ java weblogic.PlanGenerator -plan /home/oracle/weblogic12.1.1/deploy/loadtest/plan/Plan.xml -root /home/oracle/weblogic12.1.1/deploy/loadtest/ /home/oracle/weblogic12.1.1/deploy/loadtest/app/LoadTest6.earGenerating plan for application /home/oracle/weblogic12.1.1/deploy/loadtest/app/LoadTest6.earExport option is: dependenciesExporting properties...Saving plan to /home/oracle/weblogic12.1.1/deploy/loadtest/plan/Plan.xml...<Jan 17, 2012 1:56:43 PM CET> <Info> <J2EE Deployment SPI> <BEA-260086> <The descriptor information at /home/oracle/weblogic12.1.1/deploy/loadtest/plan/Web.war/WEB-INF/weblogic.xml was saved to the configuration area.><Jan 17, 2012 1:56:43 PM CET> <Info> <J2EE Deployment SPI> <BEA-260072> <Saved configuration for application, LoadTest6.ear>

When the deployment plan has been generated, the files Plan.xml andweblogic.xml are created automatically. The other files, weblogic-application.xml andweblogic-ejb-jar.xml, we have to create ourselves according to the below directory structure:

/loadtest/appLoadTest6.ear/plan/LoadTest6.ear/META-INFweblogic-application.xml/Model.jar/META-INFweblogic-ejb-jar.xml/Web.war/WEB-INFweblogic.xmlPlan.xml

in which the deployment overrides (weblogic-application.xml, weblogic-ejb-jar.xml and weblogic.xml) have the following contents:

<weblogic-application xmlns="http://xmlns.oracle.com/weblogic/weblogic-application" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-application http://xmlns.oracle.com/weblogic/weblogic-application/1.4/weblogic-application.xsd"></weblogic-application>

<weblogic-ejb-jar xmlns="http://xmlns.oracle.com/weblogic/weblogic-ejb-jar" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-ejb-jar http://xmlns.oracle.com/weblogic/weblogic-ejb-jar/1.1/weblogic-ejb-jar.xsd"><weblogic-enterprise-bean><ejb-name>Company</ejb-name><enable-call-by-reference>True</enable-call-by-reference></weblogic-enterprise-bean></weblogic-ejb-jar>

<weblogic-web-app xmlns="http://xmlns.oracle.com/weblogic/weblogic-web-app" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-web-app http://xmlns.oracle.com/weblogic/weblogic-web-app/1.4/weblogic-web-app.xsd"></weblogic-web-app>

The Java EE specification requires that EJB components invoked through their remote interfaces must use pass-by-value semantics, meaning that method parameters are copied during the invocation. Changes made to a parameter in the bean method are not reflected in the caller’s version of the object. Copying method parameters is required in the case of a true remote invocation, of course, because the parameters are serialized by the underlying RMI infrastructure before being provided to the bean method. Pass-by-value semantics are also required between components located in different enterprise applications in the same Java virtual machine due to classloader constraints. EJB components located in the same enterprise application archive (.ear) file are loaded by the same classloader and have the option of using pass-by-reference semantics for all invocations, eliminating the unnecessary copying of parameters passed during the invocation and improving performance. By setting theenable-call-by-reference parameter to true in weblogic-ejb-jar.xml, we enable this feature a specific bean in the application. Local references always use pass-by-reference semantics and are unaffected by theenable-call-by-reference setting. When we deploy an EJB with a remote interface and do not enable call by reference, WebLogic will issue a warning of the performance cost, i.e.,

<Jan 16, 2012 5:01:56 PM CET> <Warning> <EJB> <BEA-010202> <Call-by-reference is not enabled for EJB Company. The server will have better performance if it is enabled. To enable call-by-reference, set the enable-call-by-reference element to True in the weblogic-ejb-jar.xml deployment descriptor or corresponding annotation for this EJB.>

Load balancing

To install the Apache HTTP Server we can follow these steps:

• Unpack the httpd-2.2.21.tar.gz file:
  • gzip -d httpd-2.2.21.tar.gz
  • tar xvf httpd-2.2.21.tar
  • cd httpd-2.2.21
• The next step is to configure:
  • ./configure --prefix=/home/oracle/weblogic12.1.1/apache
• Next, compile the various parts for the Apache HTTP Server by using:
  • make
• To install the Apache HTTP Server we use:
  • make install
• Open the httpd.conf (/home/oracle/weblogic12.1.1/apache/conf) file and adjust the following directives:
  • Listen 7777
  • ServerName 172.31.0.113
• To test the set-up, start the Apache HTTP Server:
  • Navigate to /home/oracle/weblogic12.1.1/apache/bin.
  • Run: ./apachectl -k start (To stop the Apache HTTP Server we can use:./apachectl -k stop).
  • Open a browser en type the following URL: http://172.31.0.113:7777.

To install the the WebLogic plugin we can follow these steps:

• The WebLogic plugins can be downloaded here.
• Unzip the distribution that is appropriate for the operating system, for example,WLSPlugin11g-64bitApache2.2-linux64-x86_64.zip.
• Copy the libclntsh.so.11.1, libnnz11.so and libwlssl.so files to the${APACHE_HOME}/lib directory. Note in our case ${APACHE_HOME} is/home/oracle/weblogic12.1.1/apache.
• Copy the mod_wl.so file to the ${APACHE_HOME}/modules directory.
• Create a configuration file (for example, mod_wl.conf) for the WebLogic plugin and place this file in the${APACHE_HOME}/conf directory.

The contents of the mod_wl.conf look as follows:

LoadModule weblogic_module   "/home/oracle/weblogic12.1.1/apache/modules/mod_wl.so"<IfModule weblogic_module>ConnectTimeoutSecs 10ConnectRetrySecs 2DebugConfigInfo ONWLSocketTimeoutSecs 2WLIOTimeoutSecs 300Idempotent ONFileCaching ONKeepAliveSecs 20KeepAliveEnabled ONDynamicServerList ONWLProxySSL OFF</IfModule><Location /LoadTest6>SetHandler weblogic-handlerWebLogicCluster 172.31.0.175:7002,172.31.0.113:7003</Location>

To let the Apache HTTP Server pick up the configuration we add the following to thehttpd.conf file:

# put it near the end of the file where all the other includes are present# mod_wl configurationInclude conf/mod_wl.conf

Restart the HTTP Server (./apachectl -k stop and ./apachectl -k start).

We are not happy with the load (do not know why, just a hypotheses) and need an extra server. We have two scaling options: vertical and horizontal scaling. Vertical scaling relates to adding more CPUs to a machine. To better utilize the server hardware we can add more WebLogic instances to the machine that could lead to increased application throughput. To determine if this is indeed the case we need to benchmark. Benchmarking for scalability is about measuring resource utilization. Good scalability means that service levels can be maintained while the workload is increased. If an application does not scale well, it is not fully utilizing the hardware. Consequently, throughput will degrade. Ideally, a linear load increase should lead to a linear degradation in service levels and performance. Linear scalability can be approached when so-called share nothing clusters are used. The nodes provide the same functionality and know nothing about other nodes in the cluster (no HTTP session replication). In this case, the computing ability of the cluster increases almost linearly as more nodes are added to the cluster, if the back-end information systems, such as a database, are powerful enough.

Applications that ‘share nothing’ are usually sharing state through the database. The application-tier can scale as far as when the database becomes a bottleneck. In general, relying on a single shared resource will eventually cause contention for that resource and thus limit the scalability. Caching is a good resolution. When we cache data at the application-tier we avoid calls to the database (and also avoid relational data to object data conversions). Caching solutions, such as Coherence, provide different kind of caching, i.e., replicated and partitioned. Replicated does not scale well when cache writes are involved as the data needs to be replicated across all the nodes in the grid. A partioned cache, on the other hand, scales very well when cache writes are involved as data ownership is spread throughout the cluster (the system automatically rebalances the data when the number of nodes in the grid changes – we do not need to decide on how to partition the data, it comes out of the box). Another plus is that access to the cache means at most one network trip, this in order to maintain linear scalability. An optimization on read-access can be made when data can be obtained locally (sticky access) in this case a hybrid solution such as the near cache can be applied.

Horizontal scaling relates to adding more machines to the environment, which gives a failover capability that we cannot get with vertical scaling. A good approach is to combine both scaling techniques to obtain better CPU utilization and failover capability.

In order to create a new server based on server1 open the admin console:

• Click environment, servers, select server1 and click clone.
• Change the server name to server3 and listen port to 7004.
• Click OK.

Create a JMS server and a file store and target these to server3:

• Click services, messaging, JMS servers.
• Click new and enter the name as JMSServer3.
• Click create a new store:
  • Choose file store as type and click next.
  • Enter the following parameters:
    • name: FileStore3
    • target: server3
    • directory: /home/oracle/weblogic12.1.1/deploy
  • Click OK.
• Click next and choose as target, server3.
• Click finish.

Add the created JMS server to the subdeployment of the JMS module:

• Click services, messaging, JMS modules and choose systemmodule.
• Click the subdeployments tab and choose subdeployment.
• Add JMSServer3 and click save.

Start server3 by using the admin console:

• Click environment, servers.
• Click the control tab.
• Select server3 and click start.

To test the configuration we can use the URL: http://172.31.0.113:7777/LoadTest6/?__WebLogicBridgeConfig. When the URL is accessed the General Server List should show the servers in the cluster, i.e.,

WebLogic Server Plugin version 1.1, <WLSPLUGINS_11.1.1.4.0_LINUX.X64_101209.1115>Query String: '?__WebLogicBridgeConfig'WebLogic Cluster List:General Server List:Host: '172.31.0.175' Port: 7004 SecurePort: 0 Status: OKHost: '172.31.0.113' Port: 7003 SecurePort: 0 Status: OKHost: '172.31.0.175' Port: 7002 SecurePort: 0 Status: OKConnectRetrySecs: '2'ConnectTimeoutSecs: '10'WLCookieName: JSESSIONIDDebug: 'no debugging'DebugConfigInfo: 'ON'DefaultFileName: ''DisableCookie2Server: OFFDynamicServerList: 'ON'ErrorPage: ''FileCaching: ONIdempotent: ONKeepAliveEnabled: ONKeepAliveSecs: 20MaxPostSize: '0'MaxSkipTime: '10'PathPrepend: ''PathTrim: ''QueryFromRequest: OFFWLForwardUriUnparsed: OFFWLAllowDoubleEscapedURI: OFFSecureProxy: 'OFF'StatPath: 'false'WLDNSRefreshInterval: '0'WLIOTimeoutSecs(old name is HungServerRecoverSecs): '300'WLLogFile: ''WLSocketTimeoutSecs: '2'WLProxySSL: OFFWLProxyPassThrough: OFFWLProxySSLPassThrough: OFFRuntime statistics:requests: 0successful requests: 0Exception objects created: 0Exception Objects deleted: 0URL Objects created: 1URL Objects deleted: 0connections recycled: 0UNKNOWN_ERROR_CODE exceptions: 0CONNECTION_REFUSED exceptions: 0CONNECTION_TIMEOUT exceptions: 0READ_ERROR_FROM_CLIENT exceptions: 0READ_ERROR_FROM_SERVER exceptions: 0READ_ERROR_FROM_FILE exceptions: 0WRITE_ERROR_TO_CLIENT exceptions: 0WRITE_ERROR_TO_SERVER exceptions: 0WRITE_ERROR_TO_FILE exceptions: 0READ_TIMEOUT exceptions: 0WRITE_TIMEOUT exceptions: 0UNKNOWN_HOST exceptions: 0NO_RESOURCES exceptions: 0PROTOCOL_ERROR exceptions: 0CONFIG_ERROR exceptions: 0FAILOVER_REQUIRED exceptions: 0POST_TIMEOUT exceptions: 0REQUEST_ENTITY_TOO_LARGE exceptions: 0HALF_OPEN_SOCKET_RETRY exceptions: 0BAD_REQUEST_FROM_CLIENT exceptions: 0SSL_INIT_ERROR exceptions: 0Build date/time: Dec 20 2010 22:11:08Change Number: 1013

The runtime statistics depict results when a number of requests (http://172.31.0.113:7777/LoadTest6/testservlet) are made. The new server is added to the general server list (which is due to the usedDynamicServerList configuration). As the application is stateless and no sessions are created (no users need to login), we have no session binding. To get an idea how the load is balanced, click in the admin console on deployments, LoadTest6 and subsequently click on the workload, monitoring tab.

Testing

We can test the environment by using the The Grinder. To set-up The Grinder, we create the following scripts:

A script to set the environment (setGrinderEnv.sh):

#!/bin/shGRINDERPATH=/home/oracle/grinder-3.4export GRINDERPATHJAVA_HOME=/home/oracle/jdk1.6.0_24export JAVA_HOMEUSER_MEM_ARGS="-server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC"export USER_MEM_ARGSGRINDERPROPERTIES=${GRINDERPATH}/examples/grinder.propertiesexport GRINDERPROPERTIESCLASSPATH=${GRINDERPATH}/lib/grinder.jar:${CLASSPATH}export CLASSPATHPATH=${JAVA_HOME}/bin:${PATH}export PATH

A script to start the console (startConsole.sh):

#!/bin/shsource setGrinderEnv.shjava ${USER_MEM_ARGS} -cp ${CLASSPATH} net.grinder.Console

A script to start an agent that starts the test processes (startAgent.sh):

#!/bin/shsource setGrinderEnv.shjava ${USER_MEM_ARGS} -cp ${CLASSPATH} net.grinder.Grinder ${GRINDERPROPERTIES}

The grinder.properties file has, among others, the following properties:

# The file name of the script to run.## Relative paths are evaluated from the directory containing the# properties file. The default is "grinder.py".grinder.script = test.py# The number of worker processes each agent should start. The default# is 1.grinder.processes = 1# The number of worker threads each worker process should start. The# default is 1.grinder.threads = 1# The number of runs each worker process will perform. When using the# console this is usually set to 0, meaning "run until the console# sneds a stop or reset signal". The default is 1.grinder.runs = 0

The test.py script has the following contents:

from net.grinder.script.Grinder import grinderfrom net.grinder.script import Testfrom net.grinder.plugin.http import HTTPRequesttest1 = Test(1, "Request resource")request1 = test1.wrap(HTTPRequest())class TestRunner:    def __call__(self):        result = request1.GET("http://172.31.0.113:7777/LoadTest6/testservlet")

Start the test by running startConsole.sh and subsequently startAgent.sh:

During the loadtest we are interested in monitoring the paging. The Linux memory handler manages the allocation of physical memory by freeing portions of physical memory when possible. All processes use memory, but each process does not need all its allocated memory all the time. Taking advantage of this fact, the kernel frees up physical memory by writing some or all of a process’ memory to disk until it is needed again. The kernel uses paging and swapping to perform this memory management. Paging refers to writing portions (pages) of a process’ memory to disk. Swapping refers to writing the entire process to disk. When pages are written to disk, the event is called a page-out, and when pages are returned to physical memory, the event is called a page-in. A page fault occurs when the kernel needs a page, finds it does not exist in physical memory because it has been paged-out, and re-reads it in from disk. When the kernel detects that memory is running low, it attempts to free up memory by paging out. Though this may happen briefly from time to time, if page-outs are plentiful and constant, the kernel can reach a point where it is actually spending more time managing paging activity than running the applications, and system performance suffers. To monitor paging we can use, for example, vmstat 60 10 (which runs vmstat with ten updates, 60 seconds apart):

# Output machine1 (where server1 and server3 are running)procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st 2  0      0 1185576  40744 521408    0    0    63    10  384  347  6  3 90  1  0 2  0      0 1178368  40852 527824    0    0     0   132 2921 2509 21 10 68  0  0 1  0      0 1174384  40924 530588    0    0     0    63 2442 2130 17  9 74  0  0 0  0      0 1169796  41020 534336    0    0     0    77 2798 2408 19 10 71  0  0 0  0      0 1162728  41124 540188    0    0     1   130 2941 2614 18 11 70  0  0 1  0      0 1155028  41216 544020    0    0     0    76 2899 2549 21 10 69  0  0 0  0      0 1150812  41316 548804    0    0     0   105 3130 2790 21 11 68  0  0 0  0      0 1140256  41420 554224    0    0     0   120 2947 2595 19 11 69  0  0 1  0      0 1121904  41528 561132    0    0     0   140 2896 2557 19 11 70  0  0 3  0      0 1110760  41628 567212    0    0     0   123 2963 2657 18 13 68  0  0Output machine2 (where the admin server and server2 are running)procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st 0  0      0 1622200  34556 574508    0    0    86    15  642  305  2  0 97  1  0 0  0      0 1617552  34656 578236    0    0     0    87 2685 1967  5  2 93  0  0 0  0      0 1615476  34748 580480    0    0     0    55 2309 1562  3  1 95  0  0 0  0      0 1612872  34844 582992    0    0     0    64 2616 1898  4  2 95  0  0 1  0      0 1609964  34948 585576    0    0     0    73 2735 2017  4  2 94  0  0 1  0      0 1605508  35052 589116    0    0     0    81 2712 1985  4  2 94  0  0 1  0      0 1602076  35152 592352    0    0     0    77 2913 2313  5  2 93  0  0 0  0      0 1596448  35260 595104    0    0     0    69 2773 2095  4  2 94  0  0 0  0      0 1590000  35360 599212    0    0     0    92 2693 2038  4  2 94  0  0 0  0      0 1587164  35456 601748    0    0     0   106 2633 1861  5  2 93  0  0 FIELD DESCRIPTION FOR VM MODE   Procs       r: The number of processes waiting for run time.       b: The number of processes in uninterruptible sleep.   Memory       swpd: the amount of virtual memory used.       free: the amount of idle memory.       buff: the amount of memory used as buffers.       cache: the amount of memory used as cache.       inact: the amount of inactive memory. (-a option)       active: the amount of active memory. (-a option)   Swap       si: Amount of memory swapped in from disk (/s).       so: Amount of memory swapped to disk (/s).   IO       bi: Blocks received from a block device (blocks/s).       bo: Blocks sent to a block device (blocks/s).   System       in: The number of interrupts per second, including the clock.       cs: The number of context switches per second.   CPU       These are percentages of total CPU time.       us: Time spent running non-kernel code. (user time, including nice time)       sy: Time spent running kernel code. (system time)   id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.       wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.       st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.

The values for si and so are both zero, indicating there are no page-ins and page-outs.

Let us look at some of the diagnostics collected by WebLogic. To this end, open the admin console, click deployments and click the monitoring tab. This page displays monitoring information for all applications deployed to the domain. The JMS tab displays monitoring information for all JMS destinations (note that the application only sends a message when a new person is added):

JMSServer1!JMSServer1.TemporaryQueue0 - Bytes Received Count: 1230, Messages Received Count: 40JMSServer2!JMSServer2.TemporaryQueue0 - Bytes Received Count: 1230, Messages Received Count: 40JMSServer3!JMSServer3.TemporaryQueue0 - Bytes Received Count: 1230, Messages Received Count: 40SystemModule!JMSServer1@DistributedQueue - Bytes Received Count: 719274, Messages Received Count: 2347SystemModule!JMSServer2@DistributedQueue - Bytes Received Count: 703990, Messages Received Count: 2297SystemModule!JMSServer3@DistributedQueue - Bytes Received Count: 715648, Messages Received Count: 2335

The EJB (stateless and message-driven) tab displays monitoring information for all the Enterprise JavaBeans (EJBs):

Companyexample - Pooled Beans Current Count: 3, Access Total Count: 348936, Transactions Committed Total Count: 348899, Transactions Rolled Back Total Count: 37CompanyMDB example - Access Total Count: 6979, Processed Message Count: 2347, Transactions Committed Total Count: 6979

The JDBC tab displays monitoring information for all JDBC data sources:

DataSource server1 - Active Connections High Count: 2, Connection Delay Time: 155, PrepStmt Cache Access Count: 105396, Reserve Request Count (cummulative running count of requests for a connection): 116715, Waiting For Connection Total (cumulative running count of requests for a connection that had to wait before getting a connection) 0DataSource server2 - Active Connections High Count: 2, Connection Delay Time: 127, PrepStmt Cache Access Count: 10173, Reserve Request Count (cummulative running count of requests for a connection): 116716, Waiting For Connection Total (cumulative running count of requests for a connection that had to wait before getting a connection) 0DataSource server3 - Active Connections High Count: 2, Connection Delay Time: 143, PrepStmt Cache Access Count: 99600, Reserve Request Count (cummulative running count of requests for a connection): 116698, Waiting For Connection Total (cumulative running count of requests for a connection that had to wait before getting a connection) 0

The workload tab shows statistics for the Work Managers, constraints, and policies that are configured for application deployments:

default server1 - Pending Requests: 0, Completed Requests: 118664default server2 - Pending Requests: 0, Completed Requests: 118616default server3 - Pending Requests: 0, Completed Requests: 118635

We use the JRockit Mission Control Flight Recording, to see if there are any hick-ups due to garbage collections:

One thing to note is that the garbage collections did not run at same time:

• server1 (172.31.0.175): 03:46:18 154ms – 03:46:18 295ms
• server2 (172.31.0.113): 03:43:29 508ms – 03:43:29 751ms
• server3 (172.31.0.175): 03:43:52 399ms – 03:43:52 650ms

In general, JVM instances running on the same machine will typically not run the garbage collection at the same time. This means that we will have a JVM available to process application requests on other available CPUs. This is an advantage of vertical scaling that leads to a higher application throughput.

To get some insight what effect the garbage collection had on the application we use the events environment of the flight recording. We enable the following events:

• Java application – Java blocked
• Java application – Java wait
• Java virtual machine – GC – garbage collection
• WebLogic Server – EJB – EJB business method invocation
• WebLogic Server – JDBC – JDBC statement execute
• WebLogic Server – Servlet – Servlet invocation

The thread group ‘thread group for queue: weblogic.socket.Muxer’ contain the muxer threads. As a native muxer is used and we have 2 CPUs available, the number of threads is equal to#CPUs + 1 = 3. These threads are always showing this behavior, i.e., one thread at a time is active; picking requests of the sockets and put it in the execute queue. Here it will be picked up by an execute thread from the thread group ‘pooled threads’, which will process it, i.e., the servlet invocation (light green), the EJB business method invocation (blue) and the JDBC statement execution (purple). Note that during the garbage collection the execute threads have to wait (yellow) and a pause in the processing of the work defined for the execute thread is introduced.

References

[1] WebLogic 12c Documentation Library.
[2] Monitoring Virtual Memory with vmstat.

load test,multiple machines,scaling,WebLogic cluster