Out of Memory Analysis

Monitor EJBs in Caches and HTTP Sessions

This script has been created by Gareth Chapman. Gareth also submitted the description that follows.

This script is designed to analyze the cause of out of memory errors. It monitors the amount of EJBs in all the caches and the amount of HTTP Sessions as this is usually where most memory is taken up.

Here are some details of what this script does:

Connect to the Weblogic server instance specified and initially log the details of the JVM and OS such as this: 

========================================================
Java Vendor Sun Microsystems Inc.
Java Version 1.3.1_06
OS Name Windows 2000
OS Version 5.0

Next it will get the JVM used heap amount and free heap size and determine if the free size in the heap is less than or greater than a limit specified. If the free size of the heap has not reduced to the limit specified then the following is logged: 

==================== Fri Apr 30 10:05:36 BST 2004 ====================
Used heap =  1073479680
Free heap =  1026626296
No need to log cache or session details yet

If the free size of the heap is less than the limit specified then the script will get all EJB Caches for all EJBs deployed on the server and log the current amount of beans in each cache, then will will get all of the Web apps deployed on the server and log the current amount of HTTP Sessions such as this: 

==================== Fri Apr 30 10:06:55 BST 2004 ====================
Used heap =  1073479680
Free heap =  1025689480
cache count for  myserver_petstore_cartEjb_TheCart 0
cache count for  myserver_petstore_customerEjb_AccountEJB 0
cache count for  myserver_petstore_customerEjb_AddressEJB 0
cache count for  myserver_petstore_customerEjb_ContactInfoEJB 0
cache count for  myserver_petstore_customerEjb_CreditCardEJB 0
cache count for  myserver_petstore_customerEjb_CustomerEJB 0
cache count for  myserver_petstore_customerEjb_ProfileEJB 0
cache count for  myserver_petstore_petstoreEjb_TheShoppingClientController 75
cache count for  myserver_petstore_petstoreEjb_TheShoppingClientFacade 76
cache count for  myserver_petstore_signonEjb_UserEJB 0
cache count for  myserver_petstore_uidgenEjb_CounterEJB 0
cache count for  myserver_petstoreopc_poEjb_AddressEJB 0
cache count for  myserver_petstoreopc_poEjb_CardEJB 0
cache count for  myserver_petstoreopc_poEjb_LineItemEJB 0
cache count for  myserver_petstoreopc_poEjb_PurchaseOrderEJB 0
cache count for  myserver_petstoresupplier_suppPOEjb_AddressEJB 0
cache count for  myserver_petstoresupplier_suppPOEjb_ContactInfoEJB 0
cache count for  myserver_petstoresupplier_suppPOEjb_LineItemEJB 0
cache count for  myserver_petstoresupplier_suppPOEjb_SupplierOrderEJB 0
cache count for  myserver_petstoresupplier_supplierEjb_InventoryEJB 0
HTTP Sessions for webapp  myserver_myserver_console_console 0
HTTP Sessions for webapp  myserver_myserver_petstore_petstore 75
HTTP Sessions for webapp  myserver_myserver_petstoreadmin_petstoreadmin 0
HTTP Sessions for webapp  myserver_myserver_petstoresupplier_supplier 0
HTTP Sessions for webapp  myserver_myserver_tour_tour 0
HTTP Sessions for webapp  myserver_myserver_uddi_uddi 0
HTTP Sessions for webapp  myserver_myserver_uddiexplorer_uddiexplorer 0
HTTP Sessions for webapp  myserver_myserver_wl_management_internal1_wl_management_internal1 0
HTTP Sessions for webapp  myserver_myserver_wl_management_internal2_wl_management_internal2 0

The main variables you will need to change and the top of the script are: 

HOST = localhost	# The host of the Weblogic server instance to connect to
PORT = 7001		# The port of the Weblogic server instance to connect to
SERVER = myserver	# The name of the Weblogic server instance
USERNAME = weblogic	# The administrator account name
PASSWORD = weblogic	# The administrator account password
SLEEP = 1		# The delay in between loops specified in minutes
FREEHEAPTRIGGER = 300	# The amount of memory that can be available free in the
			# JVM heap before cache and session logging starts,
			# specified in MB.
LOGFILE = MonitoringResults.log		# The log file to put results into

Here comes the script: 

# This script monitors EJBs in caches and HTTP Sessions for analysing OutOfMemory cause

# Thanks to Gareth Chapman for submitting this script

output off

HOST = localhost
PORT = 7001
SERVER = myserver
USERNAME = weblogic
PASSWORD = weblogic
SLEEP = 1
FREEHEAPTRIGGER = 300

LOGFILE = MonitoringResults.log

lowMemoryTrigger = expr($FREEHEAPTRIGGER * 1000000)
sleepTime = expr($SLEEP * 60 * 1000)

#####################
#Connect to the server on $HOST and $PORT
#using $USERNAME and $PASSWORD
#####################
connect $HOST:$PORT $USERNAME $PASSWORD


   #####################
   #Log the JVM and OS details at the top of the log
   #####################
   javaVendor = get /JVMRuntime/$SERVER/JavaVendor
   javaVersion = get /JVMRuntime/$SERVER/JavaVersion
   osName = get /JVMRuntime/$SERVER/OSName
   osVersion = get /JVMRuntime/$SERVER/OSVersion

   print "========================================================" >> $LOGFILE
   print "Java Vendor" $javaVendor >>$LOGFILE
   print "Java Version" $javaVersion >>$LOGFILE
   print "OS Name" $osName >>$LOGFILE
   print "OS Version" $osVersion >>$LOGFILE


while true do

    #####################
    #Get the date and log it to the file
    #####################
    date

    print "====================" $LAST "====================" >> $LOGFILE

    #####################
    #Get the heap details and log it to the file
    #####################

    heapFree = get /JVMRuntime/$SERVER/HeapFreeCurrent
    heapUsed = get /JVMRuntime/$SERVER/HeapSizeCurrent

    print "Used heap = " $heapUsed >> $LOGFILE
    print "Free heap = " $heapFree >> $LOGFILE

    if $heapFree < $lowMemoryTrigger

      #####################
      #Get the entity cache current count for each EJB
      #####################

      dir /EJBCacheRuntime
      caches = $LAST
      for $cache in $caches do
        cacheCount = get /EJBCacheRuntime/$cache/CachedBeansCurrentCount
        print "cache count for " $cache $cacheCount >>$LOGFILE
      end

      #####################
      #Get the Web App number of open sessions
      #####################

      dir /WebAppComponentRuntime
      webapps = $LAST
      for $webapp in $webapps do
        sessionCount = get /WebAppComponentRuntime/$webapp/OpenSessionsCurrentCount
        print "HTTP Sessions for webapp " $webapp $sessionCount >>$LOGFILE
      end

    else
      print "No need to log cache or session details yet" >>$LOGFILE
    end

    sleep $sleepTime

end

disconnect

exit