PS Performance Tuning

Table of Contents

1. Overview. 4

2. Assumption. 4

3. Tuning Tips For Peoplesoft Applications. 4

4. Performance Monitoring. 6

5. Tuning Application Server And Configuration Parameters. 7

6. Weblogic Setup And Configuration. 14

8. Conclusion. 18

9. Appendix. 18

1. Overview

This document helps to tune PeopleSoft applications and to find bottlenecks in PeopleSoft environments.

2. Assumption

Listed low is a list of variables used in this document and what they denote:

1. HOST_PS_HOME : The Home Directory of PeopleSoft

2. HOST_ORACLE_SID : Oracle or Application Instance Name

3. HOST_PROCESS_SCHEDULAR_NAME : Process Schedulers Name

4. HOST_WL_HOME : Weblogic Home Directory.

3. Tuning Tips for PeopleSoft Applications

PeopleSoft application layers like Database Server, Application Server, Web Server, Desk top are interdependent. All layers must work in concert to achieve maximum performance. When a given resource is exhausted, resulting bottleneck can be catastrophic. This document contains Tuning Tips of Application Server and Web Server.

Monitor System for Errors

To tune the system to monitor for errors logs are maintained. Some of logs are listed below:

· Application Server Logs :

<$HOST_PS_HOME>/appserv/<$HOST_ORACLE_SID>/LOGS/APSRV* and TUX*

· Process Scheduler Logs :

<$HOST_PS_HOME>/appserv/<$HOST_ORACLE_SID>/prcs//LOGS/TUX* and SCH*

· Web Server Logs :

<$HOST_WL_HOME>/logs

Reviewing Tuxedo Logs

To find errors in Tuxedo logs read the Tuxedo logs files, and occurrences of a given error types by giving following command.

more TUXLOG. | grep 262 |wc –l

Look for patterns in errors. For example

021626.APPSRV1 !BBL.1232: CMDTUX_CAT:1836: WARN: Server(2024) processing terminated with SIGKILL after SVCTIMEOUT

Time Stamp : 021626

Server Name :APPSRV1

Process Name : BBL

BBL PID :1232

Message Category :CMDTUX_CAT

Error Number :1836

Message Category Type :WAR

Message Details :Server(2024) processing terminated with SIGKILL

after SVCTIMEOUT 2024

Server PID :2024

These are the some bad Tuxedo Messages

CMDTUX_CAT:1836:

CMDTUX_CAT:1407:

LIBTUX_CAT:1241:

LIBTUX_CAT:1185:

LIBTUX_CAT:550:

LIBTUX_CAT:541:

LIBTUX_CAT:477:

LIBTUX_CAT:476:

LIBTUX_CAT:216:

There are more

These are the some good messages

JOLT_CAT:1185:

JOLT_CAT:1198:

LIBTUX_CAT:577:

Reviewing Application server Logs

Application server logs give more details into PeopleSoft operations generally need to increase log levels to get necessary details and look for components that are trouble

To find errors in Application Servers, read the Application logs and occurrences of a given error types by giving following command.

For example, to evaluate component usage give following command

more APPSRV_0605.LOG|grep 'component'|awk '{print "Component name " $11 , "Menu Name " $14}'|more

Typical Application Server log

PSAPPSRV.2980 [06/05/04 00:00:06 ogomezx-s@WEBSERVER1ICPanel](4) Starting Service

PSAPPSRV.2980 [06/05/04 00:00:06 ogomezx-s@WEBSERVER1(IE 6.0; WINNT) ICPanel](4)

Executing component RB_TD_360_SRCH/GBL in menu RB_TD_360

PSAPPSRV.2980 [06/05/04 00:00:06 ogomezx-s@WEBSERVER1(IE 6.0; WINNT) ICPanel](5)

(NET.100): Network API tracing TUXEDO: tpalloc {type='CARRAY', subtype=NULL, len=8192}

…

PSAPPSRV.2980 [06/05/04 00:00:06 ogomezx-s@WEBSERVER1(IE 6.0; WINNT) ICPanel](5)

(NET.100): Network API tracing TUXEDO: tprealloc return {pmem=0x09e9c180}

PSAPPSRV.2980 [06/05/04 00:00:06 ogomezx-s@WEBSERVER1](4) Service ICPanel completed:

elapsed time=0.8130

PSAPPSRV.2980 [06/05/04 00:00:06 ogomezx-s@WEBSERVER1](5) (NET.100): Network API tracing

TUXEDO: tpreturn {rval='TPSUCCESS', rcode=0, data=0x09e9c180, len=53371,

flags=0x00000000}

4. Performance Monitoring

PeopleSoft has introduced PeopleSoft performance monitor in People Tools 8.44 and later.

Peoplesoft Performance monitor enables to monitor the performance of multiple PeopleSoft systems simultaneously. The People Tools based application enables system administrators to perform the following:

• Store and view performance Data
• Analyze real-time and historical performance metrics
• Track and trend system response time across the tiers of the PeopleSoft System.

Lightweight monitoring agents run on the servers in a PeopleSoft System and collect performance data, which they send to PeopleSoft Performance monitor.

Performance Monitor stores all performance data in the database. The monitoring agents impose minimal impact on CPU and network usage, which makes the Performance Monitor suitable for monitoring development, testing and production systems in real time.

Use a single PeopleSoft performance Monitor instance to monitor multiple PeopleSoft applications. Production systems are typically monitored by a separate PeopleSoft Performance Monitor instance.

PeopleSoft Performance monitor was designed to be used in production system. There is a tradeoff between the number of statistics collected and the impact on system resource usage. For on-line system, it can be configured in such a way that its impact on system performance is minimal (around 5%system resource usage. The PeopleSoft Performance Monitor can also be switched to standby mode on-the fly so that the impact on system resource usage is negligible. This does not require rebooting or recycling of the server.

PeopleSoft Performance Monitor provides following analytic charts

• User Requests
• PeopleTools Component Statistics & Component Trace
• Portal Statistics
• PIA Statistics
• Top Portal Content Requests
• Top PeopleCode Events & Executes
• Top PeopleCode SQL Statements
• Top PeopleTools Components
• Drill-down or drill-up via PMU trees
• Sample Queries that expose DB schema
• Grids are exportable to Excel!

5. Tuning Application Server and configuration Parameters

Application Server Memory Guidelines

Lack of memory resources on the application is the most common cause of serious online performance issues.

How to determine the queue length of the specific domain?

If Application is experiencing more than four seconds response time for every web page refresh, it is very likely a problem with application server memory.

These problems are most likely caused by "memory swapping".

Here are some things that can be done to see if more memory is needed for the application server.

1. Make sure not to have too many PSAPPSRV processes booted. During peak usage times, see that some small amount of queuing should be present for the psappsrv processes.

Too many processes, this will affect performance.The extra PSAPPSRV processes will not

Only take up memory space and CPU time from the OS, the main issue is that each SAPPSRV will not be 'sufficiently' cached. i.e. each PSAPPSRV can only cache a portion of the user panels. It will take much longer to fully cache each PSAPPSRV.

To determine the queue length of the specific domain, do the following steps:

1. Run psadmin.

2. Go to “PeopleSoft Domain Administration” for the specific domain name.

3. Select “TUXEDO command line”.

4. Run the command ‘pq’.

Prog Name Queue Name # Server WK Queue #Queued Ave Len Machine

WSL 00001.00010 1 - 0 - ss-hp21b

JREPSVR 00094.00250 1 - 0 - ss-hp21b

PSSAMSRV SAMQ 1 - 0 - ss-hp21b

BBL 52201 1 - 0 - ss-hp21b

PSAPPSRV APP0 24 - 0 - ss-hp21b

JSL 00094.00250 1 - 0 - ss-hp21b

2. The guidelines for the memory footprint for the PSAPPSERV are presented below:

• CRM and HRMS use about 100-150 MB per app server process. CRM gets 50 users per process. HRMS gets less.

• Supply Chain uses 300-500 MB per process. It can get about 10-20 users per process.

• Financials use 150-300 MB per process. It can get about 20-30 users per process.

So, for example, if 1 GB of memory is available for a Financials application server, boot

no more than four PSAPPSRV processes.

3. A common problem is having four large domains on one computer with insufficient memory.

4. Check memory utilization on the app server when performance problems are experienced. Check for swapping by using the memory monitoring procedures outlined in the following sections.

5. In case of swapping, either add more memory or reduce the number of domains or

PSAPPSRV processes on the computer.

Determining PeopleSoft AppServer memory usage under NT

Use NT TaskManager to monitor all the PeopleSoft processes.

PeopleSoft processes have a process name prefix with ‘ps’.

Start TaskManager and select ‘View’ from the menu bar, and then choose ‘Select Columns’. Pick the Memory Usage and Virtual Memory Size columns.

Sort the processes by name, and the two memory columns will tell approximately the amount of memory each PeopleSoft process is using.

"Memory Usage" is not the entire memory consumption of the process, but only the amount of physical RAM that process is using.

However, with NT, an application may be using much more virtual memory (i.e., the paging file) than physical memory, At the time of gauging memory, have both these fields selected.

The memory usage columns will include the shared libraries loaded into each process, therefore, if all the processes are added up, it will be double-counting the shared libraries portion.

Since PeopleSoft shared a lot of the common libraries, the summation of all PeopleSoft processes is a quick estimator, not an absolute measurement of memory usage.

The total “Memory Usage” and the “VM Size” of the combined PeopleSoft processes should not be higher than 70% of the Real memory of the server.

Determining if excessive memory swapping happens under NT

Start NT Performance Monitor, perfmon.

Add the Object Counter, under ‘Memory’, and then the ‘Page/sec’.

Pages/sec is the number of pages read from the disk or written to the disk to resolve memory references to pages that were not in memory at the time of the reference.

This is the sum of Pages Input/sec and Pages Output/sec.

This counter includes paging traffic on behalf of the system cache to access file data for applications.

This value also includes the pages to/from non-cached mapped memory files.

This is the primary counter to observe if the concern is to know about excessive memory pressure (that is, thrashing), and the excessive paging that may result.

When the Page/sec goes above 100 hard pages per second, there is a swapping problem. The number of PSAPPSERV processes should be reduced or more memory needs to be added to the server.

Determining memory usage on UNIX

Use the ps_chk_domain.sh shell script to determine memory usage on UNIX

Here is a sample output of the ps_chk_domain script

Please Enter application server domain name?? PT84

Please Enter interval in seconds?? 10

Please Enter # of interval's ?? 100 Tue Oct 23 23:00:33 PDT 2001

For all PS Processes:

Resident memory size is: 123.543 MB

CPU for PS Processes: 11 %

Virtual memory size is: 198.098 MB

For all PSAPPSRV Processes:

Resident memory size is: 108.453 MB

CPU for PS Processes: 9 %

Virtual memory size is: 106.559 MB

---

Resident memory refers to the real physical memory currently required by the process for its operation.

Virtual memory refers to the process virtual address size, which includes memory that has been paged out to the physical disk.

If the virtual memory continues to increase and the free memory of the server falls below 30%, the “RecycleCount” of the AppServer should be lowered.

The output of ps_chk_domain.sh has divided the portion of PSAPPSRV processes from the entire domain so that the end-user (or system administrator) can get a better understanding of the distribution of the memory resources.

It is recommended that the total resident memory for the entire ‘PS Processes’ should not exceed 70% of the total real memory available on the server.

Determining if excessive memory swapping happens under UNIX

Use vmstat to determine if the OS is swapping.

To check whether paging is a problem, run vmstat and check out the pi column to see if paging in is done and check the po column for paging out.

Also run sar -q to check paging; or run vmstat -s and take multiple samples to see if the page ins and outs to paging space are the majority of the total paging.

MaxInMemoryObjects

• For PeopleTools 8.16 and before We have the MaxInMemoryObjects

• Benefit:

– Keep memory footprint low

• Disadvantage:

– very CPU intensive

• Recommendation:

– maxInMemoryObjects=0

APP SERVER RECYCLE COUNT

This configurable parameter dictates the number of services after which the AppServer will automatically restart.

The AppServer processes will use physical runtime memory to cache panel objects in order to speed up user response time instead of fetching the panel objects from the database server every time.

However, as the number of new requests (new pages/components accessed) serviced by the AppServer increases, the amount of physical memory occupied by the AppServer processes also increases.

When the amount of memory occupied by the AppServer becomes too large (relative to the real memory available at the time), paging to the file system will occur and paging will impact user experience.

In order to effectively manage the memory footprint of the AppServer, keeping the “Recycle Count” at a realistic level is important.

When the AppServer reaches the specified “Recycle Count” value, the AppServer will terminate and restart itself.

When the AppServer terminates, the occupied memory will be released.

Recommendations

• It is recommended to set recycle count at 5000.

• The recycle count should be adjusted so that no memory swapping is introduced because of a high recycle count value.

• To minimize the cost of re-caching the AppServer, it is important to enable ‘File Cache’ on the AppServer. To enable server caching, set the EnableServerCaching=2 (the default value is 1 for 8.42 and below, 2 for 8.43 and above).

;----------------------------------------------------------------------- ;

EnableServerCaching - ; 0 Server File caching disabled ;

1 Server File caching limited to most used classes ;

2 Server File caching for all types

EnableServerCaching=2

;-----------------------------------------------------------------------

SHARED CACHE FOR APPLICATION SERVER

PeopleTools 8.4x has a shared cache feature in which the various psappsrv processes in an app server domain use a single set of cache files rather than separate cache files for each process.

All managed objects are included in the shared cache files so that objects are not loaded from the database.

Note: It is required to pre-load all the necessary cache objects before enabling the Shared Cache feature.

The PeopleSoft installation is supplied with an Application Engine program called LOADCACHE to assist customers in generating this pre-loaded shared cache.

Shared Cache provides the following benefits:

• Saved disk space, since all AppServer processes use the same common directory to read the file cache contents.

• Faster operation, since all the managed objects are cached into the common directory ahead of time, instead of retrieving from the database tables on demand.

To run the LOADCACHE program:

1. Make sure that the database that the application server runs against produces clean SYSAUDIT runs.

If SYSAUDIT is not clean, the LOADCACHE program may fail.

2. Make sure the user who is executing the program has the “Load Application Server Cache” Component defined to its permission list in User Profile.

Find or add the Utilities page for the user’s permission list and add the component Load Application Server Cache.

3. (For 8.42 and below only) Check psprcs.cfg file (Process Scheduler configuration file).

psprcs.cfg is where it is required to specify the type of objects to cache using the EnableServerCaching parameter.

For PeopleTools 8.4x set it to 2.

The LOADCACHE reads this setting and caches metadata according to the value specified in the Process Scheduler configuration.

Note. Do not enable shared caching for Process Scheduler.

Make sure ServerCacheMode=0.

4. In PeopleTools 8.40 or above: Select PeopleTools, Utilities, Administration, Load Application Server Cache.

5. Enter the appropriate Run Control ID (it is required to add a new value here).

The Load Application Server Cache page appears.

6. In the Output Directory specify the directory where cached metadata is to be written.

Unix example> /ds1/home/testora/pt814c1/PT814U25/appserv

NT example> c:\temp\

7. Select the correct process scheduler and click Run.

Navigate to PeopleTools>Process Monitor. Wait for the process to become “Success”. The first time it is run, the process may take 4-5 hours.

Note. Run the program with Run Location set to “Server”.

8. Shut down application server domain.

9. Enable shared caching with the ServerCacheMode parameter (ServerCacheMode=1) in psappsrv.cfg of the domain, and reconfigure the domain so that the changes are reflected.

10. Create the \\cache\share directory for the appropriate domain.

11. Navigate back to Process Scheduler. It will now have some cache generated under a new 'stage' directory. Unix example /ds1/home/testora/pt814c1/PT814U25/appserv/CACHE/stage

NT example c:\temp\CACHE\stage

12. Search/ See a bunch of files with *.dat and *.key extensions inside the stage directory. Copy the contents of the stage directory into the share directory on appserver domain.

13. Reboot application server domain.

PSAPPSRV INSTANCES

The number of PSAPPSRV instances should be kept to a low number.

PSAPPSRV needs to cache PeopleSoft meta-objects in order to be effective.

Too many PSAPPSRV process instances will make it difficult to fully cache all of them.

This is even more difficult for a Win2000/NT Tuxedo domain because the Bulletin-board process will try to use the same PSAPPSRV process id (instead of performing a round-robin like in UNIX).

In general, it is a good idea to allow queuing to happen during peak working hours.

As a simple rule, it is normal to allow about 10 user to be waiting in queue during peak load.

It is faster to wait for a busy PSAPPSRV to free up than to spawn a new PSAPPSRV.

Spawning extra PSAPPSRV instances to handle high-volume workload is not recommended.

Spawning is a relatively new feature in Tuxedo, and is not necessary to run the PeopleSoft application servers.

There is actually a high cost incurred when each new PSAPPSRV process is spawned.

Each process has to establish its cache, which takes time, CPU, and memory.

Instead, the "min" setting should be used to indicate how many PSAPPSRV processes are required for proper throughput.

To disable spawning, set the Min Instances the same as Max Instances.

6. Weblogic Setup And Configuration

Following are the list of things which needs to be checked and considered when tuning Webserver.

MINIMUM REQUIRED SERVICE PACK IS INSTALLED

PeopleSoft Platform Database lists the minimum required service pack for WebLogic Server 6.1. %WLS_HOME%/logs/log.txt specifically indicates the service pack installed.

For PeopleTools 8.40 and 8.41 Service Pack 1 should be used, for PeopleTools 8.42 Service Pack 2 should be used, and for PeopleTools 8.43 Service Pack 4 should be used.

Under normal circumstances, WebLogic should be using the "Posix Performance Pack", and this will use the native OS's socket implementation.

When the "Performance Pack" is not loaded properly, generic socket implementation will be used, which is not efficient and there could be performance issues or socket stability problems.

To verify that WebLogic Server is loading the Performance Pack correctly, check WebLogic output message and search for the reference to "NT/Posix Performance Pack".

WEBSPHERE SETUP AND CONFIGURATION

Before beginning the installation of websphere, review the PeopleSoft Platforms Database to make sure for detailed installation instructions, especially the required eFixes needed for WebSphere please refer to the “WebSphere Install” document in Customer Connection.

SET THREADCOUNT

SET THREADCOUNT is the number of HTTP actions (such as posting a HTTP request or downloading a HTTP response) that can be executed concurrently. To set this value in WebLogic, change the entry in %WLS_HOME%//config.xml as

SET THREADCOUNT in Weblogic

ThreadCount="N" /> ...

SET THREADCOUNT in WEBSPHERE

In %WAS_HOME%/config/server-cfg.xml, locate the following line:

minimumSize="n" maximumSize="N" inactivityTimeout="100" isGrowable="false"/>

where N should be based on the “peak” concurrent HTTP usage at any given point of day.

This should not include users who are logged in but not actively performing any HTTP action. To preserve system resources, administrators can fine-tune the number of ExecuteThread to be a percentage of the estimated peak value, such as N = 90% of peak usage.

One way to obtain a more accurate peak value is to monitor the thread usage via the WebLogic console.

Note: When it is required to increase the ThreadCount, more socket connections to be established. It needs to increase the number of file descriptors (maxfiles and maxfiles_lim) accordingly. To check the current file descriptor value, use the following command:

csh –c “limit –h descriptors”.

It is also required to raise the number of threads limit per process (max_thread_proc) in UNIX.

As for Windows these are implicitly limited by other system resources such as memory, and there is no explicit parameter that controls them.

SET JVM HEAP SIZE

The Java virtual machine heap space is the memory region where the Java objects (both live and dead) reside.

Set JVM heap size to 256MB or higher.

Many customers have their JVM heap size set to a minimum heap size of 64 MB and maximum size of 256 MB.

Setting the JVM heap size to a larger minimum value (preferably equals the maximum value) avoids the performance hit incurred by dynamically growing the JVM and improves predictability; it also lessens the frequency for the JVM's garbage collection (GC).

With the improved, thread-based garbage collection mechanism in JDK 1.3 the impact on workload capacity is greatly reduced when GC occurs.

It also eliminates the inefficiencies of earlier JDKs when heap size exceeds 512 MB.

See the JVM Heap Size section below to learn how to change the heap size for a specific web server.

JVM GARBAGE COLLECTION AND JVM HEAP SIZE

When the Java heap runs out of space, the Java Garbage Collector will be invoked to relocate the dead objects and free up more space for the program to continue its operation.

To monitor the amount of heap usage and the time WebLogic takes for the Garbage Collection, add the verbosegc switch to the setEnv.cmd script file.

It is required to start WebLogic from the command line -- startPIA.cmd (instead of an NT service) to see the GC output.

In setEnv.cmd, SET JAVA_OPTIONS=-hotspot –ms256m –mx256m –verbosegc

Here is a sample output of the GC:

Sat Nov 24 22:15:34 PST 2001: Invoking garbage collection

Sat Nov 24 22:15:34 PST 2001: GC: Before free/total=46867368/67108856 (69%)

= 32), weak 0, final 559, phantom 0>

JVM GARBAGE COLLECTION AND JVM HEAP SIZE in WEBSPHERE

%WAS_HOME%/config/server-cfg.xml, locate the following lines:

To keep the performance degradation from garbage collection at a minimum, users should use the command line option -noclassgc. This will inhibit a thread that would normally clear out unused classes (thus saving the load incurred by that thread).

The goals of tuning heap size are twofold: minimize the amount of time that is spent doing GC while maximizing the amount of clients that can be handled at a given time.

SET FILE DESCRIPTOR

Set OS file descriptor to 100* ThreadCount.

A file descriptor is required for every file that is opened, every *.class file read in by WebLogic, every Jolt connection PIA/Portal makes to the appserver, every connection that has to open back to a client, plus any other socket-based communication that was occurring on that machine.

To raise the file descriptors for UNIX, use the following command:

ulimit –n 4096

In Windows NT/2000 there is not an explicit parameter for the number of file descriptors. It is implicitly limited by hardware resources, mainly system memory.

SERVLET RELOAD

Dictates how often WebLogic checks whether a servlet has been modified, and if so reloads it:

SERVLET RELOAD in Weblogic

In %WLS_HOME %/< peoplesoft web domain>/config.xml there is a parameter – ServletReloadCheckSecs – that dictates how often WebLogic checks whether a servlet has been modified, and if so reloads it:

ServletReloadCheckSecs="-1" Targets="PIA" URI="PORTAL" />

In a production environment the servlets are not modified so checking and reloading would only incur unnecessary work, and thus ServletReloadCheckSecs should be set to “–1” for each of the components.

If the field is not present it defaults to 0 – always reload, which is undesirable. In that case the field should be introduced with value –1.

SERVLET RELOAD in WebSphere

The servlet reload parameters in WebSphere are located in %WAS_HOME%//[PORTAL/PSIGW/PSINTERLINKS]/WEB-INF/ibm-web-ext-xmi (usually this is the first line):

In the PSINTERLINKS/WEB-INF directory such a file may not exist. In that case create the file “ibm-web-ext.xmi” with the following content:

Also copy the file “ibm-web-bnd.xmi” from the PORTAL/WEB-INF directory.

Moreover, %WAS_HOME %/< peoplesoft web domain>/META-INF/ibm-application-ext-xmi should look like this:

< applicationext:ApplicationExtension xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:applicationext="applicationext.xmi" xmlns:application="application.xmi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmi:id="Application_ID_Ext" reloadInterval="0">

8. Conclusion

The following points are worth considering for ensuring a better performance of the Application server and web server.

1. Make informed changes
2. Keep systems up-to-date as recommended by PeopleSoft
3. Control the environment.
4. Clearly identify end-user issues
5. Watch / Monitor errors
6. Tune the environment.
7. Watch for bottlenecks that cascade or mask each other
8. Performance should be engrained in the project from the start.