Load Sharing & Separate Report Server Project Objective The overview should outline the business intent, purpose and scope of the project. (1-2 sentences) The purpose of this project is to immediately alleviate application performance issues by deploying multiple applications in parallel to make better use of server resources, improve stability and reduce downtime. The most common sources of performance problems are large reports, and many users logged into the application simultaneously. Both cases can consume excessive server memory and processing power in certain situations, resulting in other application services slowing down while the server works harder to manage a heavier load. Typically, one application is deployed into an Oracle Application Server(OAS) instance, and each instance has a dedicated memory allocated and can be allotted to run on specific processing 'cores' of the server. Therefore, deploying multiple applications in parallel and automatically distributing logged-in/online users evenly over these applications will reduce the memory consumed by each application, while increasing the processing power available to each instance. Similarly, large Windward Reports consume a lot of memory and processing power when building report output files. Deploying an application which only launches and manages reports, in parallel to an application which only handles online-users, will prevent resource-intensive reports from affecting online users, and allow administrators to finely tune the memory allocated to report processes. While neither of these features are mandatory to run the application, a customer can choose to use one or the other, or both as needed. For example, a larger customer may choose to have 4 load sharing application instances and one report server application instance pointing to a single database. Only one project document is needed, since both features use some common logic and configuration is similar. There are a few differences with set up, and they are noted below. Project Iterations An iteration is a deliverable outside of PD. The goal of each iteration should be stated clearly. This section should be updated to show the developer involved and the date completed. 1st Iteration - DK20110401 - Release of major changes to support load sharing & separate report engine 2nd Iteration - Proposed - Split services & watchers between application and report server appropriately 3rd Iteration - Proposed - Report Server Queue - implement maximum number of simultaneous reports 4th Iteration - Proposed - Trim unneeded code from report server to reduce EAR size 5th Iteration - Proposed - Load Sharing becomes more sophisticated Load Balancing & Clustering
Related CCARE Cases/Defects A list of the known CCARE Cases/Defects related to the project and any other background material that may be relevant and helpful. Below is a list of the CCARE cases/defects related to this project: Case Defect Cust Pri Status Pgm Problem / Enhancement 45537 PD Done Release of initial changes - Report Server 45538 PD Done Release of initial changes - Load Sharing 45595 QA A Keep getting disconnected from the system
Requirements / Features Outline the specific requirements and features that are being addressed, broken down by topic if the subject area is large. These should be tied to the iterations using 2nd, 3rd, … at the beginning if NOT in the initial iteration so that it is understood by all what will be done and when. Separate Report Server • Deploy an application instance to launch and process all reports • Automatically forward report launching, running and scheduling to correct instance • Scheduled reports must be run by Report Server instance • Report status feedback information on report forms must be maintained • Only (1) Report Server should exist for each database • Requirement is that this instance is given higher system memory (heap size) • 2nd Divvy services & watchers between application and report server appropriately • 3rd Limit number of reports to run simultaneously and schedule others (Queuing) • 4th Remove unneeded packages from report server EAR file
Load Sharing • Deploy multiple applications in parallel to evenly share load of users • Automatically forward users logins to available instances in 'Round-Robin' pattern • Do not rely on connection information saved on client - check server for every login • 5th Implement Oracle specifications to support Clustering/Load Balancing
Common Features • Feature should not be mandatory (ie) only used as needed • Number and location of application instances must be configurable • Configuration file on server file system specifies available applications • Configuration file should be monitored to allow for dynamic updating without restarting applications • Must be seamless to user - login as normal and system will forward requests between available instances
Development Approach This is an optional section to describe the steps the developer would go through in fulfilling the project goals. See Implementation section below. Implementation This section MUST be filled out by the developer. It should provide an outline of any business or technical setup required, a list of all processing considerations, and a list of all functions involved in the setup and usage of the feature. Overview We have built very configurable deployment options to take advantage of these two features. A simple configuration file can define the location and quantity of instances used, whether they exist in the same application server, or completely different machines on the network. After a new instance is deployed, modifying the configuration file is all that is needed to redirect clients to the correct application server.
Application Implementation Connections There are two types of connections made when launching EPersonality; First, the client must visit a link on a web server to download several files which include the java archive (JAR) files containing our code, JARs containing pictures, icons and configuration files, as well as a JNLP file. The JNLP file contains the information to make the second connection to the application server, through which all application communication is routed thereafter. A sample of this part of the JNLP file is included below:
<application-desc main-class="com.highlinecorp.professional.EPersonality"> <argument>PORT=12411</argument> <argument>SERVER_IP=http://phobos.highlinecorp.com:7779/ePersonality_AS06</argument>
The simplest way for us to implement load sharing and a separate report engine was to alter this connection information and route the application traffic to the desired server. In other words, most of our changes to support these features were done on the client side connection logic.
Configuration File We created a configuration file on the server which contains a comma-separated list of all available application instances as well as a report server instance. A sample of the file is included below:
- Format: reportHost=xxx|port,yyy|port
A system administrator can add to the list in this file as more instances become available, or restrict the list in this file, and changes are picked up in a matter of seconds. This can reduce downtime by redirecting clients to another instance whenever a new EAR is deployed, or if there is ever a problem with the server and it needs to be restarted.
Servlet We built a small Java Servlet into the application to 'serve up' this information to clients application when they want to connect. The servlet will iterate over the list for every login, such that the first user logging in will be directed to the first server in the list, the second user login will be directed to the second server in the list, and so on. The servlet will continually repeat through the list in an effort to distribute online users evenly over the available applications. Each client will also retrieve the report server information, in the event they wish to run a report at any time during their session. The Servlet can be tested by opening (the following sample) URLs in a web browser and refreshing to get new connection information saved in the configuration file: http://phobos.highlinecorp.com:7779/selfService_AS06/loadsharingservlet?type=ep http://phobos.highlinecorp.com:7779/selfService_AS06/loadsharingservlet?type=report
Client Side Now, when launching EPersonality, three connections are made, the first to download the JARs and JNLP, the second to connect to the servlet and retrieve information to make the third connection to the server which will handle all application communication. The second step also receives connection information to a report server, and a fourth connection may be made if the user launches a report during their session. Essentially, we have added another step between the original two connections to contact the server and get the correct connection information from the configuration file.
Set Up Deployment Types The same EAR can be used for all 3 types of deployments: Load Sharing, Report Server and Stand Alone applications. An example of a stand-alone application is one application instance to one database. With the exception of stand-alone applications, all instances must define the deployment type as one of the following JVM arguments: -Dcom.highlinecorp.deployment=application (for load sharing applications) -Dcom.highlinecorp.deployment=reportServer -Dcom.highlinecorp.deployment=standAlone (not needed, but available for future use)
Instance options Load Sharing instances can be deployed in several ways: For example, if a customer wishes to use 3 instances for load sharing, they could deploy; • One instance with 3 Java Processes (JVMs) • OR Three instances could be deployed with 1 JVM each • OR The client can mix and match (eg) 2 instances with 1 and 2 JVMs respectively An instance with multiple JVMs has the feature of reduced work when deploying or restarting an instance since the action needs to be done once by an administrator, and the Oracle AS will duplicate the work for every other JVM. However, the drawback is that JVMs cannot be restarted or redeployed individually, causing all application processes to be down (mix and match option may solve this). Report Servers must have a completely separate instance with (1) JVM. Report Servers should not have any startup cache loaded - configured in AppResources/selfServiceStyle.xml
Configuration File Location A file named connections.config must be created for every application deployment, and modified by the system administrator to reflect the available deployments. The applications listens for changes to this file and they are picked up in a matter of seconds. This file can be placed in the same root directory as the JNLP file. To override this, you may also specify a file locations as a JVM argument using:
-Dcom.highlinecorp.connections.config=/server_path/connections.config
Changes to JNLP file Since the RMI port is now retrieved from the Servlet when the client side application starts up, there is no need to add it into the JNLP file. However, if a PORT argument is in the JNLP file, the client will use it to connect to the same RMI port every time (ie) It will use the old implementation to connect to the server and no load sharing or separate report server will be accessed, even if they exist. Also, note that when using this servlet, the SERVER_IP argument has changed to use the selfService context. The relevant JNLP file portion now looks like this:
<application-desc main-class="com.highlinecorp.professional.EPersonality"> <argument>SERVER_IP=http://phobos.highlinecorp.com:7779/selfService_AS06</argument>
The JNLP file SERVER_IP argument is now used to find the connection information servlet.
File Attachment Folders If the report server is setup on a different machine, the file attachments folder must be accessible to this server as well in order to save and retrieve report output files. In this case, you must also specify the network path, rather than the local machine path to the directory on the IMST preferences tab.
Suggested Deployments Every instance can be placed on a different machine, or they can all run within the same application server. It depends on available system resources, and the cost of licensing more application servers. It is recommended the Report Server instance be given higher memory (heap size) allocation due to the nature of Windward Report processes. The minimal number of instances to take advantage of both features is three; two instances for load sharing and one for a separate report engine. The correct configuration will depend on the size of the customer database and usage. It may be possible to make an educated guess on the initial number of instances in a load sharing configuration, and then fine tune through experimentation.
Questions / Additional Notes This section should contain questions that have been or need to be asked/answered so that the project can be completed. • What are the specifications needed to support Clustering in an Oracle Application Server? • Is Clustering easier to support in a WebLogic Application Server? • Should we consider supporting more than one Report Server instance? • Can the configuration file be replaced with logic reading connection info directly from OAS? • Need to form suggested deployment configurations and memory settings (Separate Project)
Development Problems / Issues Provide an outline of all problems, issues or challenges encountered while doing the development work (or anticipated up front). Where to put Context Watchers? Since there is only one report server, and there should be only one instance of a watcher for a database, it makes sense to put the watchers in the report server instance. However, the WorkFlowLogsWatcher needs to see online users to be able to send messages, and actions to them. This is why the watchers were moved back into the main application deployment. There is no difference to 'Context Watchers' when deployed in a load sharing configuration, to multiple instances pointing to the same database (current option).
Removing configuration file: Need to research how to get connection information (opmnctl) information directly from Oracle Application Server. However, this removes option to link to other machines, and/or app servers.
Resources Released This does not need to include java classes but should include icons, style changes and documents. • Sample ePersonality.JNLP file • Sample connections.config file • Deployment descriptor web.xml • Source code changes
