OPS-12: Caring for an Ailing AppServer ™ Roy Ellis Principal QA Engineer

© 2008 Progress Software Corporation2 OPS-12: Caring for an Ailing AppServer Agenda  Anatomy of the AppServer  Diagnosing Problems  Help Treating your AppServer AppServer Broker ClientsClients AgentsAgents Agent clien t Listening Port

© 2008 Progress Software Corporation3 OPS-12: Caring for an Ailing AppServer Anatomy of the AppServer What is the OpenEdge ® AppServer?

© 2008 Progress Software Corporation4 OPS-12: Caring for an Ailing AppServer Anatomy of the AppServer  Broker Java TM process Handles communication from clients to agents Manages client and agent pools Writes to.broker.log Or What is the Progress ® AppServer?.broker.log

© 2008 Progress Software Corporation5 OPS-12: Caring for an Ailing AppServer Anatomy of the AppServer  Agent “C” process Handles requests and replies to the clients Communicates “STATUS” to broker Writes to.server.log Or What is the Progress AppServer?.server.log

© 2008 Progress Software Corporation6 OPS-12: Caring for an Ailing AppServer Anatomy of the AppServer  State-aware  State-reset  Stateless  State-free 4 Operating Modes One to one Many to few - Default

© 2008 Progress Software Corporation7 OPS-12: Caring for an Ailing AppServer Anatomy of the AppServer  Client to Broker gets redirected to Agent State-aware State-reset  Client to Agent through the Broker Stateless State-free 2 Connection Models

© 2008 Progress Software Corporation8 OPS-12: Caring for an Ailing AppServer Anatomy of the AppServer 2 Connection Models State-aware State-reset clien t AppServer Broker Agent 2 Connection Models Stateless State-free

© 2008 Progress Software Corporation9 OPS-12: Caring for an Ailing AppServer Anatomy of the AppServer AppServer Broker Thread Pools AppServer Broker Listening Thread L-3090 Client Threads C-0001 C-0002 C-0003 C-0004 C-0005 C-0006 … Server Threads S-0001 S-0002 S-0003 S-0004 S-0005

© 2008 Progress Software Corporation10 OPS-12: Caring for an Ailing AppServer Anatomy of the AppServer Broker communication with Agents Broker starts agent and waits for agent to respond, status set to: STARTING Once started the agent reports to broker and broker sets status to: AVAILABLE Client request agent, is redirected to agent port, broker sets status to: CONNECTING Client connects to agent, agent reports to broker, broker sets status to: CONNECTED Agent completes request, notifies broker, broker sets status to: AVAILABLE

© 2008 Progress Software Corporation11 OPS-12: Caring for an Ailing AppServer Anatomy of the AppServer P T AS AS Application Server connected with connection id: ::asbroker1::3090::b0d80cae72::1191a17d1f0:-7ffc. (8358) P T AS 4GLTRACE Run performCardAuth "John Doe Visa -H localhost -S 6789" [Main Block - authcard.p] P T AS 4GLTRACE Run connectAuthSvc "-H localhost -S 6789 " [performCardAuth - authcard.p] P T AS 4GLTRACE Run createAuthMsg "John Doe Visa " [performCardAuth - authcard.p] P T AS 4GLTRACE Run sendAuthMsg "1000 Name=John Doe;Number= ;Type=Visa " [performCardAuth - authcard.p] P T AS 4GLTRACE Run disconnectAuthSvc "1000" [performCardAuth - authcard.p] P T AS AS Application Server disconnected. (8360) Reading Log Files P T AS AS Application Server connected with connection id: : :asbroker1::3090::b0d80cae72::1191a17d1f0:-7ffc. (8358) AS 4GLTRACE Run connectAuthSvc “-H localhost -S 6789” [performCardAuth - authcard.p]

© 2008 Progress Software Corporation12 OPS-12: Caring for an Ailing AppServer Anatomy of the AppServer  AdminServer - Required JAVA process Manages all OpenEdge Brokers and Servers Writes to admserv.log  NameServer - Optional JAVA process Traffic manager from clients to AppServers Writes to.ns.log Supporting Processes

© 2008 Progress Software Corporation13 OPS-12: Caring for an Ailing AppServer Agenda  Anatomy of the AppServer  Diagnosing Problems  Help Treating your AppServer

© 2008 Progress Software Corporation14 OPS-12: Caring for an Ailing AppServer Diagnosing Problems $DLC/bin/asbman -name asbroker1 -start OpenEdge Release 10.1C01 as of Fri May 9 21:58:30 EDT 2008 Connecting to Progress AdminServer using rmi://localhost:20931/Chimera (8280) Searching for asbroker1 (8288) Connecting to asbroker1 (8276) Unable to start asbroker1 (8297) Startup Problems

© 2008 Progress Software Corporation15 OPS-12: Caring for an Ailing AppServer Diagnosing Problems ===================================================================== P T-Main /usr1/stat/progress/101c/wrk/asbroker1.broker.log opened. P T-Main Logging level set to =2 P T-Main Log entry types activated: P T-Main 2 UB Basic ubroker version: v101c (07-May-08) (8038) P T-L UB Basic Started listenerthread: L (8043) P T-L UB Could not listen on port: 9090 (8044) P T-L UB java.net.BindException: Address already in use (errno:226) P T-L UB at java.net.PlainSocketImpl.socketBind(Native Method) P T-L UB at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359) P T-L UB at java.net.ServerSocket.bind(ServerSocket.java:325) P T-L UB at java.net.ServerSocket. (ServerSocket.java:186) Startup Problems -.broker.log

© 2008 Progress Software Corporation16 OPS-12: Caring for an Ailing AppServer Diagnosing Problems $DLC/bin/asbman -name asbroker1 -start Connecting to Progress AdminServer using rmi://localhost:20931/Chimera (8280) Searching for asbroker1 (8288) Connecting to asbroker1 (8276) Starting asbroker1. Check status. (8296) $DLC/bin/asbman -name asbroker1 -query Connecting to Progress AdminServer using rmi://localhost:20931/Chimera (8280) Searching for asbroker1 (8288) Connecting to asbroker1 (8276) Broker: asbroker1 not running (8313) Startup Problems

© 2008 Progress Software Corporation17 OPS-12: Caring for an Ailing AppServer Diagnosing Problems P T-S UB Basic Started server: /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 -logfile /usr1/stat/progress/101c/wrk/asbroker1.server.log -ubpid 435 -Ms 1 -logname asbroker1 -logentrytypes ASPlumbing,DB.Connects - logthreshold 0 -numlogfiles 3 -ASID 4 –ubpropfile /usr1/stat/progress/101c/dlc/properties/ubroker.properties -svrefresh - ipver IPv4 -db mydatabase (8108) P T-S UB Message from server : Startup Procedure Failed (8113) P T-S UB IOException reading message from server : java.io.EOFException: pipe to server broken (8117) P T-S UB Basic ERROR: cannot start server. (8100) Startup Problems -.broker.log

© 2008 Progress Software Corporation18 OPS-12: Caring for an Ailing AppServer Diagnosing Problems P T AS -- ** There is no server active for database mydatabase on -H cruz -S (704) P T AS -- Error initializing the application server. (5479) P T AS AS Application Server Shutdown. (5476) P T AS -- ** There is no server active for database mydatabase on -H cruz -S (704) P T AS -- Error initializing the application server. (5479) P T AS AS Application Server Shutdown. (5476) Startup Problems -.server.log

© 2008 Progress Software Corporation19 OPS-12: Caring for an Ailing AppServer Diagnosing Problems P T AS AS Starting application server for asbroker1. (5560) P T AS AS Application Server Startup. (5473) P T AS -- C:\tmp\ex08\roy\eg1\asstart.r Database sports2000 not connected. (1006) P T AS -- C:\tmp\ex08\roy\eg1\asstart.r startup procedure ended with an ERROR condition. (8025) P T AS AS Application Server Shutdown. (5476) Startup Problems -.server.log

© 2008 Progress Software Corporation20 OPS-12: Caring for an Ailing AppServer Diagnosing Problems  All “INITIAL” agents must start or the AppServer will fail to start Set initial agents to start to zero –“initialSrvrInstance=0” in ubroker.properties (IPv6 enabled on a machine can cause agents not to communicate with broker)  Add “–Djvmstart.debug=1” to the “jvmargs=“ in AdminServerPlugins.properties View $WRKDIR/jvmStart.log file Startup Problems

© 2008 Progress Software Corporation21 OPS-12: Caring for an Ailing AppServer Diagnosing Problems Broker Name : asbroker1 Operating Mode : State-reset Broker Status : ACTIVE Broker Port : 3090 Broker PID : Active Servers : 5 Busy Servers : 0 Locked Servers : 0 Available Servers : 5 Active Clients (now, peak) : (0, 6) Client Queue Depth (cur, max) : (0, 11) Total Requests : 124 Rq Wait (max, avg) : (259 ms, 11 ms) Rq Duration (max, avg) : (5543 ms, 24 ms) PID State Port nRq nRcvd nSent Started Last Change CONNECTED May 19, :28 May 19, : AVAILABLE May 19, :28 May 19, : AVAILABLE May 19, :28 May 19, : CONNECTED May 19, :28 May 19, : AVAILABLE May 19, :28 May 19, :28 Phantom Agents

© 2008 Progress Software Corporation22 OPS-12: Caring for an Ailing AppServer Diagnosing Problems ps -ef | grep _proapsv statqa :28:47 ? 0:24 /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 -logf statqa :28:46 ? 0:56 /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 -logf statqa :28:45 ? 0:55 /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 –log Phantom Agents protrace protrace.25322

© 2008 Progress Software Corporation23 OPS-12: Caring for an Ailing AppServer Diagnosing Problems  Look at log files.broker.log.server.log Database log files where agent is connected  Increase logging levels srvrLoggingLevel=4 (extended) srvrLogEntryTypes=4GLTrace –UBNet - New for 10.1C Diagnosing Application problems

© 2008 Progress Software Corporation24 OPS-12: Caring for an Ailing AppServer Diagnosing Problems Broker Name : asbroker1 Operating Mode : Stateless Broker Status : ACTIVE Broker Port : 3090 Broker PID : Active Servers : 1 Busy Servers : 0 Locked Servers : 0 Available Servers : 1 Active Clients (now, peak) : (0, 1) Client Queue Depth (cur, max) : (0, 1) Total Requests : 11 Rq Wait (max, avg) : (11 ms, 11 ms) Rq Duration (max, avg) : (24 ms, 11 ms) PID State Port nRq nRcvd nSent Started Last Change SENDING Apr 4, :40 Apr 4, :41 Hanging Agents

© 2008 Progress Software Corporation25 OPS-12: Caring for an Ailing AppServer Diagnosing Problems P T AS AS Application Server connected with connection id: ::asbroker1::3090::b0d80cae72::1191a17d1f0:-7ffc. (8358) P T AS 4GLTRACE Run performCardAuth "John Doe Visa -H localhost -S 6789" [Main Block - authcard.p] P T AS 4GLTRACE Run connectAuthSvc "-H localhost -S 6789 " [performCardAuth - authcard.p] P T AS 4GLTRACE Run createAuthMsg "John Doe Visa " [performCardAuth - authcard.p] P T AS 4GLTRACE Run sendAuthMsg "1000 Name=John Doe;Number= ;Type=Visa " [performCardAuth - authcard.p] P T AS 4GLTRACE Run disconnectAuthSvc "1000" [performCardAuth - authcard.p] P T AS AS Application Server disconnected. (8360) Diagnosing Agent problems

© 2008 Progress Software Corporation26 OPS-12: Caring for an Ailing AppServer Agenda  Anatomy of the AppServer  Diagnosing Problems  Help Treating your AppServer

© 2008 Progress Software Corporation27 OPS-12: Caring for an Ailing AppServer Help Treating your AppServer  -query Get status and history of AppServer broker and agents  -kill Shutdown an AppServer with busy or hanging agents May need to run twice in a row asbman

© 2008 Progress Software Corporation28 OPS-12: Caring for an Ailing AppServer Help Treating your AppServer  Available on PSDN  ABL source code available  Read and Sort multiple log files  Unsupported LogRead

© 2008 Progress Software Corporation29 OPS-12: Caring for an Ailing AppServer Help Treating your AppServer  Don’t connect using the startup parameters Any database down to a single database will cause your AppServer agents to be unresponsive  Connect using the “CONNECT” call in the srvrStartupProc= srvrConnectProc= srvrActivateProc=  Use “IF CONNECTED” ABL Database Connections

© 2008 Progress Software Corporation30 OPS-12: Caring for an Ailing AppServer Help Treating your AppServer  New in 10.1B  Described in ubroker.properties.README  Read the CAUTION! Use only when directed by Tech Support  Periodically force the shutdown of all agents and allow new agents to start when requested Lifespan

© 2008 Progress Software Corporation31 OPS-12: Caring for an Ailing AppServer Help Treating your AppServer OpenEdge Management

© 2008 Progress Software Corporation32 OPS-12: Caring for an Ailing AppServer D I S C L A I M E R Under Development  The next slide includes information about potential future products and/or product enhancements.  What I am going to say reflects our current thinking, but the information contained herein is preliminary and subject to change. Any future products we ultimately deliver may be materially different from what is described here. D I S C L A I M E R

© 2008 Progress Software Corporation33 OPS-12: Caring for an Ailing AppServer Help Treating your AppServer  New in 10.2A  Described in ubroker.properties.README  List of all clients known to the AppServer, includes: Host/IP Port Connected Agent More! List AppServer Connections

© 2008 Progress Software Corporation34 OPS-12: Caring for an Ailing AppServer Help Treating your AppServer  Technical Support  Knowledge Base (kbase)  PSDN Help from Progress

© 2008 Progress Software Corporation35 OPS-12: Caring for an Ailing AppServer AppServer Broker In Summary  Understand your AppServer  Diagnose your Problem  Care for your Ailing AppServer Agent clien t Listening Port ClientsClients AgentsAgents

© 2008 Progress Software Corporation36 OPS-12: Caring for an Ailing AppServer Relevant Exchange Sessions  OPS-15: What was Happening with my Database, AppServer, Operating System… Yesterday, Last Month, Last Year?  SOA-38: OpenEdge Middleware Road Map and Info Exchange  OPS-19: What is IPv6 and Why Should I care?  OPS-13: Building and Deploying a Highly Available Application

© 2008 Progress Software Corporation37 OPS-12: Caring for an Ailing AppServer Questions ?

© 2008 Progress Software Corporation38 OPS-12: Caring for an Ailing AppServer Thank You

© 2008 Progress Software Corporation39 OPS-12: Caring for an Ailing AppServer