WW TSS-13 Wonderware MES Support and Troubleshooting Technique Presenters: Dan Scott Jim Stella JS - This technical session will discuss the various methods and tools that can be used to support and troubleshoot Wonderware MES. Learn about preventative maintenance activities and some best practice tuning and architecture considerations.

Session Agenda MES Architecture Overview Middleware Technology Changes 3.5 to 4.x Database Tables Tips and Common Mistakes Troubleshooting JS

MES Architecture Overview DS

Wonderware MES Architecture MES Client Operator MES Client Supervisor InTouch w/ .NET Controls Supervisor Galaxy Repository Reports Configurator InTouch w/ MES API Field I/O ArchestrA IDE Supervisory (P) Shop Floor (P) MES Admin (P) AOS Platform (M) Factory Connector Server (M) Operations Capability Obj Utilization Capability Obj Middleware Server (M,S) Sample Recording Obj DS – Typical multi-node architecture. Discuss various components. Locations of proxy, server, and MES Service. Poor architecture planning up front can seriously impact the success of your project Reports Server (M or None) Database Server App Objects w/ MES API Key: M = Full Middleware P = Middleware Proxy S = MES Service

Technology change in MES Middleware from 3.5 to 4.x JS

Wonderware MES Middleware 3.5 JS – COM+ vs WCF. DLLHost memory spikes, now more traceability with new middleware.

Wonderware MES Middleware 4.x WCF (Windows Communication Foundation) JS – 3.5 was a proxy which required Msi and cab file created from server. So, you had proxy or server, not both. Now “proxy” is really a client. You now have either client or client and server locally. More ability to resolve client performance and connection issues. You can test mw connection, but does not mean 100% connectivity.

MES Database Tables JS

Common MES Database Tables Used in Troubleshooting Sessn – User sessions Who is logged into which session ID, or not in some cases Records with heartbeat older than 10 minutes can be purged These should be auto-purged by MES Service if it is running Only ever one session type 53 – Background User for MES Service Orphaned type 53 record can stop MES Service from starting Tech Note 699 covers this table and the client type in more detail Ent_Logon – User logged on entities Which user is logged into which entities User must have privilege for those entities JS

Common MES Database Tables Used in Troubleshooting Item_Prod – Production records Distinct: Default selection Can report hourly or shift OEE Records stored for every production or consumption transaction Can result in large record count Shift: Can report shift OEE only Most compact record count Hourly: New for 3.5 p02, MES 4.0 SP2+ (replaces shift) Compromise between Distinct and Shift option Should be purged periodically to maintain high performance JS

Common MES Database Tables Used in Troubleshooting Util_Log – Utilization history Should be purged periodically to maintain high performance TPM_Stat – OEE raw data Data only stored when a job is running Job – Jobs and related status Contains total production count, allowing item_prod details to be purged Job_Exec – Active job information One placeholder record per allow Job Exec per entity Never delete records JS

Common MES Database Tables Used in Troubleshooting Util_Exec – Current utilization information Lookup current raw reason OEE_Exec – Current OEE information See current Quality and Performance OEE subcomponents Quality % x Performance % x Availability % = OEE % Ent – Entities View current shift ID and times View capabilities JS

Common MES Database Tables Used in Troubleshooting Shift_to_go – Upcoming shifts Can be purged, then restart MES Service to rebuild list Tz_offset – Pre-calculated time zone offsets from UTC Only time zones of interest are calculated Purge is required if the database is moved to another time zone WO – Work Orders Should be purged at some point. (Archive/Purge has System parameters that can be defined to automatically remove the data) Item_inv – Inventory JS

MES Tips and Common Mistakes DS

MES Tips and Common Mistakes: Middleware Middleware proxy must be installed on IDE node if using Entity Model Builder (EMB) You must open/close a capability object editor window before running EMB Full middleware should be installed App Object and Manager nodes License purchase required per full middleware server (multiple inc.) Middleware is not deployed, must be installed manually Use Proxy Configuration Editor to test the connection to MW server Use MES Client or Configurator to test client connectivity since you will get better errors If middleware is remote from database, MES Client must be installed DS

MES Tips and Common Mistakes: Load Management Server / Platform Loading What else runs on this node? Available system resources? Engine Loading Engine Scan Rate? Many capability object functions result in MES database transactions. Consider separate engines for MES objects versus regular, higher speed, App Objects Middleware Loading Capability Object count? No more than 200-300 per platform* No more than 50 SROs per platform* No more than 30-50 per engine* Full middleware on the platform hosting MES App Objects Reporting? Database Loading Dedicated MES Database server? MES has high transaction count DS- Consider this when developing an implementation plan. We see a lot of systems where Dev testing went well, but once record counts and transaction counts get high in Prod, the system runs very slowly or with errors. Consider how you configure MES related to data storage, examples are rates of recording of production, raw reason code change frequency, etc. Disk IO capability important in VM environment. Refer to deployment guide. *Suggestions dependent on transaction volume and scan rate

MES Tips and Common Mistakes: Maintain Distinct Production Records Enabled to report Hourly OEE Daily/Shift OEE can still be reported with this disabled This setting can have a significant impact on the MES database size If enabled (default), a new item_prod record is created every time production is recorded If disabled, the system will update existing item_prod records New feature in 3.5 p02, 4.0 SP2 aggregated hourly prod records JS - Do you really need this? Pros: hourly reporting, more granular audit data, faster backups Cons: BIG item_prod table, slower reports, more hardware demands, slower backups, more db maintenance/monitoring reqd Considerations can be made to turn this on, but use slower rates of production recording

MES Tips and Common Mistakes: Multiple Job Execution How many Jobs can be run concurrently on each entity More than 10 can cause problems OCO has a hard cap of 50, but not recommended for entities with large transaction numbers JS – Handling multiple jobexec requires handling large amount of transactions. Can be handled incorrectly. Operator and job summary will appear differently.

MES Tips and Common Mistakes: OCO/UCO Production Counters Misconfiguration, especially with a rollover counter, can be disastrous NEVER set Deadband, Max Value, and Update Interval to 0 Production may be recorded every engine scan Every time production is recorded, you are transacting with the MES Database If Maintain Distinct Production Records is enabled (default), you are creating a new item_prod records every time production is recorded UCO rollover counter Max Value MUST be synced with PLC rollover OCO rollover counter can be reset on demand, or PLC sync (TN 710) DS - New in MES 4.0, OCO counters are significantly better than in the UCO. Multiple Job Exec, resets, etc. UCO counters can ONLY be used when there can only be 1 job run at a time run on an entity

MES Tips and Common Mistakes: Raw Reason Code Management (UCO) Every Raw Reason Code change results in multiple MES Database transactions Changing reasons too fast can cause reason codes to be missed (TN657) UCO can only handle current change plus queue the next change Micro stops should be recorded to Historian (events < 30-60s)* Future product plans in the works to allow for faster changes DS - Most common bottleneck since v3.5 was released One of the key understanding of MES is that it is NOT for microstops at this time Occasional fast changes are OK, but back-to-back-to-back fast changes will result in skipping RRC records Minimum time before activation field must be used when dealing with a large list of RRCs Consider the number of RRC being defined. How granular do you really to be here versus historian? RRC priority is important. *Suggestion dependent on transaction volume and scan rate

MES Tips and Common Mistakes: Continuously Running Work Order “But we’re only using Performance, we don’t care about Work Orders” The underlying MES database structure and table relationships makes Work Order management important Problems with continuously running work orders: Database performance degrades over time Updating database records require the system to query large datasets Archiving capabilities are limited (ex: Archive WO where closed > 30 days) General recommendation is a new Work Order daily or at shift change DS - A top 3 common problem. We are talking about the Work Order, not just the Job

MES Tips and Common Mistakes: Work Queue Grid Filtering The more that you try to display on the Work Queue grid, the slower it will load and refresh How many Jobs does an operator really need to see? Use New/Ready status to limit records Create jobs as New, have a supervisor set them to Ready when they assign the work. Hide New jobs from the operator’s work queue. Eliminates operator confusion DS - From implementation diagram, this impacts Supervisory users and Shop Floor Users The Queue control can become very slow with large record sets General recommendation is to limit to about 10 jobs depending on the number of entities the user is logged into Use Deploy Queue Filter and Sequence function to set the filter

MES Tips and Common Mistakes: Utilization Grid Filtering The more that you try to display on the Utilization grid, the slower it will load and refresh No filter will query the database for ALL utilization records for this entity What does the operator really need to see vs. what can just show up on a report? (right-click) DS - Save Configuration: Saves the filter for yourself on this entity Save Config for All Logged On Entities: Saves the filter for yourself for all logged in entities Save as Default Configuration: Saves the filter for all users (without their own filters) on this entity only Reload configuration: If made changes you have not saved, you can reload your saved fitler Revert to Default: If made changes you have not saved, you can reload the default filter

MES Tips and Common Mistakes: Security If utilizing OS-based security, the Wonderware Change Network User account must be a domain user Special steps required to enable OS Group Security - Tech Note 708 Lost Database Connection Editor password Delete C:\Program Files\Wonderware\ MES\Middleware\Middleware.cfg DS

MES Tips and Common Mistakes: OEE Calculation JS - How OEE is calculated. Larger record sets queried by these calculations will slow the system

MES Tips and Common Mistakes: Data Inconsistencies Moving database to a new timezone? Purge shift_to_go and tz_offset tables Restart MES Service Errors that Job exec limit exceeded, but Operator work queue does not show that many jobs running Job state and Job_Exec current job are out of sync JS - Often overlooked, but can be very important Maintenance intervals depend on number of records being written Good tables to monitor if maintenance is needed are item_prod and util_log Backups are being done, but truncating transactions logs and db/file shrinking is often not being done

MES Tips and Common Mistakes: What does MES Service do? Only one MES Service can run per database Executes minutely, hourly, and daily housekeeping tasks Shift changes Session clean-up Auto-ready jobs Run scheduled Supply Chain Connector tasks Run scheduled archive and purge tasks Generate time-based Quality Samples JS -

MES Tips and Common Mistakes: System Maintenance Database maintenance Truncate transaction logs and shrink files and database (TN 664) Archive and/or Purge Large tables: item_prod (Production details) util_log (Utilization states) tpm_stat (OEE stats) Custom reports can union data from Prod and Archive databases Available from MES Services, not part of the standard product Tech Note 746 - Using the New MES 4.0 Archive, Purge and Restore Functions JS - Often overlooked, but can be very important Maintenance intervals depend on number of records being written Good tables to monitor if maintenance is needed are item_prod and util_log Backups are being done, but truncating transactions logs and db/file shrinking is often not being done

MES Tips and Common Mistakes: Supply Chain Connector MES Database File Xml or delimited Supply Chain Connector (MES Service) Middleware Table 3rd party or custom Supply Chain Connector can have data brought in by either data file or database Table. JS

MES Tips and Common Mistakes: Supply Chain Connector Questions to consider when deciding whether to use files or tables: How much data will I be importing? How often will the data be imported? How much of the importing data will be new and how much will be updating existing records? What will be creating the new data that will be imported? JS

MES Tips and Common Mistakes: Supply Chain Connector File Base imports: Pros Easy to implement Good for smaller amounts of data Easy one time configuration scenarios Cons Cannot track the processing of individual records Higher chances of data integrity issues Slower performance with larger amounts of data JS

MES Tips and Common Mistakes: Supply Chain Connector Table Base imports: Pros Able to track and keep information on individual processed records Reprocessing of individual records is easier Troubleshooting bad records is easier to find. Cons More difficult to implement Better for larger amounts of data. Needs more maintenance (cleaning out old records, backing up of tables….) JS

MES Troubleshooting Tools and Techniques DS

MES Troubleshooting: Middleware Licensing MES Licenses must be installed on Middleware servers (restart MW) Performance-only or Quality-only license can use OCO for production counts and basic job execution MES 4.0 license is valid for MES 4.5 Database Connection Editor (restart Middleware if changed) Verify SQL connectivity Middleware Configuration Editor Enable XML Trace of all middleware/SP calls with parameters (restart MW) Middleware Proxy Configuration Editor Test connection to Middleware server DS

MES Troubleshooting: Error Messages MES SMC Log Flags UCO – EquipmentStatusRuntime1 OCO – JobExecutionRuntime1 SRO – SampleCharacteristicRuntime1 Entity Model Builder – EntityModelBuilder MES Service – FactMES.Server.Services MES ArchestrA Objects .ErrorMessage attribute DS

MES Troubleshooting: MES Client Debug Modify C:\Program Files\Wonderware\MES\Client Framework Modules\aaClient.exe.config Change this section: <!-- Uncomment to enable the file based trace listener. <add name="LogTraceListener" type="System.Diagnostics.TextWriterTraceListener“ initializeData="Trace.log" /> --> <!-- Uncomment to enable the file based trace listener. --> <add name="LogTraceListener“ type="System.Diagnostics.TextWriterTraceListener“ initializeData="Trace.log" /> DS

Useful Tech Notes Tech Note 753 - Resolving MES Entity Model Builder Errors on a Remote IDE Development Node Tech Note 746 - Using the New MES 4.0 Archive, Purge and Restore Functions Tech Note 744 - Resolving Incorrect Shift Times When MESDB is Moved to Another Time Zone Tech Note 723 - Solving Wonderware MES 4.0 Known Installation Issues Tech Note 711 – Wonderware MES Operation Capability Object: Configuring Multiple Job Counters

Useful Tech Notes Tech Note 710 - Wonderware MES Operations Capability Object: Configuring Rolling Counter Reset Tech Note 708 - Enabling MES 4.0 Operating System (OS) Group Security Tech Note 699 - Wonderware MES Sessn Table and Client Types Tech Note 664 - Truncating Operations & Performance Transaction Logs Tech Note 657 - Wonderware Performance 3.5 Software: Performance Tips

Other Useful Documents MES 4.0 Deployment Guide MES Database Diagrams MES 4.0 Development Guide

Useful Sessions WW TSS-09: MS SQL Server Performance & Tuning for Wonderware Database

Questions? THANK YOU