SLC Executive Message Service Slc-aware Ioc SLC Executive Message Service Diane Fairley 11/11/2004

SLC Executive Slc-aware IOC Initialization The SLC Executive Thread Startup Process Shutdown Process Some General Thread Issues Utilities

Overall Architecture

slcIoc Initialization The saIocMain.cpp file Epics initialization via st.cmd file Read environment variables slcIoc name (like the microname: XL01, XL02, … ) Proxy IP address (ip address of PX01 for now) Load epics databases Currently a Restart/Stop subroutine record start cmlog client iocInit() slcStart() iocsh(NULL)

slcStart() Initializes slc globals Starts the slcExec thread Executes the Restart function

SLC Globals slcThreads[ thread enum] Array of thread information and global resources Name Type Represents pName char * Name of thread tid epicsThreadId Thread identifier active epicsBoolean True if thread is active / looping; set by thread stop True if thread should stop; set by slcExec mqId epicsMessageQueueId Message queue identifier Cmlog_client_h long cmlog client handle saIocJobs[jobcode] = thread enum converts micro job code to an slcIoc thread enumeration slcSockets[ socket enum] Array of socket descriptors dbExists discovered_name Name string, from environment variable proxy_ip_address from environment variable The ip address string in xxx.xxx.xxx.xxx format Db downloadEvent semaphore

The SLC Executive Thread Manages the slc startup, creating all other slc threads in order Supervises threads and globals Manages a clean shutdown of slc threads Responds to slcRestart (IPL a micro) Responds to slcStop (RESET a micro) SlcExec thread initialization slcCmlogOpen Create a message queue Set slcExec global active=True Go into loop waiting for messages

slcRestart Commanded via IPL on VMS control system TST_SLC_NOTIFY message “BOOT” Received by msgRecv, forwarded to msgHdlr Commanded from ioc shell >slcRestart Commanded from channel access via subroutine record Sets CSTR:<micro>:1:REST.A=0, EDM display can set this Commanded from slcStart() initialization

SLC Restart

slcRestart Send TEST_START message to slcExec slcExec process: Phase I From msgHdlr, or from slcRestart iocsh command, or from subRecord slcExec process: Phase I If threads are running, call slcStop Wait for all threads to stop Create downloadEvent semaphore for threads to wait on Create all threads except msgRecv, dbRecv threads Wait for these threads to set their global active=True

Restart: Phase I

slcRestart Restart Phase II Create the msgRecv and dbRecv threads Wait for the Recv threads to set their global active=True Send a db_download_req message to dbSend message queue. Goes back to its loop waiting for messages and checking on thread active flags

Restart: Phase II

slcRestart Restart Phase III Database Service downloads the database Database Service sets the dbExists flag to true Database Service signals the downloadEvent semaphore to allow threads to proceed.

Ready to go

General Thread Initialization slcCmlogOpen Create resources Some will create a message queue Recv threads create a socket Async threads may create a timer Create any other thread specific resources Set active=True Perform dblists (optional) Wait on database event Go into loop waiting for messages or wake-up

slcStop Commanded via RESET on VMS control system TST_SLC_NOTIFY message “RSET” Received by msgRecv Commanded from the ioc shell >slcStop Commanded from channel access via subroutine record Sets CSTR:<micro>:1:REST.A = 1 EDM display can set this

slcStop Send TEST_STOP message to slcExec slcExec process From msgHdlr, or from iocsh command, or from subRecord slcExec process Set all the thread global stop=True; Send TEST_STOP message to all threads with message queues Wait for all thread global “active” flags to be set false Wait for all threads to stop Delete the global db downloadEvent Set dbExists to false; Set all thread global “stop” flags back to False; Could re-initialize all globals, do final global resource cleanup…. go back to loop waiting for messages

General Thread Shutdown Release resources msg queues, sockets, heap memory, timers, semaphores slcCmlogClose() set active=False Set tid=NO_THREAD Return

General slcIoc thread assumptions There are three basic thread types ‘socket’ threads that wait at sockets ‘message queue’ threads that wait at queues Async threads that wake up periodically Non-message queue threads must poll the stop flag! No ‘kill a thread’ function in epics thread library Allows for orderly clean-up in all threads at all times ‘waits’ should have a timeout Possibly, message queue threads may wait indefinitely at the queue because they can receive a TEST_STOP message. Sockets should be non-blocking or interruptable

Some Utilities Thread Utilities slcCmlog Utilities slcThreadNameEnum getThreadEnum(epicsThreadId tid); slcThreadNameEnum getThreadEnumByName(const char* name); char* getMicroname(void); eg: ‘XL02’ char* getProxyIPString(void); eg: ‘142.67.94.25’ some new wait(…) utilities To be used by threads when waiting (with a timeout) on a flag or socket or …. slcCmlog Utilities int slcCmlogOpen(void); int slcCmlogClose(void); int slcCmlogLogMsg (…,.,..,…., ...); Message queue utilities epicsMessageQueueId createMsgQ( slcThreadNameEnum thrd ); vmsstat_t sendToMsgQ ( slcThreadNameEnum thrd, void* msg, int size );

slcIoc Message Service Threads overview Message structure Some Assumptions Message Flow Utilities

Message Service

Slc Message Service The msgRecv thread The msgSend thread “owns” the msg service socket; creates and destroys it Receives all incoming messages at the msg service socket Sends the incoming message to the proper slc Hdlr thread Uses the slcIocJobs[job code] global The msgSend thread Sends all outgoing message at the message service socket “uses” the message service socket When the connection fails, waits for the msgRecv thread to re-connect. Checks for a valid socket in slcSockets[sdMsg] The msgHdlr thread Handles messages with the TEST function code Handles the new TST_SLC_NOTIFY messages

Message Structure NOTE: this structure is shared with VMS and micros Name Data type Represents fwd_hdr_ts ip_port_u len user cmd crc ip_port_tu (int4u) int4u user_field_ts(int2u) int1u Lower half of the ip address, and the port number Message bytecount minus this fwdheader User defined; chunk count for large buffers; Fwd_server command, e.g. 8 bit crc over header; currently set to 0x55 msgheader_ts source[4] dest[4] timestamp[2] func; datalen; char int2u Name of Alpha job that sent the message * Name of destination micro * VMS format timestamp Function code (job code + function code) Word count of the data in Reqdata reqdata[] Array of int2u function-specific message data; packed; max size is NETVAXMSGLEN – sizeof(msgheader_ts)/2 NOTE: this structure is shared with VMS and micros

Message Service Assumptions Sockets are non-blocking or interruptable They must be able to see the global thread stop flag Large messages will need heap memory allocations Message queues do not (efficiently) allow variable size msgs Use a memory pool to avoid memory leaks, fragmentation Messages from the VMS control system are little-endian, VMS format, and packed. The message service threads are responsible for converting the header portions of the message The other ‘job’ threads are responsible for converting the function specific portion of the message (reqdata)

Message Service message flow The msgRecv thread Receives the message into a local inMsg variable using a getBuffer utility. interprets the fwdheader Allocate memory to store a multi-packet message. converts/unpacks the msgheader into native format replaces timestamp in msgheader Translates the incoming job code to an slcIoc thread id sends the entire message on to the correct queue. Message queue send actually copies the message Large messages: reqdata portion would include a pointer

Message Service message flow The job thread Reads the message into an inMsg variable. Ignores the fwdheader portion Reads the message data length from the msgheader converts/unpack the incoming function specific data Releases large message memory when it can, if necessary uses the data – unidentified requirements here. converts/packs the function specific reply into an outMsg variable. Allocates memory for large response message fills in the new msgheader. (leaves it in native format) The outMsg reply is sent to the msgSend queue.

Message service message flow The msgSend thread Reads the message into a msg variable converts/packs the msgheader fills in the fwdheader sends reply to VMS control system Creates and sends multiple buffers for large messages De-allocates memory used by large messages

Converting and packing/unpacking Forward header and Msgheader The forward header follows networking standard in VMS and micros – uses hton and ntoh Jobs don’t need to see the forward header Conversion and unpacking handled in by Message Service Function Specific Data (requests and replies) Typically a structure is defined for the data and shared by the VMS, micros and slcIocs See nfs_ref/inc_c for xxx_reqreply_struc.* To unpack, must memcpy data field-by-field from incoming msg.reqdata[] to a variable of this structure, then convert. To pack, do the reverse, convert then memcpy field-by-field from structure to outgoing msg.reqdata[] Utilities will be written for each shared structure.

Conversion Utilities void sswap(const unsigned short* s); void lswap(const unsigned long* l); void swap_int2u(void* pOut2u, const void* pIn2u); void swap_int4u(void* pOut4u, const void* pIn4u); void cvt_vms_to_ieee(void* pOutR4, const void* pInR4); void cvt_ieee_to_vms(void* pOutR4, const void* pInR4); int getVMSTime(void* pDest); int epicsTimeToVMSTime(void* pDest, const epicsTimeStamp* eTime); long int diffVMSTimes_10usec(void* pRef, void* pLater); long int diffVMSTimes_10msec(void* pRef, void* pLater);

TCP Utilities SOCKET getConnection(unsigned short port, unsigned short socket); Gets a connection with the proxy Registers with the proxy Reports error conditions vmsstat_t getBuffer(SOCKET sd, fwd_hdr_ts *pFwd, void *pBuf, int bytesReqd, int *pBytesRcvd); Reads incoming message Allocates memory and re-creates full message from multi-packet