1. SLC Executive Message Service
Slc-aware Ioc
SLC Executive
Message Service
Diane Fairley
11/11/2004

2. SLC Executive Slc-aware IOC Initialization The SLC Executive Thread
Startup Process
Shutdown Process
Some General Thread Issues
Utilities

3. Overall Architecture

4. slcIoc Initialization
The saIocMain.cpp file
Epics initialization via st.cmd file
Read environment variables
slcIoc name (like the microname: XL01, XL02, … )
Proxy IP address (ip address of PX01 for now)
Load epics databases
Currently a Restart/Stop subroutine record
start cmlog client
iocInit()
slcStart()
iocsh(NULL)

5. slcStart() Initializes slc globals Starts the slcExec thread
Executes the Restart function

6. SLC Globals slcThreads[ thread enum]
Array of thread information and global resources
Name
Type
Represents
pName
char *
Name of thread
tid
epicsThreadId
Thread identifier
active
epicsBoolean
True if thread is active / looping; set by thread
stop
True if thread should stop; set by slcExec
mqId
epicsMessageQueueId
Message queue identifier
Cmlog_client_h
long
cmlog client handle
saIocJobs[jobcode] = thread enum
converts micro job code to an slcIoc thread enumeration
slcSockets[ socket enum]
Array of socket descriptors
dbExists
discovered_name
Name string, from environment variable
proxy_ip_address
from environment variable
The ip address string in xxx.xxx.xxx.xxx format
Db downloadEvent semaphore

7. The SLC Executive Thread
Manages the slc startup, creating all other slc threads in order
Supervises threads and globals
Manages a clean shutdown of slc threads
Responds to slcRestart (IPL a micro)
Responds to slcStop (RESET a micro)
SlcExec thread initialization
slcCmlogOpen
Create a message queue
Set slcExec global active=True
Go into loop waiting for messages

8. slcRestart Commanded via IPL on VMS control system
TST_SLC_NOTIFY message “BOOT”
Received by msgRecv, forwarded to msgHdlr
Commanded from ioc shell
>slcRestart
Commanded from channel access via subroutine record
Sets CSTR:<micro>:1:REST.A=0,
EDM display can set this
Commanded from slcStart() initialization

9. SLC Restart

10. slcRestart Send TEST_START message to slcExec slcExec process: Phase I
From msgHdlr, or
from slcRestart iocsh command, or
from subRecord
slcExec process: Phase I
If threads are running, call slcStop
Wait for all threads to stop
Create downloadEvent semaphore for threads to wait on
Create all threads except msgRecv, dbRecv threads
Wait for these threads to set their global active=True

11. Restart: Phase I

12. slcRestart Restart Phase II Create the msgRecv and dbRecv threads
Wait for the Recv threads to set their global active=True
Send a db_download_req message to dbSend message queue.
Goes back to its loop waiting for messages and checking on thread active flags

13. Restart: Phase II

14. slcRestart Restart Phase III Database Service downloads the database
Database Service sets the dbExists flag to true
Database Service signals the downloadEvent semaphore to allow threads to proceed.

15. Ready to go

16. General Thread Initialization
slcCmlogOpen
Create resources
Some will create a message queue
Recv threads create a socket
Async threads may create a timer
Create any other thread specific resources
Set active=True
Perform dblists (optional)
Wait on database event
Go into loop waiting for messages or wake-up

17. slcStop Commanded via RESET on VMS control system
TST_SLC_NOTIFY message “RSET”
Received by msgRecv
Commanded from the ioc shell
>slcStop
Commanded from channel access via subroutine record
Sets CSTR:<micro>:1:REST.A = 1
EDM display can set this

18. slcStop Send TEST_STOP message to slcExec slcExec process
From msgHdlr, or
from iocsh command, or
from subRecord
slcExec process
Set all the thread global stop=True;
Send TEST_STOP message to all threads with message queues
Wait for all thread global “active” flags to be set false
Wait for all threads to stop
Delete the global db downloadEvent
Set dbExists to false;
Set all thread global “stop” flags back to False;
Could re-initialize all globals, do final global resource cleanup….
go back to loop waiting for messages

19. General Thread Shutdown
Release resources
msg queues,
sockets,
heap memory,
timers,
semaphores
slcCmlogClose()
set active=False
Set tid=NO_THREAD
Return

20. General slcIoc thread assumptions
There are three basic thread types
‘socket’ threads that wait at sockets
‘message queue’ threads that wait at queues
Async threads that wake up periodically
Non-message queue threads must poll the stop flag!
No ‘kill a thread’ function in epics thread library
Allows for orderly clean-up in all threads at all times
‘waits’ should have a timeout
Possibly, message queue threads may wait indefinitely at the queue because they can receive a TEST_STOP message.
Sockets should be non-blocking or interruptable

21. Some Utilities Thread Utilities slcCmlog Utilities
slcThreadNameEnum getThreadEnum(epicsThreadId tid);
slcThreadNameEnum getThreadEnumByName(const char* name);
char* getMicroname(void); eg: ‘XL02’
char* getProxyIPString(void); eg: ‘ ’
some new wait(…) utilities
To be used by threads when waiting (with a timeout) on a flag or socket or ….
slcCmlog Utilities
int slcCmlogOpen(void);
int slcCmlogClose(void);
int slcCmlogLogMsg (…,.,..,…., ...);
Message queue utilities
epicsMessageQueueId createMsgQ( slcThreadNameEnum thrd );
vmsstat_t sendToMsgQ ( slcThreadNameEnum thrd, void* msg, int size );

22. slcIoc Message Service
Threads overview
Message structure
Some Assumptions
Message Flow
Utilities

23. Message Service

24. Slc Message Service The msgRecv thread The msgSend thread
“owns” the msg service socket; creates and destroys it
Receives all incoming messages at the msg service socket
Sends the incoming message to the proper slc Hdlr thread
Uses the slcIocJobs[job code] global
The msgSend thread
Sends all outgoing message at the message service socket
“uses” the message service socket
When the connection fails, waits for the msgRecv thread to re-connect.
Checks for a valid socket in slcSockets[sdMsg]
The msgHdlr thread
Handles messages with the TEST function code
Handles the new TST_SLC_NOTIFY messages

25. Message Structure NOTE: this structure is shared with VMS and micros
Name
Data type
Represents
fwd_hdr_ts
ip_port_u
len
user
cmd
crc
ip_port_tu (int4u)
int4u
user_field_ts(int2u)
int1u
Lower half of the ip address, and the port number
Message bytecount minus this fwdheader
User defined; chunk count for large buffers;
Fwd_server command, e.g.
8 bit crc over header; currently set to 0x55
msgheader_ts
source[4]
dest[4]
timestamp[2]
func;
datalen;
char
int2u
Name of Alpha job that sent the message *
Name of destination micro *
VMS format timestamp
Function code (job code + function code)
Word count of the data in Reqdata
reqdata[]
Array of int2u
function-specific message data; packed; max size is NETVAXMSGLEN – sizeof(msgheader_ts)/2
NOTE: this structure is shared with VMS and micros

26. Message Service Assumptions
Sockets are non-blocking or interruptable
They must be able to see the global thread stop flag
Large messages will need heap memory allocations
Message queues do not (efficiently) allow variable size msgs
Use a memory pool to avoid memory leaks, fragmentation
Messages from the VMS control system are little-endian, VMS format, and packed.
The message service threads are responsible for converting the header portions of the message
The other ‘job’ threads are responsible for converting the function specific portion of the message (reqdata)

27. Message Service message flow
The msgRecv thread
Receives the message into a local inMsg variable using a getBuffer utility.
interprets the fwdheader
Allocate memory to store a multi-packet message.
converts/unpacks the msgheader into native format
replaces timestamp in msgheader
Translates the incoming job code to an slcIoc thread id
sends the entire message on to the correct queue.
Message queue send actually copies the message
Large messages: reqdata portion would include a pointer

28. Message Service message flow
The job thread
Reads the message into an inMsg variable.
Ignores the fwdheader portion
Reads the message data length from the msgheader
converts/unpack the incoming function specific data
Releases large message memory when it can, if necessary
uses the data – unidentified requirements here.
converts/packs the function specific reply into an outMsg variable.
Allocates memory for large response message
fills in the new msgheader. (leaves it in native format)
The outMsg reply is sent to the msgSend queue.

29. Message service message flow
The msgSend thread
Reads the message into a msg variable
converts/packs the msgheader
fills in the fwdheader
sends reply to VMS control system
Creates and sends multiple buffers for large messages
De-allocates memory used by large messages

30. Converting and packing/unpacking
Forward header and Msgheader
The forward header follows networking standard in VMS and micros – uses hton and ntoh
Jobs don’t need to see the forward header
Conversion and unpacking handled in by Message Service
Function Specific Data (requests and replies)
Typically a structure is defined for the data and shared by the VMS, micros and slcIocs
See nfs_ref/inc_c for xxx_reqreply_struc.*
To unpack, must memcpy data field-by-field from incoming msg.reqdata[] to a variable of this structure, then convert.
To pack, do the reverse, convert then memcpy field-by-field from structure to outgoing msg.reqdata[]
Utilities will be written for each shared structure.

31. Conversion Utilities void sswap(const unsigned short* s);
void lswap(const unsigned long* l);
void swap_int2u(void* pOut2u, const void* pIn2u);
void swap_int4u(void* pOut4u, const void* pIn4u);
void cvt_vms_to_ieee(void* pOutR4, const void* pInR4);
void cvt_ieee_to_vms(void* pOutR4, const void* pInR4);
int getVMSTime(void* pDest);
int epicsTimeToVMSTime(void* pDest, const epicsTimeStamp* eTime);
long int diffVMSTimes_10usec(void* pRef, void* pLater);
long int diffVMSTimes_10msec(void* pRef, void* pLater);

32. TCP Utilities
SOCKET getConnection(unsigned short port, unsigned short socket);
Gets a connection with the proxy
Registers with the proxy
Reports error conditions
vmsstat_t getBuffer(SOCKET sd, fwd_hdr_ts *pFwd, void *pBuf, int bytesReqd, int *pBytesRcvd);
Reads incoming message
Allocates memory and re-creates full message from multi-packet