How to write a MSGQ Transport (MQT) Overview Nov 29, 2005 Todd Mullanix

2 2 TDM - 11/29/05 MSGQ Module The MSGQ module is comprised of three components: – MSGQ API: Interface that applications use. They shield the application from the transports and allocators. – Allocators:Interface for allocating messages. – Transports:Interface for transporting messages between processors. MSGQ APIs Allocators IOM codec-driver Transports Drivers

3 3 TDM - 11/29/05 Transport The job of the transport is to send a message across whatever physical link to the destination message queue on another processor. The transport is composed of a transport on both sides of the physical link. Processor 0 Processor 1 MSGQ_put Physical Link MSGQ_get MSGQ_put By having a transport interface, an application can change the underlying communication mechanism without changing the application (except for the configuration of the transport). This approach hides the technical nuances of a physical link and allows more portability of an application. Msg Repos. MSGQ APIs Allocator Transport Allocator … Msg Repos. MSGQ APIs Allocator Transport Allocator … Msg Repos.

4 4 TDM - 11/29/05 What is an Transport? A transport is Library Header file parameter structure Name of interface function table (extern const MSGQ_TransportFxn MQTIOM_FXNS) Other required public structures Optionally source code Note: The MSGQ APIs interface to the transport’s library, not the user application.

5 5 TDM - 11/29/05 Transports Array The way BIOS knows about the MQTs is via the MSGQ_config variable. More specifically, the MSGQ_config.transports field. This contains all the information about the MQT instance. /* Configuration for processor 0 */ static MSGQ_TransportObj transports[NUMPROCESSORS] = { MSGQ_NOTRANSPORT, {TESTMQT_init, &TESTMQT_FXNS, &params1, NULL, 1} }; MSGQ_Config MSGQ_config = {msgQueues, /* Array of message queues */ transports, /* Array of transports */ NUMMSGQUEUES, /* # of message queues in array*/ NUMPROCESSORS, /* # of transports in array */ 0, /* 1st uninitialized msg queue */ MSGQ_INVALIDMSGQ, /* no error handler queue */ POOL_INVALIDID}; /* allocator id for errors */ The order of the transports array is by processor. The first entry communicates with processor 0, the next entry with processor 1, etc. Note: in this example, there is no entry for the first element. This is because there is no transport to oneself (e.g. processor 0).

6 6 TDM - 11/29/05 Transports Array [cont.] The following are the fields to a MQT instance. typedef struct MSGQ_TransportObj { MSGQ_MqtInit initFxn; /* Transport init function */ MSGQ_TransportFxns *fxns; /* Transport interface functions */ Ptr params; /* Transport-specific setup parameters */ Ptr object; /* Transport-specific object */ Uint16 procId; /* Processor Id that mqt talks to */ } MSGQ_TransportObj; initFxn:Initialization function for the transport fxns: Interface functions for the transport params:MQT specific instance configuration object: Holds instance state information. Filled in by when the MQT instance is opened. procId: processor id that this MQT instance talks to.

7 7 TDM - 11/29/05 MQT Functions There are 6 functions that need to be implemented: init: Initialize the physical medium. open: Initialize a MQT instance. close: Close a MQT instance. locate: Locate a message queue. release: release a located message queue. put: send an application message to the other side.

8 8 TDM - 11/29/05 BIOS call sequence When BIOS app is ran, the following are the MSGQ specific functions called. _cint_00 GBL_initFxn() BIOS_init() main() BIOS_start() Your app + BIOS scheduler

9 9 TDM - 11/29/05 BIOS_init Details Closer look at BIOS_init BIOS_init() { … if (bios.POOL.ENABLEMSGQ == true) { POOL_init() Inits some internal stuff Calls all the POOL’s init functions Calls all the POOL’s open functions } … if (bios.MSGQ.ENABLEMSGQ == true) { MSGQ_init() Inits some internal stuff Calls all the MQT’s init functions } … }

10 TDM - 11/29/05 BIOS_start Details Closer look at BIOS_start BIOS_start { … if (bios.MSGQ.ENABLEMSGQ == true) { MSGQ_startup() Calls all the MQT’s open functions } … TSK_startup /* Threads start to run in TSK_startup */

11 TDM - 11/29/05 Init Function Here is the prototype: typedef Void (*MSGQ_MqtInit)(Void); This function is called once during BIOS_init() (in MSGQ_init()), which is after cinit processing and user global init function, but before main(). The function is for initializing the hardware and setting up things for the transport. If there is a critical error, call SYS_abort(). If the transport needs system configuration information, have the user supply a _config variable of type _Config. For example in the RapidIO MQT’s header file (rapidiomqt.h): typedef struct RAPIDIOMQT_Config { Ptr bufDescAddr; … } RAPIDIOMQT_Config; And in the application code RAPIDIOMQT_Config RAPIDIOMQT_config = { (Ptr)0x02e00000, // bufDescriptorAddr … };

12 TDM - 11/29/05 Open Function Here is the prototype: typedef Int (*MSGQ_MqtOpen)(MSGQ_TransportHandle mqtHandle); This function is called once per MQT instance during BIOS_startup() (in MSGQ_startup()), which is after main(), but before threads start running. The function is for initializing the individual MQT instance (e.g. allocating resources, etc.). If there is a critical error, return an error code and SYS_abort() will be called. The order of the opening of the MQT instances is dictated by the transports array. The MQT that communicates with processor 0 is opened first, next the one that communicates with processor 1, etc. Note: for processor N, there is no transports[N] MQT.

13 TDM - 11/29/05 Open Function [cont.] State Information: Each MQT instance will generally have state information (e.g. MAC address of remote processor, rapidIO device id for remote processor, etc.). This structure can be allocated in the open function and stored in the mqtHandle->object field. All the MQT’s interface functions are passed the mqtHandle, so each function has access to this state information. params: The parameters (mqtHandle->params) that are passed into the open function are not persistent. If the instance needs to keep track of any of these values, it must store the data in its state information data structure (mqtHandle->object).

14 TDM - 11/29/05 Close Function Here is the prototype: typedef Int (*MSGQ_MqtClose)(MSGQ_TransportHandle mqtHandle); This function reverses everything that is done in the open function. Currently there is no mechanism in BIOS to call this function.

15 TDM - 11/29/05 Sync Locate If it is a synchronous locate, the MQT Send an internal sync locate request to the remote processor block waiting for a response. Use the syncAttrs->timeout as the max time to wait for a response from the other processor. If found, return the located message queue in the msgqQueue pointer. If not found, return SYS_ENOTFOUND.

16 TDM - 11/29/05 Async Locate If it is a asynchronous locate, the MQT Send an internal async locate request to the remote processor Return SYS_OK The other processor is responsible for sending the MSGQ_AsyncLocateMsg message. If there are any problems and the MQT cannot process the asynchronous locate, the function should return an appropriate error code (e.g. SYS_EALLOC).

17 TDM - 11/29/05 Locate Function Here is the prototype: typedef Int (*MSGQ_MqtLocate)(MSGQ_TransportHandle mqtHandle, String queueName, Bool sync, MSGQ_Queue *msgqQueue, Ptr locateAttrs) This function is called by either MSGQ_locate() or MSGQ_locateAsync(). Arguments mqtHandle:Transport object queueName:Name of the message queue being located sync:Synchronous or asynchronous locate msgqQueue:If sync locate: place to return the located message queue. If async locate: message queue to send the MSGQ_AsyncLocateMsg message if located locateAttrs:If sync locate: pointer to MSGQ_LocateAttrs structure. If async locate: pointer to MSGQ_LocateAsyncAttrs structure.

18 TDM - 11/29/05 Release Function Here is the prototype: typedef Int (*MSGQ_MqtRelease)(MSGQ_TransportHandle mqtHandle, MSGQ_Queue msgqQueue); This function reverses everything that is done in the locate function. It is called from MSGQ_release().

19 TDM - 11/29/05 Put Function Here is the prototype: typedef Int (*MSGQ_MqtPut)(MSGQ_TransportHandle mqtHandle, MSGQ_Msg msg); This function is called in MSGQ_put() if the message queue is on the processor that the MQT instance communicates with. Here is what the MSGQ_put() function looks like. Int MSGQ_put(MSGQ_Queue msgqQueue, MSGQ_Msg msg) { Uint16 dstProcId = msgqQueue >> 16; msg->dstId = (MSGQ_Id)msgqQueue; if (dstProcId != GBL_getProcId()) { /* Call the transport associated with this message queue */ return (MSGQ->transports[dstProcId].fxns->put( &(MSGQ->transports[dstProcId]), msg)); } The mqtHandle is the same one that was passed into the open() function, so all the state information for the instance is in the mqtHandle->object field.

20 TDM - 11/29/05 Put Functions [cont.] The put() function must be deterministic. It may require that it queues up the message internally and actually send the message in a ISR. One key item with a MSGQ message is that the first field is reserved for the MQT (and MSGQ). The MQT can use the reserved fields for whatever it wants (e.g. for placing the message on a QUE). typedef struct MSGQ_MsgHeader { Uint32 reserved[2]; /* Transport specific */ Uint16 srcProcId; /* Proc Id for the src message queue */ Uint16 poolId; /* Id of the allocator that allocated the msg */ Uint16 size; /* Size of the allocated msg */ Uint16 dstId; /* Destinaton message queue id */ Uint16 srcId; /* Source message queue id */ Uint16 msgId; /* User specified message id */ } MSGQ_MsgHeader, *MSGQ_Msg; The srcProcId field is set if the application called MSGQ_setSrcQueue(). It is set with the value of the BIOS processor id (i.e. GBL_getProcId()). The MQT can safely set the srcProcId() to GBL_getProcId() if it wants to. This comes in handy for the ISR on the other processor sometimes.

21 TDM - 11/29/05 Put Functions [cont.] Ownership: The put() function owns the message. In non-shared memory systems, the MQT is responsible for freeing the message back to the POOL. Internal Messages: There is a band of msgIds for MQTs: #define MSGQ_MQTMSGIDSSTART 0xFF80 /* Start of transport msg id range */ #define MSGQ_MQTMSGIDSEND 0xFFFE /* End of transport msg id range */ If the MQT wants to send internal messages to the other MQT on the remote processor, it can use any msgIds in the above range and be assured that the application will not be using any of these ids.

22 TDM - 11/29/05 Receiving Side Generally there is an ISR on the remote side that receives the messages (both internal and application). How it receives the messages is total transport specific. Once an application message is received, it must be placed onto the final destination via a MSGQ_put(). MSGQ_put() must be called to allow the post() function for the message queue to be called. The ISR has to construct the message queue value that is used in the MSGQ_put() function. This is done by ORing in the dstId of the message with the local processor Id shifted 16 bits. msgqQueue = ((Uint32)GBL_getProcId() dstId; MSGQ_put(status = MSGQ_put(msgqQueue, msg); How internal messages are handled is total up to the transport. Note: remember to try a keep the execution time of an ISR low. You may have to defer work to a SWI that the transport creates and manages.

23 TDM - 11/29/05 Asynchronous Errors The MQT might hit an error condition not in the context of a MSGQ call (e.g. in the ISR). It can call MSGQ_sendError(). The application can specify a message queue that will receive all async mqt errors. Here is the format of the message: typedef struct MSGQ_AsyncErrorMsg { MSGQ_MsgHeader header; MSGQ_MqtError errorType; Uint16 mqtId; Uint16 parameter; } MSGQ_AsyncErrorMsg; Here are the possible errorTypes:  MSGQ_MQTERROREXIT:MQT died  MSGQ_MQTFAILEDPUT:MQT could not send a message  MSGQ_MQTERRORINTERNAL: Generic internal error  MSGQ_MQTERRORPHYSICAL: Problem with the physical link  MSGQ_MQTERRORALLOC: Could not allocate resources

24 TDM - 11/29/05 Asynchronous Errors The MQT that logs an error should include its mqtId (mqtHandle->procId). If this is not know, use -1. The parameter is flexible. Here are some guidelines MSGQ_MQTFAILEDPUT:msgId of failed message MSGQ_MQTERRORINTERNAL: Source Line number(?) MSGQ_MQTERRORPHYSICAL: Physical link information (e.g. RapidIO uses completion codes) MSGQ_MQTERRORALLOC: Size of the allocation that failed. The MQT should document its errors why they happen in its user guide.