draft-ietf-ips-iser-00 Mike Ko November 8, 2004 iSER Draft Status draft-ietf-ips-iser-00 Mike Ko November 8, 2004

iSER Flow Control for Control-Type PDU As currently defined, the iSER protocol does not provide additional flow control beyond that provided by the iSCSI layer on control-type PDUs From section 8.1: “The iSER Layer SHOULD provision enough Untagged buffers for handling incoming RDMAP Send Message Types to prevent a buffer underrun condition” iWARP Verbs mechanisms such as the Shared Receive Queue mechanism can be used to effectively address the Send Message flow control question Most iSCSI PDUs are paced by the CmdSN window mechanism in iSCSI Question: Should some form of send side flow control be established for iSCSI control-type PDUs not regulated by the CmdSN window? 11/8/2004 M. Ko

Flow Control Alternatives for Unexpected PDUs Require the receiver to replenish buffers faster than the arrival rate of incoming PDUs including the unexpected ones Imposes a constraint on the implementation that may not be achievable in all circumstances Require the RNIC to do graceful handling on lack of receiver buffer situations Mandates the implementation of an optional feature Declare a limit on the number of unexpected PDUs that can be sent If architected correctly, can also accommodate options 1, 2, and 5 11/8/2004 M. Ko

Flow Control Alternatives for Unexpected PDUs Define an explicit credit based flow control mechanism Basically a receive side flow control Extra overhead even when options 1, 2, and 5 are used Require the RNIC to use Shared RQ Mandates the implementation of an optional feature Drop the link if there is buffer overrun Default solution if no other flow control alternatives Drastic solution compared with other alternatives 11/8/2004 M. Ko

Declared Limit on Sender Side on the Number of Unexpected PDUs First discussed on the IPS reflector in July/August of 2003 between Mike Ko, Caitlin Bestler, David Black, etc. Idea revived and refined in October 2004 A new key, MaxOutstandingUnexpectedPDU, can be used by both the initiator and the target to declare the maximum number of outstanding “unexpected” control-type PDUs it can receive Key has session wide scope Default is none “None” works for options 1, 2, and 5 If not “none”, is there a good default which is not implementation dependent? 11/8/2004 M. Ko

iSCSI Control-Type PDUs from the Initiator Regulated by CmdSN Window If the Initiator sends this PDU It must provision a buffer for this PDU from the Target SCSI Command SCSI Response or Reject before the task is active Task Management Function Request Task Management Function Response Text Request Text Response Login Request Login Response NOP-out (request) ITT /= 0xffffffff TTT = 0xffffffff NOP-in (response) TTT =0xffffffff 11/8/2004 M. Ko

iSCSI Control-Type PDUs from Initiator with Known Buffering Requirement at Target SCSI Data-out PDU For unsolicited data, the iSER layer at the target can use FirstBurstLength to determine the amount of buffering required Solicited data is handled using RDMA Read Response Data cannot be “solicited” since an R2T is always transformed into an RDMA Read Request and is never sent as is by the target Section 7.3.4 to be updated to remove references on sending “solicited” SCSI Data-out PDUs NOP-out PDU with ITT = 0xffffffff and TTT /= 0xffffffff (ping echo) which is sent as a response to a NOP-in PDU from the target 11/8/2004 M. Ko

iSCSI Control-Type PDUs from the Initiator Subject to Declared Limit Any PDU with the immediate flag set For the PDUs listed on slide 6, the PDU is outstanding until the target responds with the corresponding response PDU For NOP-out PDU with ITT = TTT = 0xffffffff, the PDU is outstanding until the target responds with a control-type PDU with ExpCmdSN set to at least x + 1 where x is the CmdSN of the PDU with the immediate flag set NOP-out PDU with ITT = 0xffffffff and TTT /= 0xffffffff (ping echo) is not subject to the declared limit since it is sent as a response to a NOP-in PDU from the target SNACK PDU Not needed in iSER If sent, PDU is outstanding until the target responds with a SCSI Response PDU for the referenced command 11/8/2004 M. Ko

Potential Problem with Overuse of Unidirectional NOP-out When the target sends a PDU with ExpCmdSN set to x+1, up to MaxOutstandingUnexpectedPDU unidirectional NOP-out PDUs with ITT = TTT = 0xffffffff with CmdSN = x could be in flight from the initiator Upon receiving the PDU from the target, the initiator can send MaxOutstandingUnexpectedPDU unidirectional NOP-out PDUs with CmdSN = x+1 To guard against this pathetic case, the target needs to provision the number of buffers equal to twice MaxOutstandingUnexpectedPDU in this situation for the unexpected PDUs The number of buffers for unexpected PDUs can be reduced to MaxOutstandingUnexpectedPDU when the initiator sends a non-immediate PDU with CmdSN = x+1 Recommend that we add a cautionary note instead against the overuse of the unidirectional NOP-out 11/8/2004 M. Ko

iSCSI Control-Type PDUs from the Target Subject to Declared Limit The following PDUs are outstanding until the initiator sends a control-type PDU with ExpStatSN set to at least x + 1 where x is the StatSN of the “unexpected” PDU Asynchronous Message PDU Reject PDU sent after the task is active NOP-in PDU initiated by the target with ITT = TTT = 0xffffffff A NOP-in PDU sent as a ping request by the target with ITT = 0xffffffff and TTT /= 0xffffffff is outstanding until the initiator responds with a NOP-out PDU (ping echo) 11/8/2004 M. Ko

Potential Problem with Overuse of Unidirectional NOP-in For a NOP-in PDU from the target with ITT = TTT = 0xffffffff, StatSN is not advanced When the initiator sends a PDU with ExpStatSN set to x+1 where x is the StatSN of this NOP-in PDU, up to MaxOutstandingUnexpectedPDU unidirectional NOP-in PDUs with ITT = TTT = 0xffffffff with StatSN = x could be in flight from the target Upon receiving the PDU from the initiator, the target can send MaxOutstandingUnexpectedPDU unidirectional NOP-in PDUs with CmdSN = x+1 To guard against this pathetic case, the initiator needs to provision the number of buffers equal to twice MaxOutstandingUnexpectedPDU in this situation for the unexpected PDUs The number of buffers for unexpected PDUs can be reduced to MaxOutstandingUnexpectedPDU when the target sends a PDU with StatSN = x+1 Recommend that we add a cautionary note instead against the overuse of the unidirectional NOP-in 11/8/2004 M. Ko