Page 1 Queue Storage Jeff Chu |

Page 2 Agenda Queue Storage Overview Programming Queue Storage Queue Storage tips Lab Time

Page 3 Queue Storage Overview MessageQueueAccount order processing customer ID order ID cohowinery

Page 4 Queue Storage Overview Data Structure… –FIFO (First-in, First-out) –FCFS (First-come, First-serve) Unlimited message stores –Message size limits: 8KB –Base on byte count, not text length. Inter-role communication –Web to Worker communication –Worker to Web communication

Page 5 Queue Storage Overview Azure Queue Input Queue (Work Items)

Page 6 QUEUE’S RELIABLE DELIVERY Guarantee delivery/processing of messages (two-step consumption) 1.Worker Dequeues message and it is marked as Invisible for a specified “Invisibility Time” 2.Worker Deletes message when finished processing it If Worker role crashes, message becomes visible after the invisibility time for another Worker to process Azure Queue Input Queue (Work Items)

Page C1C1 C1C1 C2C2 C2C2 Removing Poison Messages Producers Consumers P2P2 P2P2 P1P1 P1P GetMessage(Q, 30 s)  msg 2 1. GetMessage(Q, 30 s)  msg

Page 8 C1C1 C 1 C2C2 C2C2 Removing Poison Messages Producers Consumers P2P2 P2P2 P1P1 P1P GetMessage(Q, 30 s)  msg 2 3. C2 consumed msg 2 4. DeleteMessage(Q, msg 2) 7. GetMessage(Q, 30 s)  msg 1 1. GetMessage(Q, 30 s)  msg 1 5. C 1 crashed msg1 visible 30 s after Dequeue

Page 9 C1C1 C1C1 C2C2 C2C2 Removing Poison Messages Producers Consumers P2P2 P2P2 P1P1 P1P Dequeue(Q, 30 sec)  msg 2 3. C2 consumed msg 2 4. Delete(Q, msg 2) 7. Dequeue(Q, 30 sec)  msg 1 8. C2 crashed 1. Dequeue(Q, 30 sec)  msg 1 5. C 1 crashed 10. C1 restarted 11. Dequeue(Q, 30 sec)  msg DequeueCount > Delete (Q, msg1) msg1 visible 30s after Dequeue 9. msg1 visible 30s after Dequeue

Page 10 Programming Queue Storage Get Client –CloudQueue.CreateIfNotExist() Create Message –CloudQueueMessage class –CloudQueue.AddMessage() Get Message –Get CloudQueueMessage, and change message’s property to invisible. –GetMessages() accept max 32 messages. Peek Message –Get CloudQueueMessage, but DON’T change invisible property. Delete Message –CloudQueue.DeleteMessage() –CloudQueue.Clear ()

Page 11 Programming Queue Storage Get queue count –RetrieveApproximentMessageCount() : must call this to update active message count. –ApproximentMessageCount Property CloudQueueMessage class –Message can archive any string or data, but length cannot over 8KB. –DequeueCount Times –ExpirationTime: Time of message will be deleted by Queue Storage. –InsertionTime: Time of message inserted into queue. –NextVisibleTime: Time of next visible time. Metadata –Queue Container support metadata (like BLOB storage).

Page 12 Programming Queue Storage CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); CloudQueueClient queueClient = new CloudQueueClient( account.QueueEndpoint, account.Credentials); //Create Queue CloudQueue queue = queueClient.GetQueueReference(queueName); queue.CreateIfNotExist(); //Add Message CloudQueueMessage message = new CloudQueueMessage(“some content"); queue.AddMessage(message); //Get Message message = queue.GetMessage(TimeSpan.FromMinutes(3) /*Invisibility timeout*/); // Process Message within the Invisibility Timeout //Delete Message queue.DeleteMessage(message);

Page 13 Queue Storage tips Messages can be up to 8KB –Use blob to store large messages, and store blob ref in message A message may be processed more than once –Make message processing idempotent Work should be repeatable and can be done multiple times Assume messages put into queue can be processed in any order For higher throughput –Batch multiple work items into a single message or into a blob –Use multiple queues Use DequeueCount to remove poison messages –Enforce threshold on message’s dequeue count Monitor message count to dynamically increase/reduce workers

Page 14 Queue Storage tips If your message must over 8KB… –Store your message into BLOB storage and pass the reference to queue. Producer and Consumer Problem –Delete after data consumed. –Pass reference to consumer to process finalized data.

Page 15 Show time for you!

Page 16 References Windows Azure SDK – Table Storage – Tech.Ed 2010 North America Session –COS310: Windows Azure Storage Deep Dive – 邊做邊學 Windows Azure: Table, BLOB and Queue – 認識 Table Storage – 01/vs2010.and.azure.platform.understanding.table.stora ge.service.aspxhttp:// 01/vs2010.and.azure.platform.understanding.table.stora ge.service.aspx