1. Page 1 Queue Storage Jeff Chu | 2010-6-24

2. Page 2 Agenda Queue Storage Overview Programming Queue Storage Queue Storage tips Lab Time

3. Page 3 Queue Storage Overview MessageQueueAccount order processing customer ID order ID http://… cohowinery

4. Page 4 Queue Storage Overview Data Structure… –FIFO (First-in, First-out) –FCFS (First-come, First-serve) Unlimited message stores –Message size limits: 8KB –Base on byte count, not text length. Inter-role communication –Web to Worker communication –Worker to Web communication

5. Page 5 Queue Storage Overview Azure Queue Input Queue (Work Items)

6. Page 6 QUEUE’S RELIABLE DELIVERY Guarantee delivery/processing of messages (two-step consumption) 1.Worker Dequeues message and it is marked as Invisible for a specified “Invisibility Time” 2.Worker Deletes message when finished processing it If Worker role crashes, message becomes visible after the invisibility time for another Worker to process Azure Queue Input Queue (Work Items)

7. Page 7 2121 2121 1111 1111 C1C1 C1C1 C2C2 C2C2 Removing Poison Messages 1111 1111 2121 2121 3 3 4040 4040 Producers Consumers P2P2 P2P2 P1P1 P1P1 3030 3030 2. GetMessage(Q, 30 s)  msg 2 1. GetMessage(Q, 30 s)  msg 1 1111 1111 2121 2121 7 1010 1010 2020 2020

8. Page 8 C1C1 C 1 C2C2 C2C2 Removing Poison Messages 3 4040 4040 Producers Consumers P2P2 P2P2 P1P1 P1P1 1111 1111 2121 2121 2. GetMessage(Q, 30 s)  msg 2 3. C2 consumed msg 2 4. DeleteMessage(Q, msg 2) 7. GetMessage(Q, 30 s)  msg 1 1. GetMessage(Q, 30 s)  msg 1 5. C 1 crashed 1111 1111 2121 2121 6. msg1 visible 30 s after Dequeue 3030 3030 8 1212 1212 1111 1111 1212 1212

9. Page 9 C1C1 C1C1 C2C2 C2C2 Removing Poison Messages 3 4040 4040 Producers Consumers P2P2 P2P2 P1P1 P1P1 1212 1212 2. Dequeue(Q, 30 sec)  msg 2 3. C2 consumed msg 2 4. Delete(Q, msg 2) 7. Dequeue(Q, 30 sec)  msg 1 8. C2 crashed 1. Dequeue(Q, 30 sec)  msg 1 5. C 1 crashed 10. C1 restarted 11. Dequeue(Q, 30 sec)  msg 1 12. DequeueCount > 2 13. Delete (Q, msg1) 1212 1212 6. msg1 visible 30s after Dequeue 9. msg1 visible 30s after Dequeue 3030 3030 9 1313 1313 1212 1212 1313 1313

10. Page 10 Programming Queue Storage Get Client –CloudQueue.CreateIfNotExist() Create Message –CloudQueueMessage class –CloudQueue.AddMessage() Get Message –Get CloudQueueMessage, and change message’s property to invisible. –GetMessages() accept max 32 messages. Peek Message –Get CloudQueueMessage, but DON’T change invisible property. Delete Message –CloudQueue.DeleteMessage() –CloudQueue.Clear ()

11. Page 11 Programming Queue Storage Get queue count –RetrieveApproximentMessageCount() : must call this to update active message count. –ApproximentMessageCount Property CloudQueueMessage class –Message can archive any string or data, but length cannot over 8KB. –DequeueCount Times –ExpirationTime: Time of message will be deleted by Queue Storage. –InsertionTime: Time of message inserted into queue. –NextVisibleTime: Time of next visible time. Metadata –Queue Container support metadata (like BLOB storage).

12. Page 12 Programming Queue Storage CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); CloudQueueClient queueClient = new CloudQueueClient( account.QueueEndpoint, account.Credentials); //Create Queue CloudQueue queue = queueClient.GetQueueReference(queueName); queue.CreateIfNotExist(); //Add Message CloudQueueMessage message = new CloudQueueMessage(“some content"); queue.AddMessage(message); //Get Message message = queue.GetMessage(TimeSpan.FromMinutes(3) /*Invisibility timeout*/); // Process Message within the Invisibility Timeout //Delete Message queue.DeleteMessage(message);

13. Page 13 Queue Storage tips Messages can be up to 8KB –Use blob to store large messages, and store blob ref in message A message may be processed more than once –Make message processing idempotent Work should be repeatable and can be done multiple times Assume messages put into queue can be processed in any order For higher throughput –Batch multiple work items into a single message or into a blob –Use multiple queues Use DequeueCount to remove poison messages –Enforce threshold on message’s dequeue count Monitor message count to dynamically increase/reduce workers

14. Page 14 Queue Storage tips If your message must over 8KB… –Store your message into BLOB storage and pass the reference to queue. Producer and Consumer Problem –Delete after data consumed. –Pass reference to consumer to process finalized data.

15. Page 15 Show time for you!

16. Page 16 References Windows Azure SDK – Table Storage –http://msdn.microsoft.com/en-us/library/dd179423.aspxhttp://msdn.microsoft.com/en-us/library/dd179423.aspx Tech.Ed 2010 North America Session –COS310: Windows Azure Storage Deep Dive –http://www.msteched.com/2010/NorthAmerica/COS310http://www.msteched.com/2010/NorthAmerica/COS310 邊做邊學 Windows Azure: Table, BLOB and Queue –http://msdn.microsoft.com/zh-tw/azure/ff477598.aspxhttp://msdn.microsoft.com/zh-tw/azure/ff477598.aspx 認識 Table Storage –http://www.dotblogs.com.tw/regionbbs/archive/2010/03/ 01/vs2010.and.azure.platform.understanding.table.stora ge.service.aspxhttp://www.dotblogs.com.tw/regionbbs/archive/2010/03/ 01/vs2010.and.azure.platform.understanding.table.stora ge.service.aspx