INFO 344 Web Tools And Development CK Wang University of Washington Spring 2014

Announcements PA2 grading = started! Many are done. – Let me know if you have any questions – Please check Canvas! For messages from me Start PA3 ASAP! – PA2 is conceptually harder but… – PA3 = lowest grade last quarter, “real server-side” – Key to success = Start Early! Ask Early!

Announcements Startup weekend! – This weekend! Highly recommended! – Help in finding jobs/internships – I’ll give you Extra Credit – Doesn’t matter where I add it but I’ll add to PA3 – +5pts if you go to demo day (Sunday) – +20pts if you participate and write code to make something! I’m going :D me a write up (100 words) +screenshot if you built a prototype!

PA2 Josh Edwards – fit everything (list/trie) spell check (steve jozs) user query Super fast, just works. Nice! I owe you lunch : )

Information Architecture on Azure Recall Amazon Web Services – EC2, S3, RDS, CloudFront, Load Balancer, etc Azure – A lot of similarities EC2 => Virtual Machines S3 => Blob RDS => SQL Azure CloudFront => App Services CDN Load Balancer = built in

New Services on Azure * either Azure only or we didn’t talk about in AWS

Compute Virtual Machine = EC2 Web Site – Easy way to host & scale websites – Very little customization Mobile Service (New) – Backend for mobile apps – Database, Auth, Push to client, Scheduled jobs

Compute Cloud Service Web Role (User-facing instance) OnStart() – initialization such as copy files such as wiki titles & read into Trie so your instance will start with everything ready! – Worker Role (off-line processing) OnStart() – initialization Run() – do work here Open VS and try this!

Storage Blob = same as S3 SQL Azure = same as RDS

Table Storage Large key-value pair – Key = {PartitionKey, RowKey} – Put different partitions on different machines = improve throughput Read more -

Choosing PartitionKey/RowKey Clustered index (unique) {PartitionKey,RowKey} Choose carefully depending on query patterns For example: Amazon Books {Name, Price, PublishDate} – Query by date (get all books from 02/10/13 – 04/10/13) PartitionKey = year, RowKey = month+date – Query by Price (get all books between USD) PartitionKey = price, RowKey = name – Query by Name PartitionKey = name – Query by Category (get all horror books) ParititionKey = category, RowKey = name

Queue Storage Message passing between – Web – Worker – Worker Off-line jobs. Off-line => async, does not block user

HDInsight Similar to AWS ElasticMapReduce Big Data processing MapReduce – programming model to process large data sets in parallel distributed across machines in a cluster Algorithm – Map Distribute input via PartitionKey to N machines – Reduce Collect the output of Map-step and combines to final output For example: – Sort 10 billion numbers – Map step, key = number, map to 10 machines, 0-1B => #1, etc. optional

HDInsight Let’s try it! Your first big-data processing? – Go to Azure portal, create HDInsight cluster, 1 data node, password = “Password344!”, WestUS, make sure you use the same storage account as where your wiki data set is for PA2 – Install Microsoft Web Platform ( – Load Windows Azure Power Shell and type in “Add-AzureAccount” and login – Type the following commands to run a job & download to /example/data/… Part 2: Run this on your wiki dataset! $subscriptionName = "azpad286JYQ7806" $clusterName = "chunkaiwinfo344" $wordCountJobDefinition = New-AzureHDInsightMapReduceJobDefinition -JarFile "wasb:///example/jars/hadoop-examples.jar" - ClassName "wordcount" -Arguments "wasb:///example/data/gutenberg/davinci.txt", "wasb:///example/data/WordCountOutput" Select-AzureSubscription $subscriptionName $wordCountJob = Start-AzureHDInsightJob -Cluster $clusterName -JobDefinition $wordCountJobDefinition Wait-AzureHDInsightJob -Job $wordCountJob -WaitTimeoutInSeconds 3600 Get-AzureHDInsightJobOutput -Cluster $clusterName -JobId $wordCountJob.JobId -StandardError $storageAccountName = "uwinfo344" $containerName = "chunkaiwinfo344-1" Select-AzureSubscription $subscriptionName $storageAccountKey = Get-AzureStorageKey $storageAccountName | %{ $_.Primary } $storageContext = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey Get-AzureStorageBlobContent -Container $ContainerName -Blob example/data/WordCountOutput/part-r Context $storageContext -Force

Questions?