Lecture IV: REST Web Service with Google App Engine CS 4593 Cloud-Oriented Big Data and Software Engineering

Outline Course Project Google App Engine Google Development Console Google Custom Search API 2

Course Project Topic –Keyword trends analysis services for a certain topic Platform –Google App Engine –Any programming language supported by GAE Collaboration –Work individually –Hopefully you are not working on the same idea, but it is fine if you happen work on one by accident 3

Course Project Requirements –Develop a web service based on Google Search API to follow the weekly trends of keywords on a certain topic –Search one or multiple websites each day with Google Search API free quota and record the query results during a period of time (hopefully more than a week) –Perform statistics on query results to provide daily keywords frequency and the trends of the week 4

Course Project Requirements –Provide at least two web services: one for the keyword frequency of each day, and the other for trend of the week –Provide output as either JSON or XML –The developed web service should be deployed on Google App Engine 5

Course Project Advanced Requirements –You are going to choose the topic and the website to search, but the results should be meaningful (searching a static website for a week will not give you any different results) –Provide HTML presentation for the web services 6

Course Project Additional Information –Feel free to use additional free web services to support your task –Searching a more volatile website often provides you more interesting results, e.g., searching "Obama" in CNN or NY Times. 7

Course Project Evaluation –fetching and recording Google Search API results (2.5 points) –extracting keywords (2.5 points) –presentation of daily results (2.5 points) –presentation of weekly results (2.5 points) –The advanced features will be evaluated by quality. 8

Course Project Deliverables –Demo –Code –Commends and User Guide –Blackboard Due date –Demo: Oct. 5th –Code: Oct. 4th 9

Outline Course Project Google App Engine Google Development Console Google Custom Search API 10

Google App Engine: Overview Google App Engine (GAE) is a PaaS cloud computing platform for developing and hosting web applications in Google-managed data centers. Google App Engine lets you run web applications on Google's infrastructure. –Easy to build. –Easy to maintain. –Easy to scale as the traffic and storage needs grow 11

infrastructure vs. platform - What is “The Platform”? Platform: same for all applications Libraries: shared by multiple applications Application-specific code infrastructure: hidden by platform

Scaling Low-usage apps: many apps per physical host High-usage apps: multiple physical hosts per app Stateless APIs are trivial to replicate Datastore built on top of Bigtable; designed to scale well –Abstraction on top of Bigtable –API influenced by scalability No joins Recommendations: denormalize schema; precompute joins 13

Google App Engine: Overview Free –1GB storage –1 virtual machine instance –5 million page views / month –10 applications / Google account 14

Preserving Fairness Through Quotas Everything an app does is limited by quotas, for example: –request count, bandwidth used, CPU usage, datastore call count, disk space used, s sent If you run out of quota that particular operation is blocked (raising an exception) until replenished 15

Preserving Fairness Through Quotas Free quotas are tuned so that a well-written app (light CPU/datastore use) can survive The point of quotas is to be able to support a very large number of small apps (analogy: baggage limit in air travel) Large apps can request raised quotas Three types of quotas –Free Quota –Billed Limits –Safety Limits 16

Google App Engine: Programming Languages Java –App Engine runs JAVA apps on a JAVA 7 virtual machine (currently supports JAVA 6 as well). –Uses JAVA Servlet standard for web applications: WAR (Web Applications ARchive) directory structure. Servlet classes Java Server Pages (JSP) Static and data files Deployment descriptor (web.xml) Other configuration files 17

Google App Engine: Programming Languages Python –Uses WSGI (Web Server Gateway Interface) standard. –Python applications can be written using: Webapp2 framework Django framework Any python code that uses the CGI (Common Gateway Interface) standard 18

Google App Engine: Programming Languages PHP –Local development servers are available to anyone for developing and testing local applications. –Only whitelisted applications can be deployed on Google App Engine. ( 19

Google App Engine: Programming Languages Go –Go is an Google’s open source programming environment. –Tightly coupled with Google App Engine. –Applications can be written using App Engine’s Go SDK. 20

Google App Engine: Data Storage App Engine Data Store –NoSQL schema-less object based data storage, with a query engine and atomic transactions. –Data object is called a “Entity” that has a kind (~ table name) and a set of –properties (~ column names). –JAVA JDO/ JPA interfaces and Python datastore interfaces. 21

Google App Engine: Data Storage Google Cloud Store –RESTful service for storing and querying data. –Fast, scalable and highly available solution. –Provides Multiple layers of redundancy. All data is replicated to multiple data centers. –Provides different levels of access control. –HTTP based APIs. 22

Google App Engine: Services Google Services –URL Fetch –Mail –Memcache 23

Google App Engine: Security Sandboxes –All hosted applications run in a secure environment that provides limited access to the underlying operating system. –Sandbox isolates the application in its own secure, reliable environment that is independent of hardware, operating system and physical location of a web server. 24

Google App Engine: Security Sandboxes: Limitations –Application can only interact with other computers over internet using URL fetch, , and HTTP/ HTTPS requests on the standard ports –Applications cannot write to local file system in any of the runtime environments. –Application code runs only in response to a web request, a queued task or a scheduled task and must return the response data within 60 seconds. 25

Google App Engine: Demo Create a project on Google App Engine –At Initialize the environment of Restlet + eclipse + GAE –Tutorial at – framework/guide/2.3/introduction/first-steps/first- applicationhttp://restlet.com/technical-resources/restlet- framework/guide/2.3/introduction/first-steps/first- application –Download Restlet-GAE, GAE SDK, and Eclipse-GAE plugin 26

Google App Engine: Demo Deployment of Applications –Follow e?_ga= Google Search API –Generate your search engine –Manage API usages – 27

Google Map API Provide Map Services –Http: –Android –IOS –JavaScript Parameters –Center –Zoom –Size –Language 28

Google Map API Pricing –Free Quota: 2500 / day –Bill Quota: 1000 / $0.5 –Safety Quota: 100, 000 / day 29

Google Place API Searching for places around certain map location Usage Example –Finding all food places within 500 miles from (long,lat) – earch/json?location=long,lat&radius=500&types=food &name=cruise&key=API_KEYhttps://maps.googleapis.com/maps/api/place/nearbys earch/json?location=long,lat&radius=500&types=food &name=cruise&key=API_KEY Place ID –A ID of a place that can be used later (e.g., Costco at I UTSA Boulevard) 30

Google Place API Parameters –Type: food, airport, hospital, … – shttps://developers.google.com/places/supported_type s Pricing –Free quota: 1000/day –Free up to 150,000/day with credit card info 31

{ "geometry" : { "location" : { "lat" : , "lng" : } }, … "name" : "Rhythmboat Cruises", "opening_hours" : { "open_now" : true }, "photos" : […], "place_id" : "ChIJyWEHuEmuEmsRm9hTkapTCrk", "scope" : "GOOGLE", "types" : [ "travel_agency", "restaurant", "food", "establishment" ], "vicinity" : "Pyrmont Bay Wharf Darling Dr, Sydney" }, Google Place API: response

Google+ API Intro Usages Parameters Pricing 33

Google+ API A series of APIs working with Google+ –People: Get public profiles with query or user ID –Activities: List activities of a user or searching activities with query –Comments: List comments of an activity or searching comments with query –Moments: Retrieve, insert or delete moments Usage – –10,000/day free quota 34

Google Translation API Translate strings to other languages Parameters –source –target –q(query) Pricing –$20 for 1M characters –50M / day limit 35

Youtube Data API Provide various supports to retrieve and manage data from Youtube Some important services –Video: List, search, Insert, Delete, Rate, … –Playlist: Insert, delete, List, … –Captions: List, insert, update, download, delete, … 36

Youtube Data API Parameters –Part –videoCategoryId –myRating Pricing –Unit based quota calculation –Services with different parameters may cost different quota – quota_cost –50M unit / day, and 30K unit / second, always free 37

Facebook Graph API A web API library for Facebook social network Provide support for various platforms –RESTful –IOS –Android –JavaScript –PHP 38

Facebook Graph API Usages –Provide similar functions to google+ APIs –List friends, List events, Fetch comments, … –Uploading videos, photos, … Pricing –Free –Need to contact Facebook for more than 100M API calls per day 39

MediaWiki API API to get information from Wikipedia programmatically –Entry: Parameters –Action: query, modules –Titles: specify the query –Limit: max results –Section: retrieve from which section Pricing –Free: with unofficial limits 40

Bing API Search API from Microsoft Root: – Parameters –Query –NewsCategory –Sources: web, image, video, news, … 41

Bing API Pricing –5000 / month free –After it: 10K/month for $20 42

Summary Google App Engine Application Deployment on Google App Engine Popular free Web Services 43