1. GRID AND CLOUD COMPUTING
_x005F_x0001_
Google Apps Engine
Courtesy of Professor Ahmed Ezzat, Ph.D

2. UNIT 12: Google Apps Engine
Introduction
Google App Engine Components:
Programming Languages and Runtime / Sandbox
Datastore
Services/APIs
Java Persistence API
Memcache
Fetching URLs and Web Resources
Sending & Receiving Mail and Instant Messages
Task Queues and Scheduled Tasks
Web-based Admin Console
SDK
Deploying & Managing Applications
Things that the App Engine doesn’t Do … Yet
Summary and Conclusions

3. Google App Engine: Introduction - Challenges Building Web Apps

4. Google App Engine: Introduction - Practical Considerations
Software patches & upgrades
Idle capacity
License fees
Lots of maintenance
Traffic & utilization forecasting
Upgrades
Google App Engine does the
difficult work for you

5. Google App Engine: Introduction
App Engine is a Web application hosting service
You build & test your app
Then upload your app to Google
App Engine runs everything
No need to worry about machines, network, storage, scalability, etc.

6. Google App Engine: Introduction – App Engine Components
By “web application,” we mean an application or service accessed over the Web with a web browser by many simultaneous users
You pay only for the resources you use; no monthly fees or upfront charges
Google App Engine provides two possible runtime environment (Java VM, Python) and fully support Google Web Toolkit (GWT)
Google App Engine supports static file servers, away from the Application Servers where the Datastore is used to store data while serving HTTP request for future need
Datastore stores the data in one or more datastore entities. An entity has one or more properties, each has a name (one of named kind) and a value based on several value types
Datastore query returns zero or more entities of a simple kind
App Engine provides a set of indexes for simple queries, based on which properties exists on entities of a kind
When an App creates new entities and when updating existing ones, the datastore updates every corresponding index

7. Google App Engine: Introduction – App Engine Components
To support many clients attempting to R/W the same data simultaneously, App Engine ensures all updates to an entity succeed or the entire updates fail; an update of a single entity occurs in a transaction
When an App calls the Datastore to update an entity, controls does not return to the application until the transaction succeeds or fails
An App can bundle multiple datastore operations in a single transaction, i.e., start transaction….. Commit transaction
Google App Services are the datastore’s relationship with the runtime environment. An App uses an API to access a supported service. These services are self scaling independent from the language runtime environment. These services include Memcache (K/V short term store), URL fetch, Mail Service, task queues & Cron job, etc.
Developer tools: Google provides free tools (SDK for different languages/OS) for developing App Engine application in Java (Java SDK as plug-in for Eclipse) or Python (Python SDK in the form of GUI application for Windows & Mac OS)
Administrative Console: needs to creates an administrative acct and set up the application on the App Engine. You sign to the Administrative Console using your Google account

8. Google App Engine: Introduction – Scalable Infrastructure

9. Google App Engine Components: Programming Languages & Runtime/Sandbox

10. Google App Engine Components: Programming Languages & Runtime/Sandbox
Python:
Based on Python runtime v & v3.6.2
Extremely rapid development
Very low barrier of entry
Simple but yet robust syntax
Rich library of packages/modules
Java:
Ubiquitous in Enterprise computing
Adheres to Java servlet standard & JSP
Includes Java SE Runtime Environment (JRE) 6 platform
Access to most App Engine services using Java APIs including: Java Data Objects (JDO), Java Persistence API (JPA), JavaMail API, Memcache, URL fetch service, etc.
Eclipse Plug-in support
Alternative language support such as Javascript, Ruby, Scala, etc.
Alternative API/Runtime are available

11. Google App Engine Components: Programming Languages & Runtime/Sandbox
Flexible Alternative

12. Google App Engine Components: Programming Languages & Runtime/Sandbox
Alternative API/Runtime are available

13. Google App Engine Components: Programming Languages & Runtime/Sandbox
Extended Language Support through JVM:
Java
Scala
Jruby (Ruby)
Groovy
Quercus (PHP)
Rhino (JavaScript)
Jython (Python)

14. Google App Engine Components: Programming Languages & Runtime/Sandbox
The Sandbox:
Limited access from application to the underlying OS
Sandbox isolates your application in its own secure, reliable environment that is independent of the hardware, OS, and physical location of the web server
Application can access other computers on the Internet through the provided URL fetch and services
Other computers can only connect to the application by making HTTP or HTTPS requests on the standard ports
Applications cannot write to the file system in any of the runtime environments
An application can read file, but only files uploaded with the application code
Application uses the App Engine datastore, memcache or other services for all data that persists between requests

15. Google App Engine Components: Programming Languages & Runtime/Sandbox
The Sandbox:
Python 2.7 environment allows bytecode to be read, written, and modified
Application code only runs in response to a web request, a queued task, or a scheduled task, and must return response data within 30 seconds in any case
A request handler cannot spawn a sub-process or execute code after the response has been sent
Sandboxing allows App Engine to run multiple applications on the same server w/o the side- effect of one application affecting another
Runtime environment limits the amount of clock time, CPU use, and memory a single request can take
App Engine runtime ensures stricter limits to applications that use up more resources to protect shared resources from “runaway” applications

16. Google App Engine Components: Datastore: Datastore Entities
Many scalable web applications use separate systems for handling web requests and for storing data
The App Engine let the application interact with an abstract model that hides the details of managing the pool of datastores
Entities, Keys, and Properties:
Datastore is thought of as object database
An object is an entity
Each object has a key to uniquely identify and fetch the object across the system. Key is <application-id, kind (category), entity-id generated by the datastore or name assigned by the application to the entity-id>

17. Google App Engine Components: Datastore: Datastore Entities
Once a key is created it cannot be changed
The entity data is stored in one or more properties; each property has name and at least one value (i.e., can’t be zero value), and each value is one of a set of data types. A property can have multiple values (multivalued properties) of different data types
For Java, App Engine supports two standard APIs for the Databases: Java Data Objects (JDO) and Java Persistence API (JPA). Using these APIs make the application database independent (could be RDBMS or object database, etc).
These APIs translate to App Engine datastore concepts:
Classes are kinds (RDBMS table)
Objects are entities (RDBMS row)
Fields are properties (RDBMS attribute)

18. Google App Engine Components: Datastore: Datastore Entities
JDO and JPA are built on top of low-level API for the App Engine Datastore
You must use the low-level API to manipulate entities with properties of unknown names or value types
You can use the low level API to build your own data management layer
Each value data type supported by the datastore is represented by a primitive type in the language for the runtime or a class provided by the API

19. Google App Engine Components: Datastore: Datastore Queries (GQL)
Data is useful because you query them
Significant difference between App Engine and traditional database is that when user asks a query, App engine does not go to the data records and determine the answer. Instead, the App Engine pre-compute the answers and when asked query, it searches for the answer in a list of possible answers! As a result, answer is very quick
This is possible because the App Engine maintains an index for every query the application can perform! List of answers are indexes
The above indexing strategy has drawback; the datastore’s built-in query engine is weak compared to RDBMS and is not suitable for sophisticated data processing but is rather adequate for web applications, i.e., calculate the answers to known questions when the data is written so query response would be fast

20. Google App Engine Components: Datastore: Datastore Queries (GQL)
Given a key your can retrieve an entity from the datastore
In most cases applications do not know the key but rather has only a general idea about criteria that it needs entities to meet
To extract such entities, a query include:
The kind of entities to query
Zero or more filters (criteria that property values must meet for an entity to be returned)
Zero or more sort orders that determine the order in which results are returned based on property values
A query based on property values can only return entities of a single kind
It is also possible to perform limited set of queries on entities regardless of kind
Query Results and Keys: when retrieving query results, the datastore returns the full entity with all its properties; it is not possible to query/return a subset of the entity properties

21. Google App Engine Components: Datastore: Datastore Queries (GQL)
GQL: Python and Java runtime environment provide several ways to formulate queries; they all do the same thing
They all return entities (or keys for entities) whose keys and properties meet some filter criteria – optionally returned in sorted order
Example: SELECT * FROM Player
WHERE level > 5 AND level < ORDER BY level ASC, score DESC The above query retrieve all player entities whose “level” property is an integer between 5 and 20, sorted by level in ascending order, then by score in descending order
From the administrative console, you can use GQL to browse the contents of the datastore
The Java Query API: if you are using JPA or JDO APIs, you will use the query facilities of those interfaces to perform datastore queries

22. Google App Engine Components: Datastore: Datastore Queries (GQL)
Example: import com.google.appengine.api.datastore.DatastoreService; import com.google.appengine.api.datastore.DatastoreServiceFactory; import com.google.appengine.api.datastore.Entity; import com.google.appengine.api.datastore.PreparedQuery; import com.google.appengine.api.datastore.Query; //…………. DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); Query q = new Query(“Book”); // Book is the name of the kind q.addFilter (“copyrightYear”, Query.FilterOperator.LESS_THAN_OR_EQUAL, 1950); q.addSort(“title”); PrepareQuery pq = ds.prepare(q); for (Entity result : pq.asIterable() ) { String title = result.getProperty(“title”); …. }

23. Google App Engine Components: Datastore: Datastore Transactions
With web applications, many users access/update data concurrently; needs transactions to maintain data consistency and perform multiple operations atomically
Web applications are at odds with transactions: response to operations should be independent of how much data we have or how it is distributed across servers. Being slow will block others from accessing shared which will impact the overall system throughput!
App Engine provides transactions with strong consistency (other sees the changes once the transaction complete vs. eventual consistency) but with low overhead; this is done by limiting the scope of a transaction and leverage Google’s BigTable
A single transaction can only R/W to entities that belong to a single entity group. Every entity belongs to a single entity group (default is group of your own). When an entity is created it is assigned to an entity group permanently
With concurrent transactions, datastore treats each group independently when applying concurrent transactions; 2 transactions using different groups can execute independently w/o harm – maximize throughput

24. Google App Engine Components: Services/APIs

25. Google App Engine Components: Services: Memcache
Most high-performance web applications use a memory cache
App Engine provide distributed memory cache service (memcache) which stores K/V pairs. You can set/get a value for a key & value can be up to megabyte in size & key can be up to 250 bytes
Updating a value is an atomic operation (not transaction support like the datastore)
Ability to increment/decrement value atomically
Typically memcache is used to cache the datastore entities by their keys
When entity is updated in the datastore, you should attempt to update memcache as well, no transactional update to both – one may successed and the other fails!
Memcache can be used for datastore queries, and web services calls made with URL fetch
You can specify name space for memcache key with any setter or getter calls. Namespace is meant to avoid key collisions
Cache expiration – you can tell memcache to delete/evict a value automatically after a period of time when you set/update the value

26. Google App Engine Components: Namespace API - Memcache

27. Google App Engine Components: Services: Fetching URLs and Web Resources
App Engine can connect to other sites on the Internet to retrieve data and communicate with web services
It does not do that by opening a connection to the remote host but rather through a scalable service called the URL Fetch service
This takes the burden of maintaining connections by the app server and ensures that resource fetching performs well regardless of how many request handlers are fetching resources simultaneously
Fetching URL supports both subset of HTTP operations (GET, POST, PUT, HEAD, DELETE) and HTTPS
URL Fetch service cannot fetch URL that maps to the request handler doing the fetching
The outgoing request can contain URL parameters, request body, and HTTP header
Both Python & Java runtime environments include interfaces to the URL Fetch service. These interfaces implement synchronous URL fetching, as well as asynchronous interface that allows the app to trigger fetch request, do other work, and retrieve results when the request is ready

28. Google App Engine Components: Services: Sending & Receiving Mail and Instant Messages
App Engine can communicate with the outside world in 3 ways:
HTTP requests using URL Fetch service
App can send/receive messages by calling the Mail service
Instant Messaging (XMPP-compatible chat service) including Google Talk and any Jabber server
An app might send to notify users of system events or the actions of other users, confirm user actions, follow up on long term user actions, or send system notification to administrators
App can use XMPP (useful for chat interfaces) to push real-time updates to clients
Sending and chat messages is similar to initiating HTTP request
Receiving and chat messages is similar to receiving HTTP response
Every app has its own set of & XMPP addresses, based on its application ID; currently App Engine does not support & XMPP addresses with custom domains

29. Google App Engine Components: Services: Task Queues and Scheduled Tasks
App Engine is optimized for handling web requests (small amount of work that run in a stateless environment) with very fast response back to the client
Other web applications that don’t fit the above model, instead can write the request to persistent store, respond immediately back to the user, and do the work in the background within seconds/minutes.
The 1st model is appropriate for READ and the 2nd is for Update entities in datastore
3rd model is to get work done independently of user actions (Cron jobs)
App Engine support both three models: task queues (2nd model) and scheduled tasks (3rd model). Scheduled tasks are not retried, but a scheduled task can use a task queue to take advantage of the retry behavior
In the above models, task is a request to request handler. The task specifies the URL path of the application to be called with optional data passed by the App Engine to the request handler
URL of a networked application to perform a task is known as a web hook
App Engine invokes a request handler for a task in the same environment as it does a handler for a user request, i.e., task handler runs with the same sandbox and resource restrictions including 30 sec deadline

30. Google App Engine Components: Web-based Administration

31. Google App Engine Components: Web-based Administration
Application Monitoring:
App Engine Dashboard

32. Google App Engine Components: Web-based Administration
Application Monitoring:
App Engine Health History

33. Google App Engine Components: SDK

34. Google App Engine Components: Java SDK as Plug-in for Eclipse

35. Google App Engine: Deploying & Managing Applications
Uploading your application to App Engine is easy: just click a button or run a command, then enter your acct and password.
All your code, configuration, and static files are sent to the App Engine
No worries about server software, how servers connect to services, scalability issues, system software maintenance, etc.
App Engine support application versioning. You can test new versions before making it public, revert to previous version, and migrating the datastore & service configuration for the app from one version to another
Service configuration is shared across all versions of an app, including datastore indexes, task queues, and scheduled tasks. When you upload the app, the service configuration files on your computer are uploaded as well
Service configuration is separate from application configuration, which includes URL mapping, runtime environment selection and inbound service activation. App configuration is bound to a specific app version
App Engine provides facilities to inspect app performance during runtime as well maintenance tasks (Administrative Console)

36. Google App Engine: Things that the App Engine doesn’t Do … Yet
App Engine does not support secure connections to custom domains, but supports secure connections to appspot.com only
App Engine can use URL Fetch service to make HTTPS request to another site, but App Engine does not verify the certificate used on the remote server
An app can receive incoming mail and XMPP chat messages at several addresses; none of these addresses can use a custom domain name
App Engine does not host long-running background processes. Task queues & scheduled tasks can invoke request handlers outside of a user request and can drive batch processing; this is different from full-scale distributed computing tasks
App Engine does not support streaming or long-term connections
App Engine only support web requests via HTTP/HTTPS, and XMPP messages via the services; does not support other connections’ types
App Engine does not support full-text search queries

37. Google App Engine: Summary and Conclusion:
Current Status
Always free to get started
~5 M pageviews/month
1 GB/day bandwidth (each for incoming/outgoing)
1 GB data storage
2K recipients, 650K IM & 650K URL fetches
6.5 CPU hours daily
Purchasing additional resources
Outgoing bandwidth $0.12/GB
Incoming Bandwidth $0.10/GB
CPU Time $0.10/hr
Storage Data $0.15/GB/month
Recipients ed $0.0001$
Future
More features
More robustness
Additional libraries

38. Google App Engine: Summary and Conclusion:
Google App Engine is meant to serve web applications, and is built over scalable infrastructure
It consists of set of components: multiple languages runtime/sandbox environment, Datastore, Services, Web-based Administrative Console, and SDK
App Engine supports rich set of services including: JPA, JDO, Memcache, URL Fetch, send/receive mail and instant messages, and task queues & scheduled tasks
The App Engine relieves the user from many concerns including scalability, system software maintenance, etc.
App Engine supports application versioning
Still App Engine has few limitations

39. Assignment #11
- Write short note with example code compared to Java of each language mentioned in slide No. 13 that extend Google Apps languages support using Java?
- Watch the video at and Explain the 5 possible Cloud Storage options that Google Apps offers?

40. Questions and Comments?
Thank You
Questions and Comments?