© Donald F. Ferguson, 2015. All rights reserved. Topics in Computer Science: COMS E6998-07-2015-03 Micro-service Application and API Development Lecture.

© Donald F. Ferguson, 2015. All rights reserved. Topics in Computer Science: COMS E6998-07-2015-03 Micro-service Application and API Development Lecture 1: Overview, Concepts, REST Dr. Donald F. Ferguson Donald.Ferguson@software.dell.com (Admin: Melissa.Peoples@software.dell.com)

3 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Contents Overview15 min. –A little bit about me. –Initial lecture schedule. –Coursework and grading. Concepts35 min. –Evolution to micro-services. –Micro-services concepts. –APIs. Break10 min. REST40 min. –Concepts and overview. –Some best practices. Q&A, discussion10 min.

5 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST A Little About Me … Career –Columbia –11 years as a student: B.A., M.S., M.Phil and Ph.D from Columbia University –Ph.D. Thesis – The Application of Microeconomics to the Design of Resource Allocation and Control Algorithms. –Previously taught 7 classes at Columbia –IBM –IBM Research for 10 years; IBM Software Group for 10 years –Work on web applications, J2EE, Eclipse and Web Services –IBM Fellow and Chief Architect for IBM Software Group –Microsoft –Technical Fellow –Technical strategy for future innovation in enterprise software –Initial work on BizTalk.net, and Integration-Platform-as-a-Service; Some concepts in Azure –CA technologies –Chief Architect, Distinguished Engineer and CTO –Technical strategy and product architecture –Dell Software Group –Senior Fellow and CTO –Product architecture and technical strategy –Current focuses are cloud, cloud marketplaces, mobile, IoT, iPaaS, product integration, next generation security Interests –Languages: Speak Spanish well. Learning Arabic slowly. Interested in linguistics and language theory. –Amateur astronomer –Road bicycling –Martial arts: Black belt in Kenpo karate; Krav Maga –Officer Candidate in the New York Guard (http://dmna.ny.gov/nyg/)

6 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Initial Lecture Schedule No syllabus survives first contact with the class. DateTopics 11-Sep-15Course overview, concept overview, REST 18-Sep-15REST continued; some initial patterns 25-Sep-15New database models 2-Oct-15New database models 9-Oct-15Possibly no lecture or WebEx 16-Oct-15Messaging and notification 23-Oct-15Possibly no lecture or WebEx 30-Oct-15Workflow, automation, … 6-Nov-15Event driven architectures, dataflow 13-Nov-15Docker, Kubernetes, … … 20-Nov-15Monitoring, APM, security, … 27-Nov-15Holiday 4-Dec-15Business APIs 11-Dec-15No lecture. General discussion, Q&A, complete projects, etc.

7 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Assignments and Grading Grading –Grading will be based on –5, equally weighted, standalone team projects –5-6 page architecture/design paper. –Code review. –Demo. –You will provide a contribution percentage for each team member. –Class participation –Teams –Please form 4-5 person teams and let me know the members. –Send me an email if you cannot find a team, and I will form some teams. –You may change team membership between projects. –I will take team size into consideration when assigning grade. We will not have a midterm or final.

8 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Some Observations Course decorum –Please call me Don. –Q&A, discussion, etc. is critical to the class. Many lectures in prior courses derived from excellent class questions. –There are no stupid questions. Office hours will be right before class (8:00 AM to 10:00 AM) Resources –http://piazza.com/columbia/fall2015/comse6998007http://piazza.com/columbia/fall2015/comse6998007 –Unfortunately, there is no textbook(s); books tend to be out of date. –The course is covering multiple topics, which would require many books. –The web tends to be the best source of documents and tutorials. –I will reference some texts and sites during the semester. Some initial, very optional, resources are: –Building Microservices, by Sam Newman. –Microservices: Patterns and Applications: Designing fine-grained services by applying patterns, by Lucas Krause –http://microservices.io/patterns/index.html.http://microservices.io/patterns/index.html –http://martinfowler.com/articles/microservices.html.http://martinfowler.com/articles/microservices.html

10 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST “Decomposing applications for deployability and scalability,” Chris Richardson, http://plainoldobjects.com/

11 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Evolution of Application Development (Web) application development went through a phase in which there were two dominant technologies: –J2EE: Java, JDBC, JMS, … … –.NET: C#, ADO.NET, SQL Server, … … Polyglot persistence emerged because –Usecases emerged –That were difficult to map to RDB semantics and optimization –Which drove the development of new, simple, problem focused DBs Polyglot persistence emerged because –Solving some problems seems easier with specific, focused languages –Java and C# became powerful and complicated, and many scenarios needed much simpler and some different capabilities.

12 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Polyglot Persistence (Incomplete List) Google BigTable

15 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Monolithic to Micro Cart Functions Java SQLite Recommendation Functions Node.js Redis Catalog Functions PDP MongoDB XXX MMM NNN Content Functions Ruby Amazon S3

16 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Micro-services Characteristics Componentization via Services Organized around Business Capabilities Products not Projects Smart endpoints and dumb pipes Decentralized Governance Decentralized Data Management Infrastructure Automation Design for failure Evolutionary Design

20 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST SOA Component Architecture  Microservices Five elements still matter 1.Interface and binding.

22 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Scaling https://devcentral.f5.com/articles/the-art-of-scale-microservices-the-scale-cube-and-load-balancing

23 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST SOA vs Microservices http://www.pwc.com/us/en/technology-forecast/2014/cloud- computing/features/microservices.jhtml

24 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Inject application implementation into reusable SW containers Reusable infrastructure containers Reusable SW containers but with core technology and frameworks

26 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST API Economy Sometime you just reuse an API instead of building a micro-service.

27 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST The Course – We Will Cover Patterns, technology and best practices for –Implementing base microservices. –Assembling microservices into “solutions.” –Achieving scalability, availability, agility, etc. Web callable infrastructure services that accelerate implementing microservices, e.g. –Databases –Messaging –… … Web callable application/business services that accelerate implementing a microservise’s business/domain functions.

28 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Microservice Patterns New Microservice PatternsClassic Patterns, New Realization http://microservices.io/patterns/ “Enterprise Integration Patterns,” Fowler et al.

30 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Representational State Transfer (REST) People confuse –Various forms of RPC/messaging over HTTP –With REST REST has six core tenets –Client/server –Stateless –Caching –Uniform Interface –Layered System –Code on Demand

31 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST REST Tenets Client/Server (Obvious) Stateless is a bit confusing –The server/service maintains resource state, e.g. Customer and Agent info. –The conversation is stateless. The client provides all conversation state needed for an API invocation. For example, –customerCursor.next(10) requires the server to remember the client’s position in the iteration through the set. –A stateless call is customerCollection.next(“Bob”, 10). Basically, the client passes the cursor position to the server. Caching –The web has significant caching (in browser, CDNs, …) –The resource provider must –Consider caching policies in application design. –Explicitly set control fields to tell clients and intermediaries what to cache/when.

32 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST REST Tenets Uniform Interface –Identify/locate resources using URIs/URLs –A fixed set of “methods” on resources –myResource.deposit(21.13) is not allowed –The calls are –Get –Post –Put –Delete –Self-defining MIME types (Text, JSON, XML, …) –Default web application for using the API –URL/URI for relationship/association Layered System: Client cannot tell if connected to the server or an intermediary performing value added functions, e.g. –Load balancing –Security –Idempotency Code on Demand (optional): Resource Get can deliver helper code, e.g. –JavaScript –Applets

34 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Anatomy of a URL SSOL for the Classlist https://ssol.columbia.edu/cgi- bin/ssol/DhbtiwfsFOMOeFQaDwqxAh/?p%.5Fr%.5Fid=k0F2vZ4ccAhzbcAg0Ql K4h&p%.5Ft%.5Fid=1&tran%.5B1%.5D%.5Fentry=student&tran%.5B1%.5D%. 5Fterm%.5Fid=20143&tran%.5B1%.5D%.5Fcid=COMSE6998&tran%.5B1%.5 D%.5Fsecid=005&tran%.5B1%.5D%.5Fsch=&tran%.5B1%.5D%.5Fdpt=&tran %.5B1%.5D%.5Fback=&tran%.5B1%.5D%.5Ftran%.5Fname=scrs This is –Not REST –This is some form of Hogwarts spell –This is even bad for a web page

35 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Anatomy of a URL HTTP://www.somedomain.edu/...The “server” containerHTTP://www.somedomain.edu/ …/ssol/…The module/component …/listManagerThe Application Object or …/Class/COMSE6998-01Entity Class (“Extent”) and ID.../WaitingList/…Contained Resource GET, POST, … on URLfor CRUD Some details –…/WaitlingList/dff9/IQPath navigation into resources –…/WaitlingList?op=“Approve”?CUID=“dff9”Method

37 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Database Model are Complex, even examples and samples, e.g. MySql Sakila Sample Database

38 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Some Concepts Foreign key relationships –Represent 1-1, 1-N relationships –Have “behavior,” e.g. –On Delete would prevent deleting a country if there is a city whose country_id is the country’s id. –On Cascade would automatically update all city.country_id when country.id changes Defining indices is important to avoid –Scanning the entire city table to –To find cities in a given country I would not put strings in a table for most words and string –Putting “Spain” for a country name –Prevents localization and national language enablement –Use symbols into localization resource bundles

40 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST What can we Learn? The implementation of the SOA contract/REST interface –Is a set of verbs on URLs –That manipulate a logical data model. Every logical data model has a common set of concepts that materialize through REST –ID  URI/URL –Collections supporting –Primary key  …/Customers/21 –Non-unique, secondary keys  …/Customers/Zipcode/12345 –Ad hoc query (SELECT WHERE (… …))  …/Customers?q=“id<=50&lastname=Ferguson” –Projection –SELECT iq, lastname FROM Customers  …/Customers?”Fields=iq,lastname” –UPDATE iq, shoessize WHERE …  PUT {{iq, “50},{…}}  …/Customers –Foreign keys/join tables  Hyperlinks –Iterators –SELECT * FROM Customers CREATE Cursor …  –GET …/Customers?Offset=40&Pagesize=20 –Thread/callback/promise  Asynchronous REST responses –Metadata/reflection: SQL DESCRIBE TABLE  Web UI for driving the REST API –Stored procedures  PUT…/Commands/… –Events/Notifcations  Feeds

42 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Implementation Observations Define a collection /QueuedResponses –A client can call …/QueuedResponses/21 to get a specific response. –You already know how to do this for …/Customer –The data format in the table is {id, status, JSONString} A simple implementation would be writing a façade –Accept request –Create new table entry with status = “in progress” –Return 202 and URL –Call the actual implementation –Update the database table entry with the JSON result Most application platforms have middleware approaches to support registering callbacks, threads, etc. The implementation would typically –Invoke some long running action, e.g. DB query, workflow process and register a callback –The callback implementation updates the entry in the response table.

43 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Ad Hoc Query Every collection should support ?q=“… …” –…/Customers?q=“lastName=21&IQ<21” –q is a string encoding a set of triplets with elements –Resource field, e.g. “lastName” –Comparison operation, e.g. “=“, “>”, … –Comparison value. Your code needs to –Parse and validate the query string. –Rewrite the string in the query language of the underlying database, e.g. Where clause in SQL –Execute the query –Refine the result set if the underlying database does not support query capabilities that you are surfacing through your API.

44 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Pagination {“data”: [{“user_id”:”42”, “name”:”Bob”, “links”:[{“rel”:”self”, “href”:”http://api.example.com/users/42”}]}, {“user_id”:”22”, “name”:”Frank”, “links”: [{“rel”:”self”, “href”:”http://api.example.com/users/22”}]}, {“user_id”:”125”, “name”: “Sally”, “links”:[{“rel”:”self”, “href”:”http://api.example.com/users/125”}]}], “links”: [{“rel”:“first”, “href”:”http://api.example.com/users?offset=0&limit=3”}, {“rel”:“last”, “href”:”http://api.example.com/users?offset=55&limit=3”}, {“rel”:“previous”, “href”:”http://api.example.com/users?offset=3&limit=3”}, {“rel”:”next”, “href”:”http://api.example.com/users?offset=9&limit=3”}]}

45 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Implementation Considerations Query rewrite –…/Customers?q=“lastname=Ferguson&id<5”&limit=10&offset=5 –Neatly translates into an SQL statement –Select * from customers where … limit=5 offset=5 Other databases have similar concepts. You may have to –Rewrite a push the query down –Build a result cache in another store that supports limit/offset –Paginate through the cache You should also consider adding –“field=lastname,IQ,color” –To enable selecting a subset of fields

47 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Example Considerations Service Endpoint –Endpoint –Endpoint Encryption Requests and Authentication –Request Headers –Request Timestamps –Request Authentication –Response Headers Resources –Resource Requests –Resource Representation –Resource Methods –Synchronous Operations –Asynchronous Operations –Success Response Codes –Failure Response Codes Resource Data Types –Atomic Types –Complex Type - Object or Structure –Resource Relationships –Resource References Pagination –Through HTTP Link Header –Syntax and Example of Pagination Link Header –Consistency Across Page Requests Versioning –Version Header –Version URI Saying “REST is not enough You have to define a set of patterns/ conventions of URLs, headers, …

48 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Endpoint /a/b/c?x=7&y=21 is a pretty straightforward concept, but … –How do I get info about customer “Ferguson?” –…/Customer/Ferguson/Donald –…/Customer?lastName=“Ferguson”&firstName=“Donald” –??? –Do I really want to –Find info about Don using …/Customer/Ferguson/Donald –Find info about agent using …/Agents?id=“21” –How does it work if I can find customer by name or phone number? –How do I set a relationship between customer and agent? –PUT …/Relationship/AgentFor?agent=“21”&”Customer=“Ferguson” –Or two PUTS, one on Customer and one on Agent? In the same way you have to define a framework for your application, you have to define a shape/pattern in your REST API model.

49 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Headers – Some Choices I Made HTTP Request HeaderValueMandatory auth-timestamp:The current POSIX time.Yes auth-key: The user or client’s unique API KEY. Yes auth-signature: The HMAC-SHA256 digest for the request. Yes api-version:(Optional) API version stringNo Accept: (Optional) application/xml or application/json No Nonce:One time UUID to enable idempotency/duplicate detection

50 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Response Codes OperationHTTP RequestHTTP Response Codes Supported READGET 200 - OK with message body 204 - OK no message body 206 - OK with partial message body CREATEPOST 201 - Resource created (Operation Complete) 202 - Resource accepted (Operation Pending) UPDATEPUT 202 - Accepted (Operation Pending) 204 - Success (Operation Complete) DELETE 202 - Accepted (Operation Pending) 204 - Success (Operation Complete) Examples of Link Headers in HTTP response: Link: ;rel=monitor;title="update profile" Link: ;rel=summary;title=”access report” 202 means Your request went asynch. The HTTP header Link is where to poll for rsp.

51 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Failure Response Code ErrorResponse Code Invalid Parameter400 - Invalid parameter Authentication401 - Authentication failure Permission Denied403 - Permission denied Not Found404 - Resource not found Invalid Request Method405 - Invalid request method Internal Server Error500 - Internal Server Error Service Unavailable503 - Service Unavailable

52 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Design Pattern Choices Resource Requests Collectionhttps://ENDPOINT/NAMESPACE/RESOURCE[?QUERY_PARAMETERS] Resourcehttps://ENDPOINT/NAMESPACE/RESOURCE/RESOURCE_ID[?QUERY_PARAMETERS] Collection OperationHTTP Request Get all items in the collection GET /collection Should also return the URI of the collection itself. Get an particular item in the collectionGET /collection/itemId Get items match certain criteriaGET /collection?property1=’value’ Add a new item to the collection POST /collection contents of new item … Get items starting at 100 with page size=25 GET /collection?start=100&pageSize=25 Support for Map Array Collection

53 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Relationships {“membership”: { “URI” : “http://dell.com/memberships/m12356”,http://dell.com/memberships/m12356 “created” : “2013-08-01T12:00:00.0Z”, “owner” : “user123456”, “expire” : “never”, “group” : { “ref” : “http://dell.com/groups/g123456” },http://dell.com/groups/g123456 “server” : { “ref” : “http://dell.com/servers/s123456”}http://dell.com/servers/s123456 } "link": { "href": "http://dell.com/api/resource1",http://dell.com/api/resource1 "rel": "self", “title” : “server-s123456” } Relationship as a resource Relationship as a field in resource

54 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Navigating Through Result Set GET on collections, maps, … needs pagination and cursors –Limit: What is the maximum number of elements you want? –QueryID: A tag for the query that produced the original result set –Offset references a specific element in a “page.” There is a standard for linking resources in logical sets, e.g. –Link: ; rel="previous"; title="previous chapter“> –Indicates that "chapter2" is previous to this resource in a logical navigation path. Your API/framework can use this for result sets –Example 1 –GET /api/customers?status=“Gold” returns some number of “Gold” customers and –Link Header for “next page” is Link: ; rel="next last“ –Which is the URL for the “cursor.next set,” which has 50 elements and is also “last” –Example 2 –Get returns the “next” from example 1 –With Link Link: ; rel="prev first" –Allowing you to go backwards to the previous “page.”

58 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST X-Axis and Y-Axis Scale-Out through Shared database Cloned, stateless code

60 © Donald F. Ferguson, 2015. All rights reserved.Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST 1 st Projects –Simple content management –Mail verification for registration; CAPTCH

© Donald F. Ferguson, 2015. All rights reserved. Topics in Computer Science: COMS E6998-07-2015-03 Micro-service Application and API Development Lecture.

Similar presentations

Presentation on theme: "© Donald F. Ferguson, 2015. All rights reserved. Topics in Computer Science: COMS E6998-07-2015-03 Micro-service Application and API Development Lecture."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

© Donald F. Ferguson, 2015. All rights reserved. Topics in Computer Science: COMS E6998-07-2015-03 Micro-service Application and API Development Lecture.

Similar presentations

Presentation on theme: "© Donald F. Ferguson, 2015. All rights reserved. Topics in Computer Science: COMS E6998-07-2015-03 Micro-service Application and API Development Lecture."— Presentation transcript:

Similar presentations

About project

Feedback