1. RDF FOR DEVELOPERS Paul Groth Thanks to Eyal Oren, Stefan Schlobach for slides

2. Contents The Web of Data A Data Model A Query Language Query Time… Building Apps Exposing Data

3. Why can’t we query all the information on the Web as a database?

4. Data Interoperability on the Web Different formats, different structures, different vocabularies, different concepts, different meaning Data should be structured (not ASCII) Structure should be data-oriented (not HTML) Meaning of data should be clear (not XML) Data should have standard APIs (not Flickr) Reusable mappings between data are needed (not XSLT)

5. What is the Semantic Web? “There is lots of data we all use every day, and it’s not part of the web. I can see my bank statements on the web, and my photographs, and I can see my appointments in a calendar. But can I see my photos in a calendar to see what I was doing when I took them? Can I see bank statement lines in a calendar? No. Why not? Because we don’t have a web of data. Because data is controlled by applications and each application keeps it to itself.” Tim Berners-Lee

6. The Web of Data

7. Enrich your data

8. Standard Data Model RDF: basic data format (triples form hypergraph) – john likes mary. – isbn:10993 dc:title "The Pelican Brief". RDFS : simple schema language (subclass, subproperty) – dc:title rdfs:subPropertyOf rdfs:label. – Jeep isa Car. OWL: rich schema language (constraints, relations) – likes isa owl:symmetricProperty. RDF allows interlinked graphs of arbitrary structure RDFS and OWL allow inferencing of implicit information

9. Standard API: SPARQL Query language for RDF – Based on existing ideas (e.g. SQL) – Supported widely Standard query protocol – How to send a query over HTTP – How to respond over HTTP

10. RDF BASICS

22. RDF Data Model: Summary Resources Triples – Statements: typed links between triples – May involve literals as property values Graph – Set of Statements – Interlinked by reusing same URIs acrossed triples

23. RDF Syntax Several Different Formats – Standard RDF/XML – NTriples – Turtle

25. SPARQL

26. SQL? Formulate a query on the relational model  students(name, age, address) Structured Query Language (SQL) SELECT name data needed FROM student data source WHERE age>20 data constraint nameageaddress Alice21Amsterdam

27. SPARQL Standard RDF query language  based on existing ideas  standardised by W3C  widely supported Standard RDF query protocol  how to send a query over HTTP  how to respond over HTTP

28. SPARQL Query Syntax SPARQL uses a select-from-where inspired syntax (like SQL): select: the entities (variables) you want to return SELECT ?city from: the data source (RDF dataset) FROM where: the (sub)graph you want to get the information from WHERE {?city geo:areacode “010”.} Including additional constraints on objects, using operators WHERE {?city geo:areacode ?c. FILTER (?c > 010)} prologue: namespace information PREFIX geo:

29. SPARQL Query Syntax PREFIX geo: SELECT ?city FROM WHERE {?city geo:areacode ?c. FILTER (?c > 010) }

30. SPARQL Graph Patterns The core of SPARQL WHERE clause specifies graph pattern  pattern should be matched  pattern can match more than once Graph pattern:  an RDF graph  with some nodes/edges as variables hasCapital ?? type EuropeanCountry “020”^^xsd:integer ?

31. Basis: triple patterns Triples with one/more variables Turtle syntax  ?X geo:hasCapital geo:Amsterdam  ?X geo:hasCapital ?Y  ?X geo:areacode "020"  ?X ?P ?Y All of them match this graph: hasCapital NetherlandsAmsterdam “020” areacode

32. Basis: triple pattern A very basic query PREFIX geo: SELECT ?X FROM WHERE { ?X geo:hasCapital ?Y.}

33. Conjunctions: several patterns A pattern with several graphs, all must match PREFIX geo: SELECT ?X FROM WHERE { {?X geo:hasCapital ?Y } {?Y geo:areacode "020" } } PREFIX geo: SELECT ?X FROM WHERE { ?X geo:hasCapital ?Y. ?Y geo:areacode "020". } equivalent to

34. Conjunctions: several patterns A pattern with several graphs, all must match PREFIX geo: SELECT ?X FROM WHERE { {?X geo:hasCapital ?Y } {?Y geo:areacode "020" } } PREFIX geo: SELECT ?X FROM WHERE { ?X geo:hasCapital [ geo:areacode "020" ]. } equivalent to

35. Note: Turtle syntax again ?X geo:name ?Y ; geo:areacode ?Z. ?X geo:name ?Y. ?X geo:areacode ?Z. ?country geo:capital [ geo:name “Amsterdam” ].

36. Alternative Graphs: UNION A pattern with several graphs, at least one should match PREFIX geo: SELECT ?city WHERE { { ?city geo:name "Parijs"@nl } UNION { ?city geo:name "Paris"@fr.} }

37. Optional Graphs RDF is semi-structured  Even when the schema says some object can have a particular property, it may not always be present in the data  Example: persons can have names and email addresses, but Frank is a person without a known email address email name person001 “aisaac@few.vu.nl” “Antoine” name person002 “Frank”

38. Optional Graphs (2) “Give me all people with first names, and if known their email address” An OPTIONAL graph expression is needed PREFIX : SELECT ?person ?name ?email WHERE { ?person :name ?name. OPTIONAL { ?person :email ?email } }

39. Testing values of nodes Tests in FILTER clause have to be validated for matching subgraphs RDF model-related operators  isLiteral(?aNode)  isURI(?aNode)  STR(?aResource) Interest of STR? SELECT ?X ?N WHERE { ?X ?P ?N. FILTER (STR(?P)="areacode") } For resources with names only partly known For literals with unknown language tags

40. Testing values of nodes Tests in FILTER clause Comparison :  ?X <= ?Y, ?Z < 20, ?Z = ?Y, etc. Arithmetic operators  ?X + ?Y, etc. String matching using regular expressions  REGEX (?X,"netherlands","i") matches "The Netherlands" PREFIX geo: SELECT ?X ?N WHERE { ?X geo:name ?N. FILTER REGEX(STR(?N),"dam") }

41. Filtering results Tests in FILTER clause  Boolean combination of these test expressions && (and), || (or), ! (not) (?Y > 10 && ?Y < 30) || !REGEX(?Z,"Rott") PREFIX geo: SELECT ?X FROM WHERE {?X geo:areacode ?Y ; geo:name ?Z. FILTER ((?Y > 10 && ?Y < 30) || !REGEX(STR(?X),"Rott")) }

42. Boolean comparisons and datatypes Reminder: RDF has basic datatypes for literals  XML Schema datatypes:xsd:integer, xsd:float, xsd:string, etc. Datatypes can be used in value comparison  X < “21”^^xsd:integer and be obtained from literals  DATATYPE(?aLiteral)

43. Solution modifiers ORDER BY SELECT ?dog ?age WHERE { ?dog a Dog ; ?dog :age ?age. } ORDER BY DESC(?age) LIMIT SELECT ?dog ?age WHERE { ?dog a Dog ; ?dog :age ?age. } ORDER BY ?dog LIMIT 10

44. SELECT Query Results SPARQL SELECT queries return solutions that consist of variable bindings  For each variable in the query, it gives a value (or a list of values).  The result is a table, where each column represents a variable and each row a combination of variable bindings

45. Query result: example Query: “return all countries with the cities they contain, and their areacodes, if known” Result (as table of bindings): XYZ NetherlandsAmsterdam“020” NetherlandsDenHaag“070” PREFIX geo: SELECT ?X ?Y ?Z WHERE { ?X geo:containsCity ?Y. OPTIONAL {?Y geo:areacode ?Z} }

46. SELECT Query results: format Query: return all capital cities Results as an XML document : PREFIX geo: SELECT ?X ?Y WHERE { ?X geo:name ?Y.} http://example.org/Paris Paris http://example.org/Paris Parijs... Header Results

47. Result formats RDF/XML N3 JSON TEXT 30 http://work.example.org/bob/

48. Query Result forms SELECT queries return variable bindings Do we need something else?  Statements from RDF original graph Data extraction  New statements derived from original data according to a specific need Data conversion, views over data

49. SPARQL CONSTRUCT queries Construct-queries return RDF statements  The query result is either a subgraph of the original graph, or a transformed graph PREFIX geo: CONSTRUCT {?X geo:hasCapital ?Y } WHERE { ?X geo:hasCapital ?Y. ?Y geo:name "Amsterdam" } Subgraph query: hasCapital NetherlandsAmsterdam

50. SPARQL CONSTRUCT queries Construct-queries return RDF statements  The query result is either a subgraph of the original graph, or a transformed graph PREFIX geo: PREFIX my: CONSTRUCT {?Y my:inCountry ?X} WHERE { ?X geo:hasCapital ?Y} inCountry Transformation query: NetherlandsAmsterdam

51. SPARQL queries SELECT: table (variable bindings) select ?x where { … } CONSTRUCT: graph construct { … } where { … } ASK: yes/no ask { … } DESCRIBE: graph describe dbpedia:Amsterdam

52. Conclusion RDF is the data model SPARQL is the query language URIs enable data integration and information enrichment

53. SPARQL Exercise http://github.com/tialaramex/sparql-query Try it on any sparql query endpoint

54. RDF Conclusions revisited Make explicit statements about resources on the web Machine readable data – Machines know that these are statements – Machines know how statements relate resources – Machines can compare values – Machines can dereference URLs

59. Building Apps that use the Web of Data

60. A Simple View of a Web App Template with variables – $x Fill in the variables from a data source – $x = $db->get(“name”) Parameters to query supplied by the user interface Hard part: building the right query, and using the right data structure

61. A Programmatic Approach LAMP stack Use mySQL for data storage and query Allows for procedural modification of data Allows for integration of user input What do we gain by this substitution?

62. APIs PHP: RAP – RDF – http://www.seasr.org/wp- content/plugins/meandre/rdfapi-php/doc/ Python: RDFLib – http://www.rdflib.net/ C: Redland – http://librdf.org/ Java: Jena – http://jena.sourceforge.net/

63. APIs PHP: RAP – RDF – http://www.seasr.org/wp- content/plugins/meandre/rdfapi-php/doc/ Python: RDFLib – http://www.rdflib.net/ C: Redland – http://librdf.org/ Java: Jena – http://jena.sourceforge.net/ SPARQL API Graph API

64. SPARQL Revisited How do we parse these results?

65. SPARQL APIs Build a string that contains the query Execute the query on an endpoint Results are returned as an array or iterator over key-value pairs Usually some sort of basic type mapping, e.g. uri’s are turned into URI objects Can access the variable names from the query

66. Example in PHP $url = “http://example.net:2020/sparql”; $query_str = “select * where {?s a ?o} limit 10; $client = ModelFactory::getSparqlClient(url); $query_obj = new ClientQuery(); $query_obj->query($query_str); $results = $client->query($query_obj) foreach($results as $line){ $value = $line[’?s']; if($value != "") echo $value->toString()." "; }

67. Retrieving Resources What if a resource is described outside of the database? What if you want to work with all the data about a resource? Solution: – Download (curl, urlget) the RDF graph of the resource – Then what…

68. … dbowl:Country a rdfs:Class ; rdfs:label "countries". dbowl:capital a rdf:Property ; rdfs:label "countries capital". … a dbowl:Country ; rdfs:label "countries #USA" ; dbowl:capital "Washington D.C.", ; vocab:hasName "USA". a dbowl:Country ; rdfs:label "countries #Netherlands" ; dbowl:capital "Amsterdam", ; vocab:hasName "Netherlands". rdfs:label a rdf:Property. a dbowl:Country ; rdfs:label "countries #Germany" ; dbowl:capital "Berlin", ; vocab:hasName "Germany". Do we work with this directly?

69. RDF is a graph…so let’s work with graphs Netherl ands “Netherl ands” Country Db:Am sterda m Db:Bel gium Db-owl: City Db-owl: PopulatedPlace/capital rdfs:label a a neighbours a @prefix Db: http://dbpedia.org/resource/http://dbpedia.org/resource/ @prefix Db-owl: http://dbpedia.org/ontology/

70. Graph Model API Model = load(file) Model.getResource(URI) Resource Objects – StmtIter = listProperties() – StmtIter = getProperty(“property”) Statement – Resource = Statement.getSubject() – Resource = Statement.getPredicate() – Resource = Statement.getObject() Using these we can traverse the graph…

71. Note on Graphs Remember objects may be literals They also support building your own RDF model programmatically – Model.createResource(URL) – Resource.addProperty(value) Many APIs also have “sparqlesque” languages for querying your data This may be a way of mashing up your data

72. PHP Example // Get the netherlands resource // don’t let create resource fool you $netherlands = $model->createResource($netherlandsURL); // Retrieve the value of the neighbor property $statement = $netherlands->getProperty($neighbors); $value = $statement->getObject(); // List the Neighbors of the Netherlands echo ' Neighbors of '.$netherlands->getLabel().': '; foreach ($netherlands->listProperties($neighbors) as $country) { echo $country->getLabelObject().' '; };

73. What about ontologies? PopulatedPlace – City Amsterdam – Country Netherlands Belgium

74. Ontology Models Provides methods for querying using a specific vocabulary Generally, RDFS or OWL Changes the perspective of a particular RDF Graph You can use models both approaches simultaneously

75. Ontology API listNamedClassess() listSubClass(Class) getInstances(Class) listProperties() getDomain(Property) getRange(Property) getSubProperties(Property)

76. A note on ontology models The model allow for navigation and creation not reasoning Other inference- models can be used to perform reasoning – These may be expensive – Better to reasoning in a specialized store

77. When would you use an ontology model? When would you use a graph model?

78. Basic APIs Performing Queries – sparql Choosing the right data structure – Graph – Ontology But.. – You still have to build the template – You still have to interact with the user – You still have to create the queries

79. Web Frameworks and RDF ActiveRDF – Ruby on Rails and RDF – http://www.activerdf.org/ http://www.activerdf.org/ Django and RDF – http://semanticdjango.org/ http://semanticdjango.org/

80. Object RDF Mapping Equivalent to Object relational mapping – Multiple inheritance – Focus on properties Python – RDF Alchemy – Surfrdf Java – Elmo – jenabean

81. Other APIs Sindice.com – Semantic web search engine – API for finding thins on the semantic web Sameas.org – Expanding searches – Eliminating duplicates Yahoo Boss – Live search engine – Find web pages with RDFa descriptions OpenCalais – Entity extraction on text to linked data

82. Other cool things Javascript RDF parser – http://www.jibbering.com/rdf-parser/ PHP RDF Classes – http://arc.semsol.org/ Or just see… – http://esw.w3.org/topic/SemanticWebTools

83. Exposing Data on the Semantic Web

84. The Web of Data My Source My Source

85. Databases CountryCapitalRemarkNeighbor NetherlandsAmsterdam“The Hague…Belgium ….… …

86. We want RDF… @prefix :. :Netherlands a :Country; rdfs:label "TheNetherlands" ; :capital :Amsterdam ; :neighbours :Germany, :Belgium.

87. Relational Terminology CountryCapitalRemarkNeighbor NetherlandsAmsterdam“The Hague…Belgium ….… … Country Relation/TableAttribute Tuple

88. Classes Table is a class But instances may have classes… Don’t forget about keys… CountryCapitalRemarkNeighbor NetherlandsAmsterdam“The Hague…Belgium ….… … Country Relation/TableAttribute Tuple

89. Properties Attributes = properties Watch out for naming (country? or label?) What are the domain and range? CountryCapitalRemarkNeighbor NetherlandsAmsterdam“The Hague…Belgium ….… … Country Relation/TableAttribute Tuple

90. Namespaces What is the domain of your application? What is the scope of your application? http://example.org/countries# CountryCapitalRemarkNeighbor NetherlandsAmsterdam“The Hague…Belgium ….… … Country Relation/TableAttribute Tuple

91. URIs and instances Tuple = instance Objects = instances CountryCapitalRemarkNeighbor NetherlandsAmsterdam“The Hague…Belgium ….… … Country Relation/TableAttribute Tuple

92. An RDF Model Netherl ands “Netherl ands” Country Amste rdam Belgiu m City has_capital rdfs:label a a neighbours a

93. What have we gained? Clear definition of types Explicit definitions of relations Graph queries with SPARQL But… we are still not connected to the linked data cloud… My So urc e My So urc e

94. Linked data principles Use URIs as names of things Use HTTP URIs so people can lookup stuff Provide useful descriptions at your HTTP URIs Include links to other URIs See linkeddata.org

95. Use External Concepts and Properties Netherl ands “Netherl ands” Country Db:Am sterda m Db:Bel gium Db-owl: City Db-owl: PopulatedPlace/capital rdfs:label a a neighbours a @prefix Db: http://dbpedia.org/resource/http://dbpedia.org/resource/ @prefix Db-owl: http://dbpedia.org/ontology/

96. Finding External Ontologies From http://semanticweb.org/wiki/Ontology Also see prefix.cc

97. What have we gained? (2) Clear definition of types Explicit definitions of relations Graph queries with SPARQL An enriched data set!

98. Practically: Use a mapping tool Triplify – http://triplify.org Squirrel Rdf - http://jena.sourceforge.net/SquirrelRDF/ Virtuso RDF View D2RQ - http://www4.wiwiss.fu- berlin.de/bizer/d2rq/

99. D2RQ Architecture

100. Getting started with D2RQ Download and install D2RQ server – http://www4.wiwiss.fu-berlin.de/bizer/d2r- server/ http://www4.wiwiss.fu-berlin.de/bizer/d2r- server/ d2r-server-0.7/doc/example Go through the quick start Load a database into Mysql – Make one.. – mysql < test.spl

101. A Simple Database CountryCapital NetherlandsAmsterdam USAWashington D.C. GermanyBerlin Countries

102. Creating a Mapping File > generate-mapping -o mapping.n3 –u db- user -p db-password jdbc:mysql://localhost/test > d2r-server mapping.n3

103. Browsing

104. RDF Output Defines a lot for us

105. Mapping File (Preamble) @prefix map:. @prefix db: <>. @prefix vocab:. @prefix rdf:. @prefix rdfs:. @prefix xsd:. @prefix d2rq:. @prefix jdbc:. map:database a d2rq:Database; d2rq:jdbcDriver "com.mysql.jdbc.Driver"; d2rq:jdbcDSN "jdbc:mysql://localhost/test"; d2rq:username "root"; d2rq:password “****"; jdbc:autoReconnect "true"; jdbc:zeroDateTimeBehavior "convertToNull";.

106. Defining Classes ClassMap defines classes Attached to a particular database Has a set of property bridges # Table countries map:countries a d2rq:ClassMap; d2rq:dataStorage map:database; d2rq:uriPattern "countries/@@countries.country|urlify@@"; d2rq:class vocab:countries; d2rq:classDefinitionLabel "countries";.

107. Defining Properties Map database columns to RDF properties Example: Attaches a property capital to all countries map:countries_country a d2rq:PropertyBridge; d2rq:belongsToClassMap map:countries; d2rq:property vocab:countries_country; d2rq:propertyDefinitionLabel "countries country"; d2rq:column "countries.country";. map:countries_capital a d2rq:PropertyBridge; d2rq:belongsToClassMap map:countries; d2rq:property vocab:countries_capital; d2rq:propertyDefinitionLabel "countries capital"; d2rq:column "countries.capital";

108. Assigning identifiers countries/@@countries.country|urlify@@ 4 Mechanisms to assign identifiers to instances in a database 1. URI Pattern – http://example.org/countries/@@countries.count ry@@ – @@ identifies the column to use – |urlify converts special characters for use in a URL

109. Assigning Identifiers (2) 2. Relative URIs - countries/@@countries.country|urlify@@ - relative to the base URL 3. URI Columns - if the column already has a uri - use d2rq:uriColumns 4. Blank Nodes - d2rq:bNodeIdColumns - blank node per distinct set of values

110. Updating a Mapping File Mapping files do not automatically link to linked data or external ontologies Notice: @prefix vocab: See http://www4.wiwiss.fu- berlin.de/bizer/d2rq/spec/ – Many ways of making your mapping more expressive

111. Ways to Connect Refer to external ontologies – @prefix dbowl: http://dbpedia.org/ontology/ – d2rq:property dbowl:capital – d2rq:class dbowl:Country; Refer to external items – d2rq:uriPattern http://www.dbpedia.org/resource/@@countries. capital@@;

112. Use SameAs Say that your concept is the “same as” another concept using owl:sameas See http://sameas.org/http://sameas.org/ Has a special meaning so that systems no that your concept can be considered identical to another.

113. Results

114. Accessing the data Sparql Queries (on the fly) Dumping the data to RDF –./dump-rdf –m mapping.n3 -f N3 -b http://localhost:2020/ > test.nt

115. … dbowl:Country a rdfs:Class ; rdfs:label "countries". dbowl:capital a rdf:Property ; rdfs:label "countries capital". … a dbowl:Country ; rdfs:label "countries #USA" ; dbowl:capital "Washington D.C.", ; vocab:hasName "USA". a dbowl:Country ; rdfs:label "countries #Netherlands" ; dbowl:capital "Amsterdam", ; vocab:hasName "Netherlands". rdfs:label a rdf:Property. a dbowl:Country ; rdfs:label "countries #Germany" ; dbowl:capital "Berlin", ; vocab:hasName "Germany".

116. Databases -> RDF Allows us to integrate existing databases with the Semantic Web Relational Model maps fairly directly to RDF – Caveats: No unique identifiers, lack of explicit ontological structure Mapping Systems work well Experiment with D2RQ! – It will be used in your final assignment