1. G. Papastefanatos 1, P. Vassiliadis 2, A. Simitsis 3, K. Aggistalis 2, F. Pechlivani 2, Yannis Vassiliou 1 (1) National Technical University of Athens {gpapas,yv}@dbnet.ece.ntua.gr (2) University of Ioannina {pvassil,kostazz,fpechliv}@cs.uoi.gr (3) IBM Almaden Research Center asimits@us.ibm.com Language Extensions for the Automation of Database Schema Evolution

2. ICEIS'08, Barcelona, June 20082 Outline Motivation Graph-based modeling A language for policy annotation Evaluation Conclusions

3. ICEIS'08, Barcelona, June 20083 Outline Motivation Graph-based modeling A language for policy annotation Evaluation Conclusions

4. ICEIS'08, Barcelona, June 20084 Data Warehouse Environment WWW Act1 Act2 Act3 Act4 Act5

5. ICEIS'08, Barcelona, June 20085 Motivation WWW Act1 Act2 Act3 Act4 Act5

6. ICEIS'08, Barcelona, June 20086 Database Schema Evolution Current database systems are continuously evolving environments, where design constructs are –Added –Removed –Modified Evolution is not handled by current DBMS with an automatic way –Syntactic as well as semantic adaptation of queries and views is a time-consuming task, treated in most of the cases manually Evolution-driven database design is missing

7. ICEIS'08, Barcelona, June 20087 Evolution Effects SW artifacts around the DBMS are affected: –Syntactically – i.e., become invalid –Semantically – i.e., query must conform to the new source database semantics Adaptation of activities, queries and views –time-consuming task –treated in most of the cases manually by the administrators/developers

8. ICEIS'08, Barcelona, June 20088 We would like to know… What part of the process is affected and how if e.g., an attribute is deleted? Can we predict the impact of changes? To what extent can readjustment be automated? Can we perform what-if analysis for potential changes of source configurations?

9. ICEIS'08, Barcelona, June 20089 Outline Motivation Graph-based modeling A language for policy annotation Evaluation Conclusions

10. ICEIS'08, Barcelona, June 200810 Database Schema Evolution – Our approach Mechanism for performing what-if analysis for potential changes of database configurations Graph based representation of database constructs (i.e., relations, views, constraints, queries) Annotation of graph with rules for adapting queries to database schema evolution Evolving databases Queries Database Schema Graph-based modeling for uniform representation Metrics for Evaluating Evolution Design Evolving applications Rules for Handling Evolution

11. ICEIS'08, Barcelona, June 200811 SELECT EMP.Emp#, Sum(WORKS.Hours) as T_Hours FROM EMP, WORKS WHERE EMP.Emp# = WORKS.Emp# GROUP BY EMP.Emp# Graph based representation

12. ICEIS'08, Barcelona, June 200812 Annotating the graph with adaptation rules Set of evolving database constructs: relations attributes constraints 1 Set of potential evolution changes: addition deletion modification 2 Set of reaction policies: propagate block prompt 3

13. ICEIS'08, Barcelona, June 200813 Query Adaptation - Example Annotated Query Graph Event Add attribute Phone to relation EMP Transformed Query Graph Q NameEID Name EID S S EMP S S map-select … ON attribute addition TO EMP THEN propagate Q: SELECT EID, Name FROM EMP Q: SELECT EID, Name, Phone FROM EMP Q NameEID Name EID S S EMP S S map-select … ON attribute addition TO EMP THEN propagate Phone S S map-select

14. ICEIS'08, Barcelona, June 200814 Contributions Based on –a graph-based model for database constructs (relational tables, views, database constraints) and SQL queries we present –a mechanism for the annotation of the graph’s constructs with elements that facilitate what-if analysis and predetermine the reaction to evolution events occurring in the database schema –SQL extensions that enable the implementation of our approach for evolution management Case study

15. ICEIS'08, Barcelona, June 200815 Outline Motivation Graph-based modeling A language for policy annotation Evaluation Conclusions

16. ICEIS'08, Barcelona, June 200816 Annotation of graph constructs with policies {relation, query, view, attribute constraint, condition} specific nodes ON TO THEN {delete update insert} {block propagate prompt}

17. ICEIS'08, Barcelona, June 200817 Global constraints ON DELETE NODE THEN PROPAGATE ON DELETE ATTRIBUTE THEN PROMPT Example: CREATE DATABASE company ON DELETE ATTRIBUTE THEN PROMPT

18. ICEIS'08, Barcelona, June 200818 Top – level constructs Example for relations: CREATE TABLE works (EMP# NUMBER(3), PROJ# NUMBER(3), HOURS NUMBER(5), ON Add Attribute TO works THEN propagate) This statement for relation works allows the addition of attributes and propagates this addition to all queries and views accessing this relation

19. ICEIS'08, Barcelona, June 200819 Top – level constructs Example for views: CREATE VIEW emps-prjs AS SELECT E.Emp#, E.Name, P.Projname FROM Emp E,Works W,Proj P WHERE E.EMP#=W.EMP# AND W.Proj#=P.Proj# ON Modify Condition TO emps-prjs THEN block This syntax blocks changes in the WHERE clause of the view definition for view emps-prjs

20. ICEIS'08, Barcelona, June 200820 Top – level constructs Example for queries: Q:SELECT EP.Emp#, EP.Name FROM emps-prjs EP WHERE EP.PRJNAME = ‘Olympic Games’ ON Add Attribute TO emps-prjs THEN block The above syntax blocks the inclusion of added attributes in the underlying view emps-prjs in the select clause of the query.

21. ICEIS'08, Barcelona, June 200821 Specific node annotation CREATE TABLE emp (EMP# NUMBER(3), Name Varchar2(150),..., ON Delete Attribute TO Name THEN block) Q: SELECT E.Emp#, E.Name, P.Projname FROM Emp E,Works W,Proj P WHERE E.EMP#=W.EMP# AND W.Proj#=P.Proj# ON Delete Attribute TO Name THEN propagate

22. ICEIS'08, Barcelona, June 200822 Constraints CREATE TABLE emp (EMP# NUMBER(3), Name Varchar2(150), Constraint EMP.PK PRIMARY KEY (EMP#), ON Modify Constraint TO EMP.PK THEN propagate)

23. ICEIS'08, Barcelona, June 200823 Conditions CREATE CONDITION AS CREATE CONDITION Emp_Age_Cond AS AGE>50 CREATE CONDITION Works_Emp_FK AS WORKS.EMP# IN EMP.EMP# CREATE CONDITION Works_Emp_J AS WORKS.EMP#=EMP.EMP# Then, a query SELECT * FROM EMP WHERE AGE_COND would simply use the condition as a macro. Parametric conditions, to allow referring to aliases in SQL queries are straightforward. Reverse eng. of existing code: automatic condition names can be assigned to all the queries.

24. ICEIS'08, Barcelona, June 200824 Outline Motivation Graph-based modeling A language for policy annotation Evaluation Conclusions

25. ICEIS'08, Barcelona, June 200825 Testbed A data mart of an agency for the Greek public sector 6 months of monitoring 52 “queries” (in reports, scripts) over 18 relations When converted to our graph structure: 2500 nodes

26. ICEIS'08, Barcelona, June 200826 Distribution of policies over the annotated nodes Event # of nodes Propagate Block Rename relations 180 Add attributes 6413 Delete Attributes 160892 Rename Attributes 161585 Domain Modification 169010 Condition Modification 021 Total Annotations 4995221

27. ICEIS'08, Barcelona, June 200827 Global policy and individual annotations Scope # of operations Annotations Policy Clauses Query scope 4869 Relation Scope 5180293 Database Scope 362

28. ICEIS'08, Barcelona, June 200828 Outline Motivation Graph-based modeling A language for policy annotation Evaluation Conclusions

29. ICEIS'08, Barcelona, June 200829 Conclusions Coherent framework for propagating potential changes of the database to all the software artifacts around the database based on a model-policy-event scheme Extension to the SQL language specifically tailored for the management of evolution. Limited overhead imposed on both the system and the humans, who design and maintain it. Tested in a real-world scenario of the Greek public sector.

30. ICEIS'08, Barcelona, June 200830 On-going/Future Work Hecataeus: A tool for visualizing and performing what-if analysis for several evolution scenarios. Patterns of evolution sequences

31. ICEIS'08, Barcelona, June 200831 Acknowledgment Information dissemination of this work was supported by the European Union in the framework of the project “Support of Computer Science Studies in the University of Ioannina” of the “Operational Program for Education and Initial Vocational Training” of the 3rd Community Support Framework of the Hellenic Ministry of Education, funded by national sources and by the European Social Fund (ESF).

32. ICEIS'08, Barcelona, June 200832 Gracias ! Sources: http://en.wikipedia.org/wiki/Image:Barcelona_-_planol_ciutat_vella_1860.jpg http://maps.google.com

33. ICEIS'08, Barcelona, June 200833 http://www.cs.uoi.gr/~pvassil/projects/architecture_graph/ Questions?

34. ICEIS'08, Barcelona, June 200834 Algorithm Propagate changeS (DaWaK 2007) Input: an ETL summary S over a graph G o =(V o,E o ) and an event e Output: a graph G n =(V n,E n ) Variables: a set of events E, and an affected node A Begin dps(S, G o, G n, {e}, A) End dps(S, G n, G o, E, A) { I = Ins_by_policy(affected(E)) D = Del_by_policy(affected(E)) G n = G o – D  I E = E–{e}  action(affected(E)) if consumer(A)  nil for each consumer(A) dps(S,G n,G o,E,consumer (A)) }

35. ICEIS'08, Barcelona, June 200835 Conflict resolution (DaWaK 2007) Graph constructs may have contradictory policies for the same event Policies defined on query graph structures are stronger than policies defined on view graph structures which in turn prevail on policies defined on relation graph structures Rule

36. ICEIS'08, Barcelona, June 200836 Configuration of our setting (DaWaK 2007) 7 ETL processes 53 ETL activities 3 lookup tables 7 source tables 9 target tables

37. ICEIS'08, Barcelona, June 200837 Adapted activities per Event (DaWaK 2007)

38. ICEIS'08, Barcelona, June 200838 Related work DB schema Evolution –Schema Versioning DW schema Evolution Materialized View Evolution Evolution wrt Model Mappings