Information Searching and Retrieval from Distributed Databases using Mediators, CORBA and XML Presented by:Vincent Cheung Supervised by:Prof Michael Lyu, Prof K. W. Ng Dec 18, 2000

Research objectives  Address the Use of XML in enhancing Data Representation and System Communication  Address the Use of Mediator Architecture for Searching in Distributed Environment  Address the Use of XML and HTTP to simulate CORBA IIOP calls  We have developed a CORBA-based mediator system which can overcome the limitation of firewall problem to achieve a worldwide system

Presentation outline  What is Mediator?  Using XML  Implementation with CORBA  CORBA vs. Firewalls  Overcome the Firewall  Evaluation  Future Work  Conclusion  Demonstration

Searching in distributed sources  To integrate different components in the systems, there are three approaches:  CORBA  Mediators  Agents  They are not orthogonal, e.g.:  Implement mediators by CORBA  Agents may use mediators

What is mediator?  A middle layer for forwarding clients queries to appropriate sources, and integrate the data before returning to users Database Engine Database Engine Database Engine mediator Client UI Clients UI query result

Using XML  We need a COMMON format of information in order to let distributed heterogeneous components for data exchange.  XML provides a good solution because:  XML is semi-structured and highly flexible in data representation. Highly structured traditional relational and object-oriented data schema can be mapped to XML schema without losing information.  A common XML schema for heterogeneous system can be achieved  Hence, we use XML data schema in our system.

A piece of XML data South China Morning Post April 15, 2000 Press warning appropriate, says Beijing Kong Lai-fan Greg Torode Beijing yesterday defended remarks made by senior SAR-based official Wang Fengchao that local media should avoid reporting separatist views.

XML-QL  To query semi-structured data, traditional query languages may not be adequate  We use XMLQL, which is designed for semi- structured XML data  An example: where $B in ‘’news_db.xml’’, in $B construct $B

Why use CORBA?  Common Object Request Broker Architecture  Designed for application development within distributed heterogeneous environment  We use CORBA to build our mediator system.

Architecture of our system Web-base UI 1 st tier Servlet Interface Mediator (forwarding queries and integrating results) 2 nd tier Data Source Mediator Data Source 3 rd tier UI Queries and Results QueriesResults … N th tier

Handling some special cases  Some special cases: Infinite loops, broken connection and too many layer of traversal.  Infinite loops: mediator Solution: To give each query a unique ID, and mediators will keep track all query IDs which have no replied answers yet. Duplicated IDs would indicate an infinite looping has occurred. qid = 123

Special cases  Broken connection Solution: Use a timeout parameter to specify the maximum amount of time that we are willing to wait.  Too may layers of traversal Solution: use a maximum layer parameter to specify the maximum number of layers that we want to go. mediator Timeout = Max_layer = 3 Timeout = Max_layer = 2 Timeout = 5000 Max_layer = 1

IDL design of our system  IDL (Interface Definition Language) defines export interface of CORBA objects  Our IDL design:  The parameter type for special cases handling Struct SysPara { long qid; long timeout; short maxlayer; }

IDL design of our system  Mediator may make queries to Databases or Mediators  Hence, we want Databases and Mediators can be very similar objects  Now, both of them are implementing QueryEngine Interface Interface QueryEngine { String query(in SysPara para, in string xmlquery); } QueryEngine QueryDB QueryMed Implemented by

QueryDB Object  Directly connects to the data source  Caller calls query()  It takes the query statement parameter and make query to related data source  Returns answer in XML string stream format

QueryMed Object  Same invoking method, query()  Besides QueryEngine, it implements another interface, QueryMediator public interface QueryMediator { public QueryEngine[] qelist(); public void qelist(QueryEngine[] arg); public void append_result(String res); }  qelist holds a list of QueryEngine objects, i.e. QueryMed or QueryDB objects, which will be called by that mediator.  It starts a thread for each target QueryEngine object, and the thread will call append_result() to integrate results from various sources

Problems with CORBA firewall  We cannot achieve a worldwide query system because of the firewall.  CORBA uses Internet InterORB Protocol (IIOP) for communication.  Message body of IIOP is encoded in Common Data Representation (CDR), which translates IDL data types into a byte-ordering independent octet string.  Firewalls cannot decode the message body of IIOP in application level.  But we have application level gateways for telnet, FTP, and HTTP

CORBA firewalls  We do have some firewalls which are dedicated for CORBA communications  E.g. IONA Orbix WonderWall and Visibroker Gatekeeper  They have limitations:  Vendor dependent: Orbix firewalls must use Orbix developed client and server objects  Not all features of CORBA can be used: Callbacks may not be used  Not commonly used  Extra purchase, etc …

CORBA and firewalls  IIOP cannot pass firewall, but HTTP can CORBA enclave IIOP HTTP Servlet XML firewall  A real worldwide CORBA system can be achieved

Solution to firewall problem  HTTPGateway is also implementing QueryEngine.  HTTPGateway is a virtual query engine to forward the query to the target systems QueryEngine QueryDB QueryMed Implemented by HTTPGateway

Simulated Call Mediator M HTTP Gateway H firewall Servlet Mediator SM DB Object CORBA Enclave

Sample XML message in parameter passing } \$B in "database.xml" satellite in \$B construct \$B

Its DTD <!DOCTYPE parapassing [ ]>

Evaluation ADVANTAGES:  It can overcome the CORBA IIOP vs. firewalls problem by using HTTP calls,XML, and servlet.  Security issue can still be maintained. External CORBA objects can call only the objects combined with Servlet  Both primitive data types, like String, or complex classes, SysPara, can be well represented by XML in parameter passing

Evaluation  It is not limited for the clients and servers must be CORBA implementation.  Internal CORBA objects would not notice the difference of calling an external objects when comparing to callings of internal objects  With sharing the same XML and IDL standards, we can easily achieve a worldwide query system.

Evaluation WEAKNESS:  With using HTTP calls to simulate IIOP is relative slower than using firewalls dedicated for CORBA, as we need to do many extra works to initialize the servlet, to convert parameters to XML format, invoke the servlets to work, etc.

Future Work  Generalize the IIOP simulation  For all kinds of objects, parameter types, return types, etc  Direct code generation from IDL  Design the mechanism that supports CALLBACKS  When callback can be simulated, we can enhance the features of our mediator system

Conclusion  Using Mediators in querying distributed sources  How to use CORBA and XML to integrate the system  Cooperation between XML and CORBA in simulation of IIOP calls

Demonstration  Aim to show the mediator system can:  make queries to varies sources in parallel  go beyond firewalls User Interface (SHB 1027) Servlet Mediator H111, Kuomao Hall Data Source 1 H111, Kuomao Hall HTTP Gateway H111, Kuomao Hall FI R E W A L L Servlet Mediator (SHB 913, pc90003) Data Source 2 (SHB 913, pc90003)

Q & A Session Welcome to Give Questions and Comments

Thank You