5/5/2005Toni Räikkönen Internet based data collection from enterprises using XML questionnaires and XCola engine CoRD Meeting May 11th 2005

5/5/ Background Many web based data collection systems developed in Statistics Finland since 2000 Developed using traditional web development methods like ASP, ASP.NET, COM+ etc. The collection systems are usually quite user friendly consisting of large amount of interaction with the respondent Automated fills, validations, feedback etc.

5/5/ Background (cont.) The development process usually includes a lot of customization and tailoring for a certain survey The results have typically been “unique” systems No automation in development Requires hard and usually more or less the same work time after time Separate maintenance cycle for each system Not very modern way of producing applications

5/5/ Questions How to improve the development of web collection systems? Frameworks? Generic applications? Off-the-shelf applications? What is the role of XML? etc.

5/5/ How it all started? A quick ‘n’ dirty prototype at the end of 2003 Used XML4DR to represent a static customized web form The outcome was quite nice but lacked the dynamics we required for the web surveys Questions Could we extent the use of XML, especially the XML4DR? Define the customization routines in XML? What sort of application would we need to execute all this?

5/5/ Early findings The suitability of pure XML4DR was not adequate enough We required quite a lot dynamic functionality XML4DR did not support this XForms does But is difficult to execute on the server side We later learnt about open source project Chiba (

5/5/ What is XCola? XML based collection application A generic application for web surveys Typically for surveys aimed to enterprises Meaning, the surveys conform to each other are developed using modern techniques and technologies can be developed “easily”

5/5/ …and technically? An engine that reads XML questionnaires as input files and outputs them as XHTML web forms Typically does a lot of processing for the questionnaires between the input and output phases Enabling the dynamics for the surveys

5/5/ Requirements A generic application The questionnaires purely in XML No programming, no scripting Declarative Must support customization Server side implementation Must support at least all the functionality of the older collection systems

5/5/ Requirements (cont.) Requirements for the XML format Support for all the typical question types Open questions, multiline questions, multiple choices, drop down lists etc. Support for open structures Adding / removing rows or columns Should be unaware of the presentation logic No styles, no JavaScript what so ever

5/5/ Requirements (cont.) The questionnaires could be stored in XML or in relational databases Because of our existing infrastructure we decided to store the questionnaires into databases The database structure must be generic All the questionnaires will be stored in the same format The data should be easily mapped back into the XML questionnaires

5/5/ XML questionnaire We chose the XML4DR as the base format However as it doesn’t support the required dynamics we modified it quite a bit Generalized the format to support all the imaginary questionnaire types Added some extra functionality taken from XForms Introduced customization routines Implemented these with processing instructions and XML templates

5/5/ The XCola engine Represents an instance of a web survey Processes the questionnaires and transforms them into web applications Reads and writes the contents of the XML questionnaire from/to the generic database structure Supports client and server side validations Makes use of the authentication and authorization framework

5/5/ Preprocessors The XML questionnaire contains processing instructions for the XCola engine. And sometimes also XML templates to support the PIs The engine reads these instructions and customizes the document accordingly The preprocessors are implemented using chain of responsibility design pattern. The processors are executed in a chain one at a time until all instructions are processed

5/5/ Preprocessors (cont.) So far the engine supports preprocessing for Visibility settings Pre-Filling Row fetching Dynamic Domain (column) generation Dynamic Data (cell) generation Dynamic Calculation formula generation

5/5/ Post processors Post processing takes place when the user submits the web form The types of post processors Update the XML document from the web form contents Update the XPath references Server side validation Update the dynamic structures in the XML document (e.g. adding / removing rows) Saving XML document contents into the database

5/5/ Transformation into a web form The XML questionnaire is transformed into ASP.NET web form using XSLT ASP.NET controls are used for input controls Validation controls are used for client side validation User interface related things are brought in using CSS JavaScript libraries are connected into the web page during the transformation

5/5/ Pros One application, no maintenance hell No programming required for the questionnaire Supports all the basic functionality of the web survey Works well on the enterprise surveys Typically not suitable for interview based surveys Customizable; use just the features needed in the survey Supports linking of external components in case some custom functionality is required

5/5/ Cons Generic solution can not implement any survey dependent functionality Everything must be generic Seems to fit badly into very hierarchical surveys (with more than two levels) Some parts of the XML questionnaire are not so trivial Especially constraints defined in XPath statements can be really hard to understand

5/5/ Current situation Couple of in-house test surveys during fall 2004 No major problems In production since January 2005 Currently running six different surveys The smallest with just 200+ respondents And the largest with over respondents! There will probably be 4-5 new surveys during this year Seems to be the perfect solution for Statistics Finland!

5/5/ The future Probably some minor development Some new features Scalability investigations etc. Polish the tools that are needed to administer XCola surveys

5/5/ XCola in a nutshell Think about XForms: Most of the typical things you can do with XForms you can do with XCola Although our solution isn’t so declarative and requires the power of the engine Executed on the server side Works on every modern browser Extremely simple Easy to implement the questionnaires just in hours