Developing new model for website assessment Applying evaluation theory approach for assessment of Slovenian government websites Boris Kragelj University of Ljubljana Faculty of social sciences

The “idea” of evaluation study approach Evaluation is systematic use of procedures, methods and research techniques for assessment of ideas, action plans and results of “public” programs (Rossi and Freeman 1993, 5)...that provides answers to the following questions: - How well is program (of government websites) conducted? - Does program (of government websites) functions in the way it supposed to? - How well program (of government websites) achieves desired goals? - Why desired goals (of gov. websites) are not achieved and how it is possible to achieve them? - What kind of changes did program (of gov. websites) reach and what are its effects? We applied this idea of the evaluation to the program of goals of Slovenian government websites.

Construction of evaluation study A three step model: Phase one: PROGRAM - Clear identification of websites’ goals, mission and strategies for achieving them. - Determination of program of Slovenian government websites Phase two: ANALYSIS - Complete analysis of websites and measurement of its performance in all aspects related to its goals and strategy: accessibility, usability, utility… - Holistic analysis of Slovenian government web sites of all relevant aspects of Slovenia government activities on the WWW Phase three: ASSESSMENT - Comparson of the results of analysis with the websites’ program through the seven criteria of evaluation theory: effectiveness, efficiency, adequacy, equity, responsiveness, appropriateness and economy. - Comparison of the condition of Slovenian government websites with their program and strategic goals. Making judgments about how well the program achieved desired goals.

Phase one: PROGRAM On the highest level we determined the program of Slovenian government websites in the following way: -Presentation of the government bodies, offices and providing transparency of their activities (information service, contacts, government documents,... Online) -Providing government services online (tax declaration, car registration, change of address...) -Providing possibilities for citizen’ participation in the process of government activities (through the , forums and web polls...) -Making communication and interaction with government bodies and offices simple and fficient -Generally accessible government for all citizens with no social and technological discrimination -High care for security and privacy when using information and services on government websites

Phase two: ANALYSIS Matrix of different aspects of websites, and methods, research techniques that were used for analysis of Slovenian government websites GENERAL RULE: The more aspects included in analysis and the more different methods applied for the analysis – the better assessment in the final stage of evaluation study is possible!

Phase two: ANALYSIS Combining methods is an important advantage

Phase three: ASSESSMENT General evaluation criteria for the assessment of programs and their application on the program of Slovenian government websites CriteriaDescriptionApplication to SLO government websites EffectivenessWas desired effect achieved?Was desired effect of SLO government web sites achieved? EfficiencyHow much effort was needed for effect achieved? How much effort was needed for the effect of SLO government websites achieved? AdequacyTo which extent was (with achieved effect) problem solved? To which extent was (with achieved effect) problem of SLO government websites solved EquityAre expenses and benefits equally distributed among different groups? Are expenses and benefits for SLO government websites equally distributed among different groups? ResponsivenessAre political effects of program in accordance with needs, preferences and values of parties involved? Are political effects of SLO government websites in accordance with needs, preferences and values of parties involved? AppropriatenessAre desired effects really important?Are desired effects of SLO government websites really important? EconomyWere resources spent in the right and appropriate way? Were resources for SLO government websites spent in the right and appropriate way?

RESULTS of evaluation study Major results: Complete and relevant assessment of websites on the basis of their mission, achievement of certain strategic goals Identification of major problems (with the websites) for realization of the program of websites Directions and orientation for future developments of websites in the way of reaching the strategic goals Additional value: Point of departure for revision of the program and strategy of websites Ground for measuring the progress toward the realization of the website program and its success in reaching the strategic goals Practical guidelines for improvement of websites Establishing best practices Providing ideas for reasonable changes in further developments

RESULTS of evaluation study in the case of Slovenian government websites Slovenian government websites are not meeting any of the evaluation criteria to the greater extent and are not Effective, Efficient, Adequate, Equal, Responsive, Appropriate, neither Economic The web sites (with regard to their program) therefore are not achieving their mission as they are: –not generally accessible to all citizens, –Unfriendly and very difficult to use –lacking of the right content, information and services, –they take very little care for privacy and security matters –and they are without appropriate, common appearance The major problem presents very poorly defined program of Slovenian government websites, what finally results in desynchronized functioning of government websites As an additional value of evaluation approach some important solutions for renovation of Slovenian government websites were provided: –A document with guidelines for essential reformation and improvement of Slovenian government websites was prepared –Draft of the strategy for development of Slovenian government on the WWW was launched Groundwork for measuring the progress toward the mission

Conclusion One final question: Is evaluation study approach for website assessment applicable to all different kinds of websites, or can it be used only for assessment of government websites? This dilemma will remain unsolved until the next case of assessing websites with evaluation study approach from another area will be tested!

Developing new model for website assessment Applying evaluation theory approach for assessment of Slovenian government websites Boris Kragelj University of Ljubljana, Faculty of social sciences