1. What can we expect of Evaluating ICTs? Larry Stillman, Centre for Community Networking Research Monash University, Australia ccnr.net

2. Just what do we mean by ICT networks anyway?  Geographically based?  Socially based (Wellman)  Virtual communities of interest  Defining the above helps to clarify/confuse our research questions and policy decisions  See Loader’s paper

3. Parsimony  We never have enough $, £, €!  We never have enough time!  We don’t have the skills etc  What is practical and based on good theory?

4. What is Evaluation?  A means of determining the value, merit, and worth of a program/project  A form of disciplined inquiry akin to research and policy analysis  A rigorous technique for knowledge discovery that joins together the qualitative and quantitative, pragmatically  Typically, a balance between the interests of the funder and the participants/subjects

5. Evaluation / R & D for ICT – Identified Needs A Practical community-level research methods A Building community and agency awareness and training regarding ICTs ie Capacity A Business planning skills  Collaboration among agencies, including neighbouring ICT and telecentre projects.  Roman, R. and R. D. Colle (2002). Themes and Issues in Telecentre Sustainability. (Accessed 22 July 2002.http://idpm.man.ac.uk/idpm/diwpf10.htm)

6. Action Research  A cycle of theory, discovery & research, analysis, reflection and new activity/questions  Ideally suited to new areas of activity  Uses the participants as partners  Can use qualitative and quantitative methods

7. Current Project  Working with Making the Network on their planning and management tools which have a strong participative action research component to add an evaluative component  Working with a large non-profit in Australia to trial it, and elsewhere  PROCESS with community organisations is a fundamental for capturing ‘fuzzy changes’  The accountability aspect comes through the ‘Criteria of Worth’ tool  Outcome: an international tool kit for action-ICT research

8. Network Planning Source: Makingthenetwork.org

9. Evaluation: Data and Analysis Matrix for Criteria of Worth 1 Dimensions of possible evaluative interest to ICT projects Line of Evidence/data source #1 (Quantitative)* Line of evidence/data source #2 (Quantitative)* Conclusion (value, merit and worth ) Planning /Resources +-? Community- User Involvement +++ Community/Soc ial Capital -+? Management++-

10. Evaluation: Data and Analysis Matrix for Criteria of Worth

11. Qualitative Methods  [Case studies of individuals or organisations]  Participant observation  Focus groups and open-ended discourse  Stories  Multimedia  Document review

12. Possibilities for a quantitative standard  Learning Communities: UNESCO’s Four Pillars of Learning  Learning to Know  Learning to Do  Learning to Be  Learning to Live Together  CTC Net survey tools  Social capital surveys: ICT as a particular form of social network  Social Capital consists of networks of social relations characterised by norms of trust and reciprocity”  The ‘bridging and bonding’ electronic network effects on individuals and virtual/ earth-based communities  See Clement and others for a large-scale proposal

13. Issues to be resolvedin an action-research perspective  Resources and skills are major constraints in a community setting; this isn’t the same as funded academic social science  How to measure the virtual, especially the geographically dispersed  Use of cluster evaluation methods (Kellogg Foundation  Use the technology effectively, with caveats (‘the trap’ of over-reliance on complex sytems)

14. Sources A Making the Net Work materials (www.makingthenetwork.org)www.makingthenetwork.org  Clement, A., L. Shade, et al. (2002). Toward an Evaluation Framework for Community Learning Networks.(Accessed: 11 November 2002.http://www.fis.utoronto.ca/research/iprp/) A Hurworth, R. (1996). "Qualitative Methodology. Some questions and answers about analysis of qualitative data in evaluation." Evaluation News and Comment 5(2): 63- 64. A Loader, B (2002) Whither Electronic Communities? Community informatics in principle and practice( http://www.ccnr.net/searchconf/loader.htm)  Miles, M. B. and A. M. Huberman (1988). Drawing Valid Meaning from Qualitative Data: Toward a Shared Craft. Qualitative Approaches to Evaluation in Education. D. M. Fetterman. NY, Praeger. A McConney, A., A. Rudd, et al. (2002). "Getting to the Bottom Line: A Method for Synthesizing Findings Within Mixed Method Program Evaluations." American Journal of Evaluation 23(2): 121-140. A Stone, W. (2001). Measuring Social Capital. Towards a theoretically informed measurement framework for researching social capital in family and community life. Melbourne, Institute of Family Studies. A Wellman, B. (2001). Computer Networks as Social Networks.(Accessed: 14 September 293.http://www.chass.utoronto.ca/~wellman/publications/index.html) A Wellman, B., A. Quan Haase, et al. (2001). Does the Internet Increase, Decrease, or Supplement Social Capital? Social Networks, Participation, and Community Commitment.(Accessed: June 22 2001.http://www.chass.utoronto.ca/~wellman/publications/index.html)