1. CHI 2009 Review Process Changes area-based submissions and sub-committees

2. Guiding Principles Key Problems scale (~1000+ papers + notes) difficult to assign papers to true experts research / evaluation methods differ for different types of work narrowing of field oreferee / decision bias to conventional research and methods

3. Guiding Principles Key Problems scale (~1000+ papers + notes) difficult to assign papers to true experts research / evaluation methods differ for different types of work narrowing of field Goal: expert reviews paper reviewed by true experts in the subject matter Goal: area-appropriate evaluation AC’s know area set criteria that minimize randomness, bias and errors Goal: encourage growth of field into new areas

4. Method Areas-based sub-committees papers / notes integrated authors submit to an area Each area expert area coordinator chooses area ACs Area ACs choose reviewers area-specific criteria runs as mini-PC process All areas same process (similar to existing one) Papers chairs Area 1 coordinator Area 1 AC Area 2 coordinator Area 2 AC Area n

5. Hurdles Defining areas fair coverage for all CHI interests load balance between committees Defining area criteria avoiding biases and restrictions Papers that don’t fit area covers > 1 area caught ‘between the cracks marginalization of niche areas new unanticipated area cross discipline work

6. Choosing Areas CHI sub-disciplines UIST, CSCW, DIS, … By Method how we do our research vs. what’ our research is about By Contribution Type Statistical clusters best coverage of prior submissions / accepted papers

7. By Contribution Type Letters to the community Normative (refine what we have) – empirical Improved gizmos Improves human processes Breakthrough (new idea) – ‘aha’ rationale gizmo design concept Understand what we have – field/in the wild… who we are today (social use) What we could be tomorrow (probes, etc) Process – how we do what we do design process research methods (evaluations, sampling, etc) Systems – how and why we build things

8. By CHI Sub-disciplines Interactions (CHI conventional) CSCW / Group / ECSCW UIST / IUI / Soups (Usable privacy) Ubicomp / TEI / Mobile / HRI / Pervasive DIS / DUX UPA Human Factors / ASSETS / Universal Usability Info Vis / New Media (NIME) Creativity & Cognition Digital libraries/SIGDOC/Hypertext / Info retreival …bridges to other fields …. Issue: encourages existing silos ?

9. By Method Usable techniques – Quantitative Usable techniques – Qualitative Usable techniques –rational / reqt’s analysis Understanding users and contexts Tools and Infrastructure Creativity and Vision Usability Science Theory and analytics Issue: Focuses on ‘how’ not on ‘what’. Did they do it well vs. what did they do?

10. By Statistical Clusters Input techniques8% CSCW11% Pervasive10% Multimodal12% Systems8% Design14% Applications9% Methods18% Media6% Issue: labels as ‘catch-alls’, not well understood…

11. Constraints Authors clearly understandable criteria clearly phrased acceptance criteria Logistics appropriate for committees appropriate for expertise selection equal division of labour (Cdn/UK) equal division of labor (US) Coverage / Values broadens / grows community, areas, across disciplines does not overly narrow into silos does not dis-enfranchise (perception)

12. By Statistical Clusters Input techniques8%

13. By Statistical Clusters CSCW11% Computer Supported Cooperative Work Social Computing and Social Networking Computer-Mediated Communication

14. By Statistical Clusters Pervasive10% Handheld Devices and Mobile Computing Ubiquitous Computing / Smart Environments Tangible UIs Context-Aware Computing Robots

15. By Statistical Clusters Multimodal12% Perceptual & Vision-based UIs Multimedia UIs Tangible UIs Pen-based UIs Tactile & Haptic UIs Speech and Auditory I/O 3D Interaction Multi-modal interfaces Augmented Reality and Tangible

16. By Statistical Clusters Systems8% Security & privacy Agents and Intelligent Systems Development Tools / Toolkits Prototyping End-user programming Software architecture and engineering Virtual Reality Internationalization / Localization Animation

17. By Statistical Clusters Design14% User-Centered design User Interface design User Experience design Design Methods (Design Rational) Interaction design Multidisciplinary design Concept design Product design Service design Visual design…

18. By Statistical Clusters Input techniques8% CSCW11% Pervasive10% Multimodal12% Systems8% Design14% Applications9% Methods18% Media6%

19. By Statistical Clusters Applications9% E-Learning and Education Home / Domestic Virtual Community Health Care Office and Workplace Elderly Creativity Support Children E-commerce Business Strategy

20. By Statistical Clusters Methods18% User Studies Usability Research Usability Testing and Evaluation Empirical Methods, Quantitative Empirical Methods, Qualitative Ethnography User and Cognitive models Analysis Methods (e.g. Task) Performance Metrics

21. By Statistical Clusters Media6% Visualization World Wide Web and Hypermedia Entertainment Video Content / Communications

22. Constraints Authors clearly understandable criteria clearly phrased acceptance criteria Logistics appropriate for committees appropriate for expertise selection equal division of labour (Cdn/UK) equal division of labor (US) Coverage / Values broadens / grows community, areas, across disciplines does not overly narrow into silos does not dis-enfranchise (perception)