Publishing in Measurement Journals Journals as People (Not Just Outlets), Publishing as a Process (Not Just an Event) Presentation for EDMS MSMS Steve Ferrara American Institutes for Research December 4, 2006

Measurement Journals2 Overview for this talk A story about an article in EMIP Some requested themes What some editors look for Review process Responding to review feedback Discerning the real meaning of an editorial decision letter

Measurement Journals3 Educational Measurement: Issues and Practice (EMIP) Aimed at practitioners and users of tests, as well as professional educators, legislators, school personnel, and interested citizens. Its primary purpose is to promote a better understanding of educational measurement and to encourage reasoned debate on current issues of practical importance to educators and the public. EM:IP also provides one means of communication among NCME members and between NCME members and others concerned with educational measurement issues and practices.

Measurement Journals4 Measurement journals and other excellent outlets EMIP Journal of Educational Measurement Applied Measurement in Education Educational Assessment Educational and Psychological Measurement Applied Psychological Measurement Objective Measurement Measurement: Interdisciplinary Research and Perspectives International Journal of Testing Measurement and Evaluation in Counseling and Development Language Testing Educational Leadership (e.g., theme issue on assessment November 2005) Phi Delta Kappan NEA Today Journal of Applied Testing Technology Practical Assessment, Research and Evaluation Content area journals (e.g., JRST, Reading Teacher)

Measurement Journals5 History of this article: late 2003 “Findings…are quite important…study seems to be well designed and executed.” “However, my position is that the current version of this paper is not suitable for publication in EMIP. In general, the paper…For that reason, I will not send out the paper for review.” “You may want to consider revising the current paper and resubmitting it to EMIP. If you do, please include a discussion of how you have addressed each of these concerns. I will be pleased to send out a revised paper for review.”

Measurement Journals6 History: early 2004 “The reviewers and I concur that this paper is appropriate for publication in EMIP, once you respond to a small number of issues. These issues focus on providing clarity and explicitness for the wide range of EMIP readers.” “If you choose to revise and resubmit the current paper, please include a discussion of how you have addressed each of the reviewers’ comments and mine.”

Measurement Journals7 History: summer 2005 Appeared in summer issue of EMIP: 24 (2) An excellent paper

Measurement Journals8 Review and decision process Decisions by people who follow guidelines and make judgments using explicit and implicit standards Decide whether to have it reviewed Reviewer comments to author and recommendation to the editor Weigh the recommendations, make a decision Write the editorial decision letter

Measurement Journals9 Responding to review feedback Respond to every single comment that is made Treat every single comment as advice You’ve already put yourself on the line by submitting Get past defensiveness, outrage at the unfairness of it all, etc. Treat the process as an opportunity Your professional development Improve your paper Improve your future work

Measurement Journals10 Discerning the real meaning of an editorial decision letter I have read your paper, * (SF-), submitted for consideration for publication in EMIP. I also have read and considered comments on the paper from three reviewers. You have written an interesting/thought-provoking/compelling paper, and I appreciated the opportunity to read it. The reviewers and I concur that the current version of the paper is not appropriate for publication in EMIP. I offer the following comments for your consideration. I have enclosed for your consideration comments from the three reviewers. If you choose to revise and resubmit the current paper, please include a discussion of how you have addressed each of their comments and mine. Or: You may want to consider other outlets for your work. I have chosen not to send your paper out for review.

Measurement Journals11 Discerning the real meaning (cont.) The editor does not want to discourage submissions (or may want to) EMIP: unsolicited submissions a year to fill ~16 slots JEM: 100+ unsolicited submissions a year Editors want to treat colleagues fairly and not make enemies Editors want to fill out the journal with good and compelling articles

Measurement Journals12 What editors may look for Want to influence the field Need to feel confident about decisions Want to be collegial and professional Need to uphold the standards of the journal and the field Don’t want to be embarrassed or make an error Care and caution Fairness, quality Issues, practices, methods to target Hot topics in the field Recent papers New ideas Editor’s publication record Editorial Advisory Board

Measurement Journals13 Feel free to contact me: Steve Ferrara Managing Research Director Educational Assessment Program American Institutes for Research