Only 8% members of the Scientific Research Society agreed that “peer review works well as it is.” (Chubin and Hackett, 1990; p.192).

“A recent U.S. Supreme Court decision and an analysis of the peer review system substantiate complaints about this fundamental aspect of scientific research.” (Horrobin, 2001).

Horrobin concludes that peer review “is a non-validated charade whose processes generate results little better than does chance.” (Horrobin, 2001). This has been statistically proven and reported by an increasing number of journal editors. (full text and references)

I got this email from the International Symposium on Peer Reviewing: ISPR being organized in the context of The 3rd International Conference on Knowledge Generation, Communication and Management: KGCM 2009, which will be held on July 10-13, 2009, in Orlando, Florida, USA.

I assume that I got on their mailing list because of my post from almost a year ago, Conference 2.0 – changing how sessions are selected. In that post, I mentioned the KGCM conference and how these experts are investigating how people learn at conferences and are questioning how conference sessions are selected.

In the past year, I’ve read many blogs wondering about conference sessions and the selection process. Many use a form of “peer review”, meaning that people who are presumably peers of the paper author or conference session presenter rate the submission and determine whether or not it gets selected. Academic journals use peer review as a means to claim that their articles are superior to those found in lay magazines, books, or other sources. And many conferences use this same technique to select sessions, whether or not there are academic proceedings that result from the conference.

So this research not only calls into question the methodology of selecting conference sessions, it questions that whole academic tradition of journals representing the highest level of publishing and “truth” about current research in any field. They find that the principle of peer review is well-regarded, but that the methodology and implementation is flawed. Doesn’t that sound familiar…

If peer review is a flawed process, what does that say about conference session selection? You have to ask yourself, would a completely open selection process, where everything is out in the open be more in keeping with the principle of fairness and making sure best practices and research are shared to benefit all?

Would an open selection process for conference presentations be more or less prone to favoritism, self-promotion, populism, and cronyism than the current process? Is that just throwing the baby out with the bathwater?

If you are interested in joining a conversation about improving conferences, check out this social network created by the Association for the Advancement of Computers in Education (AACE) — Spaces of Interaction. They are having an online conversation Feb 18-20 with interactive live sessions.

And by the way, what does this say about school librarians who insist that students use only peer-reviewed journal articles for their research. Do they have a leg to stand on anymore?

Sylvia

• Ten to ask – How to predict the Web 2.0 winners
• Community of interest or community of practice?
• Free – Projects, portfolios and more for creative educators
• NAEP Technology Assessment 2012
• Web 2.0, the meltdown, and education
• Children are collaboration machines
• Fair use explained for educators
• Heavy-handed filtering is a problem, not a solution
• Flunking Spore – video game failed by scientists

Get your own Postrank widget here.

Happy New Year!

Sylvia