I blogged about EdubloggerCon West the other day as an example of an “unconference” (Conferences must change with the times) and discussed how conferences must move forward to become more dynamic, timely, and better meet the needs of attendees both physical and virtual.
How conference sessions are selected sets the tone for the event. Right now, most conferences select sessions based on a review system created for academic conferences. This system usually consists of a group of readers who rate the proposals. The submissions that score the best from the most reviewers are accepted.
Sounds fair, right? But read on…
Session selection – a bit of history
For academic conferences, session selection is part of the process by which accepted papers are published as proceedings, which enhance the reputation of an academic (publish or perish). The process weeds out papers that don’t meet the conference or journal criteria, makes sure that relevant research results are properly cited, and that the papers form a serious, robust representation of the most current research in the field.
However, conferences that aren’t based on academic publishing tend to retain this legacy. It has become a tradition that no one questions. So the question is — is this unchallenged system really work?
What’s wrong with this process?
I was surprised to find a strong, well-researched argument against the standard session review process on a website from KGCM 2008 (the 3rd International Conference on Knowledge Generation, Communication and Management.)
Now, I don’t know these people at all, but the organization is interested in learning how people learn at conferences. Who knew there was such a thing!
For their own conference and journal, they found that the tried-and-true method of paper and session selection was deeply flawed. Their conclusion – the standard review process is “faulty,” “unreliable,” and worse, tends “…to refuse good papers because of the reviewers’ bias against new ideas or new paradigms.”
“So, it is evident that acceptance policies based on the positive agreement of reviewers will increase the probability of refusing good papers.”
Bias against new ideas
This is bad news for educators going to conferences looking for big ideas that will fundamentally change education. The session selection process works to deny you this opportunity.
Let’s be honest — it’s human nature to resist change. One bad score from one reviewer could easily mean a session is declined, even if other reviewers score it well. It’s why Olympics-style judging throws out the high and low scores.
It’s also natural to want to fit in with the group. When reviewers work together to rate sessions against a rubric, it’s hard not to favor sessions that you think will fit in and make the conference organizers happy.
So in looking at session selection policy, is it any wonder that this method leads to homogenous, cookie cutter selections that represent the acceptable norm? Shouldn’t we be concerned that in times that call for radical change, the standard method for conference session selection is biased against radical proposals?
So what to do instead?
The method this conference uses is different. Instead of scores, they rely on reviewers to recommend either accept or deny. Then:
- The majority rule, when there is no agreement among the reviewers with regards to acceptance or non-acceptance, of a given submission.
- The non-acceptance of the submission when there is agreement among its reviewers for not accepting it.
- Acceptance of the paper when in doubt (a draw or a tie among the opinions of the reviewers, for example).
This seems like a tiny change. Yet the outcome could be completely different.
Would this mean too many accepted sessions? Maybe. But I’m sure there are creative ways to solve these problems. If you read the research-based methodology from KGCM, they have a process designed to pare down the selections. Technology could be part of the answer too. Presentations could go online, for example. Many conferences are also experimenting with shorter sessions, blogger cafes, open forums, and other innovations.
I don’t claim to have “Conference 2.0″ figured out, but I think a necessary first step is to challenge our assumptions about how conference programs are assembled.