In the previous post I mentioned, I linked to research that said that peer review with a basic scoring system (best aggregate score wins) tends to favor bland, homogenous sessions or papers. Using a consensus model might help with that, although for a submission that is seriously challenging the status quo, it would have to find a “champion” who would stand up for it against colleagues saying that it’s too radical or out of the ballpark. I think you’d have to give a lot of thought about how free your reviewers are to disagree with each other without fear of being labeled a troublemaker.

Like you say, the criteria if often not specified well — “relevant and interesting” may well get you the latest fads and buzzwords, or whatever the reviewers think is “new”. Unfortunately not all new ideas are great ideas, and you might leave a lot of the tried and true stuff on the cutting room floor.

It’s an interesting problem!

Interesting. One organization with which I work, NYLearns, uses a collaborative peer review process to select instructional resources to be published in its online database of instructional resources. This peer review process is mediated by the presence and use of a formal set of standards for the peer review. These standards guide the peer reviewers in their analysis and assessment, which are then shared with the person who submitted the instructional resource. Comments and reflections then go back and forth between the peer reviewers and the original submitter until a consensus is achieved.

I think this type of guidance and mediation would help; I don’t know, though – is it what is currently done? I’ve never been involved with the selection of journal articles; I have run subcommittees for NECC selection a number of years ago, when the process was to gather a local group and go through paper submissions. There was some guidance, but it was basically a question of what the local group found relevant and interesting.