v28 #6 I’ll take “Sifting and Winnowing” for $1000, Alex

by Dennis Lloyd  (Director, University of Wisconsin Press)

Last year, I appeared as a contestant on Jeopardy!  I came in third.  Which sounds pretty good if you ignore the fact that the game is played with only three contestants.  Unless you also bear in mind that more than 70,000 took the online test last year — the first step in getting onto the show.  Only about 450 new players appear on air each season, which still put me in the top 0.65% — an unheard-of acceptance rate in the field of scholarly publishing, where I’ve worked for the past two decades.

Also last year, I was appointed director of the University of Wisconsin Press.  This took place around the same time that Gov. Scott Walker made the news for attempting to dismantle the Wisconsin Idea.  Most famously elucidated by UW President Charles Van Hise in 1904 when he declared he would “never be content until the influence of the university reaches every family in the state,” this philosophy is one of two cornerstones of our academic identity.  The other is a well-known quote from an 1894 Board of Regents report about academic freedom, which asserted that Wisconsin “should ever encourage that continual and fearless sifting and winnowing by which alone the truth can be found.”

After the adrenaline (and disappointment) from my game show performance wore off, I found myself returning to this phrase again and again.  While originally written to defend the liberal and pro-union economics professor Richard T. Ely against charges made by then state education superintendent Oliver Elwin Wells, it struck me — and continues to strike me — as an excellent summation of the selection process for book or journal publication, which includes peer review as well as the role of the acquisitions editor (for books) or the volume editor (for journals or essay collections).  Further, I began to see ways in which the review process by which Jeopardy! contestants are chosen might serve as a metaphor for how we determine what (and who) gets published.

The first step in appearing on “America’s favorite quiz show” is to take a fifty-question online test, which is offered once a year.  To get to the next stage one has to have both knowledge (demonstrated by answering a high percentage of questions accurately) and luck (typically more people meet the first criteria than there are audition slots available).  The 2,500–3,500 people invited to an in-person audition must take another fifty-question test and be videotaped playing a sample game and answering questions about themselves.  The producers are looking not only for individuals who can play the game well, they’re looking for people who make good TV, who smile, who look comfortable, who convey fun.  They also must aim for gender and ethnic diversity.  An invitation to travel to LA for a taping can take up to 18 months — or it may not come at all, in which case it’s back to square one for the determined contestant.

For HSS book publishing, the area in which I’ve spent my entire career, the process is similar, if on a smaller scale — with differences in percentages at each cut.  Hundreds of hopeful authors submit proposals or inquiry letters; many are politely declined, either because they don’t fit with the list or do not yet seem fully formed.  Those who make the next stage (the percentages vary, depending in part upon how one defines the initial inquiry, but 10–15% is perhaps a safe assumption) send in completed manuscripts, which are shared with peer reviewers.  Some of these are declined, others are asked to revise and resubmit, others are accepted for publication — but most make it through the review process eventually, thus highlighting the key role played by acquisitions editors in the initial selection process.  In making the final decision, publishers are looking not only for the best scholarship but the best addition to their list, the ones that will sell well or burnish their reputation, or help them acquire the next project, or some combination of the above.

In both situations, the sifting and winnowing is a key part of the process.  The television show is popular in part because of the quality of the contestants; if the screening process weren’t as severe, perhaps the show wouldn’t have lasted 33 seasons (and counting).  In addition, stories abound of aspiring players who take the test for years and are invited to multiple auditions before finally receiving the coveted “call,” not because they increased their raw knowledge but because they improved their on-camera performance.  Likewise, every university press acquisitions editor has a favorite project that went through multiple rounds of readings only to emerge as a stronger project than anyone could have initially imagined.  Built into the peer review process is the assumption that constructive, objective criticism helps the author focus their argument.  In many ways, it’s no wonder that administrators and others who help make tenure and promotion decisions have depended upon publishers’ rigorous selection criteria to help ensure scholarly merit and quality.

It is this entrenched system that some proponents of Open Access seem to want to blow up.  Particularly in the STEM fields, megajournals such as PLOS One have successfully pioneered the concept of post-publication peer review.  Put the work out there, the argument goes, and see what happens.  In metaphoric terms, it would be as if the producers of Jeopardy! chose not to whittle the contestant pool down, but gave us (almost) all 70,000 players to watch and to decide ourselves who truly deserved to appear on the show.

As ludicrous as this sounds, in certain fields this might actually represent the better approach to peer review.  Let’s say I’m conducting research in combating MS and am working with a specific protein.  As I review the existing literature, I don’t want to know only the success stories;  I want to know what failed, how similar proteins behaved, what were the effects on other conditions.  I also need to know these things urgently, in order to apply them to my ongoing research; after all, actual lives may be at stake.  In other words, I need to be able to access and review a broad swath of research, unfettered by a selection process that — from my perspective — hides things from me or a pay-to-read model that prevents me from reading the articles I can’t access.

In a world where a few thousand dollars can be added to a grant to cover the costs of publication, this is a very appealing model.  If I were the director of a publisher in the STEM fields, it would also help me extend the reach of the university throughout the state, the country, and the world, helping fulfill the Wisconsin Idea that almost sounds as though it could have been written as a pro-OA bullet point.

And yet.

Does this model translate to the humanities or nonquantifiable social sciences?  I’m not convinced.  As Karin Wulf eloquently reminded us in a Scholarly Kitchen post last year, “humanities scholarship is not a reporting of research results, but evidence-based argument developed through narrative and analysis.”  If, say, I’m a musicologist examining the development of the chorus in eighteenth-century opera, I need to focus on archival documents and pay attention to the most well-crafted interpretations of similar materials in other research projects.  I don’t need — nor do I want — to review every scrap written to offer one explanation or another.  I benefit when someone else rigorously vets similar work, focusing on quality of writing, depth of contribution to my field, and cleverness of argument.  To return to my governing metaphor: I want to watch the smallest sample possible compete on the game show.  In short, curation deeply matters to me.

In addition, within the STEM fields, a freely available article describing one’s research doesn’t prohibit one (or one’s university) from monetizing and patenting the results of that research.  That is not the case in HSS fields.  To quote Karin Wulf again, from a different Scholarly Kitchen post, “for creative writers and humanists … narrative structure and argument are the research product” (emphasis added).  Adopting a one-size-fits-all approach to “scholarly articles,” as an increasing number of universities are doing in establishing OA policies, seems problematic to me.

The impacts of technological developments on scholarly publishing have been enormous (one need only compare a mail room today with one from thirty years ago for a striking, pragmatic example).  And every library and publisher I speak with now acknowledges that, as revolutionary as Open Access has been, it won’t completely supplant other means of dissemination.  As we move forward, exploring new models, I remain convinced that the sifting and winnowing—what others have called the “gatekeeping” role of academic publishers — remains central.  Yes, perhaps this will cause a given manuscript to be delayed in reaching its audience.  But the urgency of speed of publication is different for articles on Zika research compared to an analysis of Chaucer’s description of the astrolabe.  Besides, not everyone appears on Jeopardy! the first time they try out.  I didn’t.


Pin It

One thought on “v28 #6 I’ll take “Sifting and Winnowing” for $1000, Alex

  1. Pingback: UW Daily – February 15, 2017 | UW Daily

Leave a Reply