Open Ethos Publishing at Code4Lib Journal and In the Library with the Lead Pipe
In Brief: The library world is deeply entrenched in the open ethos, yet there are few examples of library publications that engage in open editorial and peer review processes. In this article we discuss the challenges and opportunities posed by the open editorial processes used at In the Library with the Lead Pipe and Code4Lib Journal. To end, we discuss the need to grow open review and editorial processes in library and information science publications.
Photo by Flickr user Fayster (CC BY 2.0)
By Emily Ford and Carol Bean
The Open Ethos
Open source. Open access. Massive open online courses. Open government. Open data. Creative Commons licensing. These days open movements are everywhere. One might say that open is a fad or trope, but what happens in the world of open goes beyond any clichéd use of the word. With the prevalence of the open concept bridging from computer programmers to academia to democracies around the world, openness is now a pervasive ethos that has, for years, steadily gained momentum. One thing is certain: the open ethos is here to stay.
In the library and publishing field, open access has been a hot topic for quite some time. The library and information science (LIS) literature has discussed the need for an open and changing economy of academic and scholarly information for many years. Authors and publishers have engaged in discussions regarding the economics of journal publishing; many authors argue that current practices of pricing and access are unsustainable. Some authors have even proposed entirely new models of publication that are constantly changing and iterative (Perakakis, et al. 2010).
Open peer review, an emerging open movement, further opens journal publishing practices. It does more than promote free access to published content, it seeks to create open peer review and editorial processes. Open peer review is loosely defined, which caused an interesting discussion during this article’s peer review process. However, it is generally accepted that open peer review refers to non-traditional peer review processes that include disclosure of author and reviewer identities to one another. This is the definition assumed throughout the rest of this article.
There are many different ways that publications have approached open peer review. One way to quickly share research is via pre-print servers. arXiv—a pre-print community and server in physics that was started in 1991—is a great example. At arXiv authors submit articles to the pre-print server before peer review occurs. Once articles are accepted for publication at a journal, authors may replace the pre-print version on arXiv with a final version, as allowed by the terms of their author agreements with publishers. In this environment, articles must be posted by an author who has been “endorsed” by another arXiv community member. In 2011, Axel Boldt argued for arXiv to extend its process to become a true publication with an open peer review process. ((Although the article was published in Journal of Scholarly Publishing, a pre-print does reside on arXiv.))
Some journals simply require reviewers to disclose their identities and conflicts of interest, much like British Medical Journal’s reviewer guidelines. Others, such as the journal Atmospheric Chemistry and Physics (ACP), work for a completely transparent review process. ACP publishes reviewer comments and author replies alongside final versions of accepted and published articles. Some of these reviewer comments are anonymous, while others are attributed. At ACP it is up to the referees whether they would like to disclose their identity. Either way, their comments are published. There is no one uniform way that journals engage in and implement open peer review.
Although more examples of open peer review exist within the sciences, social sciences and humanities have also begun to engage the open ethos and open peer review experiments. Currently, a second special issue of Shakespeare Quarterly using open review is underway. The first special issue used a “hybrid workflow” for the peer review process. It began with article vetting, performed by editors, and then opened articles to public peer review. In this public phase, authors’ and reviewers’ identities were disclosed. Finally, in its third and closed phase, editorial decisions regarding acceptance or rejection were made. (Fitzpatrick & Rowe, 2010). Other current experiments include PeerJ, a new model of open access publication- based on a membership fee- that encourages referees and authors to engage in an optional open peer review process.
Arguments in favor of open peer review cite numerous advantages for an open peer review process. One such advantage is that open peer review enables rich academic dialogue. Wendy Lipworth and her co-authors (2011) argue that their research “findings provide additional reasons … for allowing reviewers and authors to form direct personal relationships, for allowing reviewers to discuss their reviews with their colleagues, or for opening up the review process to broader participation” (p. 278). Similarly, several authors argue that open peer review creates a more rigorous review process than traditional peer review, thus ensuring the quality of published articles. (Mahrag & Duncan, 2007; Pöschl & Koop, 2004; Pöschl, 2008; Bornmann, et al, 2010). When reviewer comments are published, reviewers are held publicly accountable for the quality of their reviews. Additionally, Bornmann (2011) and his co-authors performed citation analysis of articles reviewed and published by Atmospheric Chemistry and Physics, finding review quality is the same as other journals in the field, and that articles published by the journal are more highly cited. Boldt (2011) suggests that open peer review allows for a higher quality of referee comments than those written for closed review processes. Finally, open peer review enables publication process timelines to be substantially shortened (Cope & Kalantzis, 2009; Pöschl, 2008), thereby having quicker impact on research and academic communities.
So too, are libraries and librarianship part of the emerging open ethos. Our journals are no exception. Both In the Library with the Lead Pipe and Code4Lib Journal (C4LJ) use open peer review and editorial processes. ((See Lead Pipe’s open peer review process document, and C4LJ’s page on the journal process and structure.)) In the rest of this article, we discuss the mechanics and experiences of open editorial and peer review processes at these journals, and make a case for the growth of open process publications in library and information science.
The Journals
Both In the Library with the Lead Pipe and Code4Lib Journal are open access publications governed by editorial groups. The journals publish under Creative Commons licenses and are dedicated to participating in the open ethos of publishing. While regular Lead Pipe readers may be familiar with its founding and history from reading editorials such as Have We changed the World yet? and The Survey Says, readers may not be as familiar with Code4Lib and Code4Lib Journal.
Code4Lib Journal is a part of the Code4Lib community, where open source, open access and open process are core identifying values of the group. The Code4Lib community includes anyone who self-identifies as a member. The barrier to participating in the community is low; the primary mechanisms are through the Code4Lib mailing list and Internet Relay Chat (IRC), supplemented by many regional meetings and an annual conference. While C4LJ began as a vehicle for more formal and durable information sharing in the Code4Lib community, even the preliminary discussions expected a global audience. C4LJ’s mission defines its audience as anyone and everyone interested in “the intersection of technology and libraries.” Articles arrive in the journal’s inbox from several avenues: informal suggestions in the group’s IRC and at the group’s annual conference, personal solicitations from members of the editorial committee or Code4Lib community, and in response to announcements sent to other email lists. C4LJ’s reach is, indeed, global: it receives proposals from around the world.
The journal’s focus, however, is far from global. During the start up phase of C4LJ, the inaugural Editorial Board came up with a list of topic ideas for articles which seemed to fit the interests of the Code4Lib community, and listed them in the journals’ Article Guidelines section. The fact that the journal has a limited area of interest makes it look like a more traditional avenue of scholarly communication. The fact that it doesn’t have traditional peer review as one of its identifying features has led to some interesting queries and discussions at C4LJ.
In line with the Code4Lib community’s open ethos, everything about C4LJ is publicly available. Besides information published at the journal’s site, instructions and processes are detailed on the Code4Lib’s wiki. The wiki is a centralized place to keep everything documented and, most importantly for the editorial board, visible.
During its founding in 2008, one year after C4LJ’s, In the Library with the Lead Pipe’s Editorial Board was committed to openness. To that end, the Editorial Board wanted to use open source software and code and publish an open access journal that used creative commons licensing. Board members wanted to establish an open peer review process that mirrored the journal’s dedication to the open ethos; the Editorial Board looked to C4LJas an example of how to implement open peer review.
The Editorial Processes
In the Library with the Lead Pipe
The open peer review process at Lead Pipe begins with the formation of article topics. Although the Editorial Board sometimes receives completed articles as submissions, it is much more common for individuals to propose an idea for an article, rather than a completed article.
Frequently Lead Pipe Editorial Board members will share article proposals over its listserv. If the author is not an Editorial Board member, the Editorial Board member does this on behalf of the guest. Typically, the Editorial Board responds to article topic ideas with questions about the topic that aim to help focus the idea into an article of appropriate scope and relevance. The Editorial Board also shares with potential authors a framework document that poses questions aimed at helping potential authors think through their work. For example, when the Editorial Board discussed this proposal, a very useful email conversation ensued.
Emily: Hi everyone,
I have been thinking about my next post. I think I want to write about the peer-review process at Lead Pipe, and make an argument for why this is a great peer-review model and should/could be adopted for other peer-reviewed journals.
Thoughts?
Brett: I think this is a great topic, though I think our process should just be one of the ones included in the article. For instance, I’d like to
see code4lib represented…
I’d also like to see other journals represented as well. Just not sure which ones.
Micah: I’d like to read this too, especially with the inclusion of the code4lib voice. There is some movement in the scholarly communication world to experiment with open peer review – famously Shakespeare Quarterly ran a special issue using this method.
Kathleen Fitzpatrick wrote a good thing about it too, in her published-online-openly-then-picked-up-by-NYU-Press book, Planned Obsolescence.
Go for it! I’d love to review!
Erin: I love the idea and look forward to reading this. I can help review if you want!
From the outset peer feedback helped shape this article. Fellow Editorial Board members suggested the inclusion of a Code4Lib voice, and hence this article collaboration was born.
All articles published at Lead Pipe undergo review by at least one external peer reviewer and one internal reviewer. Normally, however, more individuals are involved. External peer reviewers are individuals who are not members of the Lead Pipe Editorial Board. Article authors are responsible for identifying and recruiting external reviewers for articles. Sometimes authors will ask the Editorial Board for recommendations on who might be well suited to review an article, but authors often find suitable peer reviewers themselves. This mirrors common practice in the sciences, when authors are often asked to recommend suitable reviewers for the articles they submit. Recruiting external peer reviewers for Lead Pipe has never been a problem, as there are plenty of smart, capable people all over the world. Since Lead Pipe attempts to reach a wide audience and to broach a wide variety of topics of interest, our community of potential and past peer reviewers is equally diverse; many reviewers have not even been librarians.
Peer review of Lead Pipe articles occurs in a variety of ways. While some reviewers may prefer to mark up a document in a word processing program, most frequently authors share a Google Doc with their reviewers (and the entire Editorial Board or its members who agreed to review the article).
Using Google Docs, reviewers make substantive comments discussing the content and organization of articles. By using a collaborative tool, article authors, editors and reviewers can read and respond to all commentary. In this environment many discussions have developed. Authors can respond to comments and directly ask for further clarification of comments. Sometimes reviewers disagree with one another, sometimes they agree. Sometimes reviewers agree with others’ comment, and expand on their ideas, asking additional substantive questions. Sometimes agreement is expressed by a comment that merely says “+1.” The peer review process at Lead Pipe is iterative and cumulative. The end product of its review process is publication of quality articles. Further, reader commentary on published articles complement and continue conversations begun by authors.
Several authors have discussed potential problems caused by publishing numerous article versions. (Prug, 2010; Bornmann et al, 2010). Versions of articles are difficult to track and may influence impact measures. Although article versioning is problematic, it provides a record of the intellectual labor put into the creation of an article. In traditional peer review this information goes unnoticed or could be entirely lost. Open peer review processes provide more acknowledgement of the intellectual labor of reviewing and editing. At Lead Pipe all peer reviewers and editors who have reviewed and edited articles are acknowledged at the end of each article.
Code4Lib Journal
Similar to Lead Pipe, the initial discussion on the Code4Lib list about starting a journal considered peer review options, including a suggestion to have articles vetted on an open wiki. In the end, general consensus was that peer review, whether open or traditional, (1) would slow down the process, (2) wasn’t necessary for the journal’s purposes, and (3) was effectively achieved by the process and makeup of the editorial committee. ((For more information see several email threads at the Code4Lib list: A Code4Lib journal proposal and An attempt to summarize where we are.))
C4LJ’s mission simply states “to foster community and share information among those interested in the intersection of libraries, technology, and the future.” To that end, C4LJ requires that authors agree to the US CC-BY license (Creative Commons Attribution), which fit the journal’s goals of permitting the widest sharing possible while preserving authors’ rights to be acknowledged for their work. Authors are strongly encouraged to release code under a suitable open source license, recognizing that open source licensing is not always possible, or desirable, for authors. ((See, for example, Terry Reese’s article on Purposeful Development, and Dale Askey’s column, We Love Open Source Software. No, You Can’t Have Our Code.)) For the Editorial Board, publishing current, usable information is preferable to requiring open sourced code.
The most important factor for the Code4Lib community, including the journal, is transparency. While editorial discussions about articles are on a closed list, everything else about C4LJ is as open and transparent as it can be. The ethos within the Editorial Committee has been to quickly move non-editorial discussions about the journal to the open, public discussion list.
One of the goals of C4LJ is a quick publication cycle of eighteen weeks between the call for proposals and the publication of an issue. Although authors sometimes send completed articles to the Editorial Board in response to the call for proposals, what is more common is brief summaries of what will be covered and why it is important. If the summary is unclear, the Editorial Board and the author may engage in an iterative process to refine the ideas. Proposals are read and voted on by the entire editorial committee. If a proposal is accepted, one of the editors volunteers to shepherd it through the publishing process, and another editor volunteers to be the “second reader,” which insures that at least two sets of eyes are reading the drafts as they come in.
Editorial discussions of proposals and articles are probably the only parts of the Code4Lib community that are not open and completely transparent. Although a transparent editing process was discussed prior to start up, the editors wanted a forum where they would be free to speak their minds without repercussions. In essence, the editorial committee is a group of peer reviewers that discuss the merits (or not) of proposals and subsequent articles. Although each accepted proposal has an assigned editor and second reader, drafts are forwarded to the whole group for comments. Most importantly, proposals and drafts are provisionally accepted. Actual publication requires another vote by all the editors. Articles that pass the second vote typically have four to six editors that have read the final draft and voted in favor of publication.
Although the editors’ discussion list is closed, it is an open peer review process in the sense that the entire Editorial Committee is identified on C4LJ’s site. All of the editors, past and present, have strong backgrounds in computer technology, libraries and archives. When someone submits an article, they are getting über peer reviewed! There was a recent Editorial Committee discussion about going beyond simply identifying the members on C4LJ’s site, to identify the assigned editor who worked on each article. The biggest concern was that placing an editor’s name on an article might end up identifying the editor in metadata as a co-author.
It is apparent that Lead Pipe and C4LJ have many similarities in their open peer review processes. Both journals are organized by committees, are open in their decision making, and have placed value on openness in their publication and editorial processes. In short, Lead Pipe and C4LJ share the open ethos and embed it into their review processes. So how has it worked in practicality?
The Editorial Processes in Practice
In the Library with the Lead Pipe
Since Lead Pipe’s Editorial Board is a group that makes decisions using consensus, it expects the best intentions from its authors and Editorial Board members. Despite the positives of the review process at Lead Pipe, the Board has had some frustrating experiences. In one instance, the Board worked with an author, spending several hours to read an article draft and provide suggestions for improvement. The author even had a scheduled publication date for the article. However, the author stopped responding to the Editorial Board, missed the publication date, and several months later the article appeared in an entirely different journal.
The Editorial Board has also faced other challenges as it relates to Lead Pipe’s peer review process. One such challenge is editorial timing. As Editorial Board member Brett Bonfield put it, “Who sees what and when is always a difficult balance.” Since many of the Editorial Board members are also regular authors for the journal they, individually, struggle with how much they should edit others’ work. Questions arise, such as: “If we over edit are we doing so to the detriment of the author’s voice? Do we leave most of the content and organization commentary to the official peer reviewers, or do editors step in?” Likewise, when do editorial opinions step in over peer-reviewers? What is the boundary? In the peer-review process, sometimes the different roles between internal and external reviewer can get blurred. While internal reviewers usually provide more feedback in terms of editing for style, mistakes, and copy editing, they also chime in to discuss content and organization of articles. To what extent internal editors have freedom with what gets published is something that we find tricky, and something about which the Editorial Board has no written policy or rule.
Positive experiences with Lead Pipe’s peer review process far outweigh the negative ones. Board members have approached fantastic individuals out of the blue to ask if they would review or be involved with particular articles. Barbara Fister, Kathryn Greenhill, Paul Ford and Gina Trapani are a few examples. Although Board Members may have no personal connections to some reviewers, they have been willing to review and offer feedback of great quality about Lead Pipe’s ideas and writing. The Editorial Board has also been able to leverage our professional networks to identify talented reviewers with specialized knowledge about the particular topics addressed in articles undergoing review.
For authors the collaborative environment is supportive and nurturing. Lead Pipe wants to publish creative ideas, so rejections occur only when a) an author proposed an idea that will fit better in a different publication, or b) authors cannot adequately respond to questions and concerns about article content. The Editorial Board rejects publication proposals where the author wishes to remain anonymous. All in all, Editorial Board members work in a supportive role to authors who have ideas and thoughts that they would like to see published. If there are problems with the execution, Board members do their best to provide the editorial support and feedback to feel good about publishing the ideas as a final product.
As editors, it is beneficial for Board members and other reviewers to see one anothers’ comments. “I can see if we’re giving conflicting advice and address that. And I learn a lot from other editors’ comments” notes Ellie Collier, a Lead Pipe Editorial Board Member. Conflicting editor opinions allow authors to make informed choices about revisions and edits of their work. While an author may understand one editor’s objection to a statement or article organization, she may feel justified in revising less if there isn’t unanimous agreement among reviewers and editors.
One challenge that Lead Pipe still faces, is that of further opening up its editorial and peer review processes. Unlike C4LJ, Lead Pipe does not have a wiki organizing its process or documents. Rather, the Editorial Board relies on a mess of Google Docs explaining its processes. Some readers asked for more transparency regarding Lead Pipe’s peer review processes via the Reader Poll. However, the Editorial Board is currently working on publishing what it has. In a recent Editorial Board Google chat meeting, one editor stated, “FWIW, we weren’t opaque, just overwhelmed.”
This brings us to possibly the largest challenge of publishing Lead Pipe. All of Lead Pipe’s editors and reviewers are volunteers. The journal is a labor of love and many Lead Pipe tasks occur at home during evenings and weekends. From the beginning the project was very much a DIY one, and the Editorial Board still accomplishes what it does via a certain je ne sais quoi or scrappiness.
Code4Lib Journal
At C4LJ the Editorial Committee likes getting complete drafts as proposals, but believes requiring only an abstract level proposal for an article speeds up the publication process. Authors receive feedback fairly quickly about whether the Committee is interested in a topic or not. The Committee tries to hold to a thirteen week publication cycle from provisional acceptance to publication, which means a proposal submitted by the deadline may result in a very tight schedule for the author. This hasn’t generally been a problem since authors that submit proposals on the deadline typically are already working on a draft. Also, since the Committee sends out a formal call for proposals four weeks prior to the deadline, it begins receiving most proposals long before the deadline.
Discussion and voting begins as soon as proposals arrive. Once an article has the required majority of votes in favor (documented on a privately shared Google Spreadsheet), the assigned editor (self-selected) notifies the author of provisional acceptance and begins the process of moving the article toward publication. Article drafts are passed to the whole committee for review, but the assigned editor, with the second reader, takes primary responsibility for working with the author. Any comments by other Editorial Committee members are summarized and passed on to the author by the assigned editor. This cuts down on confusion that can be created by comments and notes coming from several different directions.
This process requires the author to do the bulk of the the work, with the editor helping to shape the article through comments and suggestions. Experience has taught Editorial Committee members to look carefully at submitted proposals and ask for clarification up front, or to simply reject the proposals and include commentary about what would make it more appealing. If what the Committee wants isn’t stated explicitly in the proposal, it’s going to end up being a lot of work for the editor to get the article to fit Committee expectations.
Communication with authors at the proposal level is usually limited and formal, but once a proposal is accepted, this quickly changes. The Committee typically chooses to edit the proposals about which it’s already excited, and is therefore eager to see them get published. During the process of reviewing a proposal, discussions sometimes get quite fervent, with one (or more) editors championing the proposal. For the Editorial Committee, the process of editing an accepted article really is a joint effort with the author to get it published.
The feedback the Editorial Committee has received from C4LJ authors, publicly and privately, has been overwhelmingly positive. Authors appreciate the collaborative process where they get feedback that helps them shape the draft and get the article ready for publication. C4LJ has been characterized, by those who have been through its editorial and review process, as being very author friendly. Authors appreciate the short publication cycle and typically see their articles published in four months or less.
From the perspective of the C4LJ Editorial Committee, the process has worked well. Articles are published quickly and the vetting process, where Committee members read and vote on a proposal rather than a much longer, completed article, works well for all members, who volunteer their time. For each issue, editors typically take one article as “assigned” and volunteer as second reader for an additional article. Individuals with more free time might take more articles to edit; those with less free time may only volunteer as second reader. The time factor does bite sometimes, when the Committee has to delay publication of accepted proposals because everyone is already at their limit editing articles for the current issue.
Photo by Flickr user dullhunk (CC BY-NC-SA 2.0)
Expanding Open Peer Review as part of the Open Ethos
Both C4LJ and Lead Pipe have been publishing with open review processes since their founding. The journals embrace an open ethos that disrupts traditional boundaries of publishing in LIS. Libraries are at the heart of the open ethos because librarians understand and embrace open information, open access and open source. Those values should be extended into the peer review process. Our aims are to publish high quality, high impact articles that have far reach and can be quickly disseminated among our communities. Our journals’ editorial and review processes have shown that open processes are sustainable. Put simply: this model works.
There is some inherent hypocrisy in the library profession when it comes to the open ethos. We want it for our patrons, but do we want it for ourselves? Just recently ACRL made the switch to publish College & Research Libraries and College & Research Libraries News as open access publications. ALA publishing, on the other hand, has not made any visible move toward open access. Other publishers of library journals, such as Taylor & Francis—whose subscription prices for social sciences journals are among the highest—will, arguably, be among the last to open their publications and their peer review processes. ((Taylor & Francis, have, however, made recent steps to open their publications in the LIS fields. Their author agreements for LIS publications now allow authors to retain copyright of their works.)) It is up to the small journals. Journals published by libraries and library organizations need to start the movement.
Library publications should be at the frontier of open peer review. Numerous library publications are already open access, so should consider how to open up the editorial and review processes. In doing so, these publications will continue information’s path to open and free dissemination. Journals need not emulate C4LJ and Lead Pipe, but might consider how open peer review will work for them. Revealing the names of referees alongside published articles is one way this could be accomplished. Journals should also consider publishing editorial and review process documentation.
If open peer review processes are to be adopted, it is up to you, readers and writers. Contact editors at publications you read and for which you write and let them know that you want to better understand the the publications’ editorial and peer review processes. Ask them questions such as:
- Have you ever considered publishing a special issue that experiments with open access or open peer review?
- Can you tell me more about your review process? How many individuals will be reading my work? How are your article acceptance and rejection decisions made?
Consider writing editorial pieces or columns addressing open peer review to the journals you read. These editorial pieces are often where such discussions begin. As with open access, when editors see that their readership and their content creators are interested in understanding processes, they will respond.
We can push our institutions and organizations toward the open ethos. If your place of employ has made the foray into journal publishing, approach those journals. If you’re involved in an organization such as ALA, ASIST, or SLA, ask publications committee members if they have ever thought about opening up review processes? If not, see if you can talk to a publications committee chair to express your enthusiasm for this idea. Maybe the committee will investigate it further or sponsor a workshop or panel investigating open peer review. Lastly, consider whether organizational or institutional bylaw changes may help promote opening editorial and peer review processes. If so, discuss this kind of movement with appropriate stakeholders.
Openness takes time and energy. Traditional publishing and review practices will not change overnight, and when they do change these changes are made after many conversations that have deemed doing so is the right thing. As readers, writers, members of professional organizations and members of institutional communities, it is up to us to broach the topic of open peer review.
For C4LJ and for Lead Pipe open peer review processes have produced high quality, high impact publications. We are helping to shape the open ethos in librarianship, and we invite you to help shape it, too.
Many thanks to Peter Murray, Brooke Johnson, Sara Amato and Hilary Davis for providing thoughtful feedback on this article. Additional thanks to In the Library with the Lead Pipe Editorial Board Members Erin Dorney, Micah Vandegrift, Ellie Collier, and Brett Bonfield for their comments.