Product Manager, Research Square
by Tom Gilson (Associate Editor, Against the Grain) <[email protected]>
and Katina Strauch (Editor, Against the Grain) <[email protected]>
ATG: Rubriq bills itself as an independent peer review system. Can you tell us how it works?
JP: Absolutely. Currently the majority of peer review of research articles occurs within the scope of a particular journal, and the reviewers are assessing the fit of the manuscript with that journal as much as they are the technical aspects of the work. This limits the applicability of the comments and feedback on the work to this very narrow context. What we are doing is decoupling the review of the manuscript from the assessment of the fit of the manuscript for a particular journal. This in many ways is a more pure form of peer review that allows the experts in the field to focus on the strengths and weaknesses of the manuscript without those evaluations being shaped by the lens of a journal. We provide a thorough report and set of metrics based on the reviewer’s qualitative and quantitative assessment. This decoupling opens up a number of new possibilities for the use of these evaluations for researchers.
Once we receive a manuscript, a doctoral-level member of our team matches the manuscript with three experts from the field to review the paper in a double-blind format using our review scorecard. Once all three reviews have been completed, the author will receive a report with the comments compiled. At that point, the authors can use these assessments to improve the manuscript and/or determine the best place to publish the work. If they choose to, they can submit the Rubriq review with the manuscript along with a letter detailing the subsequent changes they made to the work to the journal editor to help the editor make an informed decision.
ATG: We understand that Rubriq is a division of the parent company Research Square. In what ways is Rubriq independent, and how do you plan to maintain that independence?
JP: Rubriq, American Journal Experts, and Journal Guide are all components of Research Square and are working toward providing a complete solution for researchers. The goal of all of our work at Research Square is to help researchers succeed, and Rubriq plays a part by helping researchers publish quickly in the best journal possible. While Research Square helps authors communicate their work, we are not a publishing company, and we are providing authors peer review independent and outside of the lens of any particular journal. Since we are not trying to shape the authors’ submissions to fit within the context of a journal, our feedback is neutral and can be applied universally.
ATG: Can you tell us a bit about your parent company Research Square? How did it get started in the business of providing these type of journal services? What expertise and experience does Research Square bring to the table? Who are the key players?
JP: As I mentioned, Research Square’s mission is to help researchers succeed, and the way we do that is by helping them communicate their work so they can spend more time making discoveries. Together, Research Square’s brands supported the publication of over 60,000 manuscripts last year. Research Square’s origins are in the AJE brand, which helps international researchers prepare their research for submission in English language journals. Over the last decade, researchers have shared with us additional challenges in communicating their work, which led us to expand our services beyond manuscript preparation into improving peer review with Rubriq and helping researchers find the best journal for their research with JournalGuide. In 2015, we are developing post-publication services to help researchers share the importance of their work and make it more discoverable. We are passionate about making an impact on society by helping one researcher at a time.
Our team consists of 100 postgraduates from a wide array of disciplines, and we have a unique ability to partner with our clients to support their publication needs. We see ourselves as an extension of our customers’ labs. If we perform our roles well, we serve as a communication arm of our customers’ labs, and we give those research teams more time to focus on doing what they are the best in the world at, making discoveries.
The key players are our colleagues at Research Square, who are phenomenal, fun, and vibrant. The raw brain power at one of our happy hours is staggering, and it is a real honor to work alongside them. The officials responsible for the company are Shashi Mudunuri, Founder / CEO, who comes from a technology startup background prior to founding this business, and Keith Collier, COO, who previously ran ScholarOne as a part of the Thomson Reuters Scientific and Scholarly business. These two would be the first to tell you that the key people are the rest of the employees.
ATG: You also say that you “are proactively following the existing standards and guidance to qualify for what you refer to as a “Benefit Corporation.” What does that mean exactly?
JP: Research Square’s primary goal always is to make a positive impact on society. We exist to help researchers succeed. We do not exist to maximize profits, and in this philosophy we are aligned with the principles of a Benefit Corporation.
Benefit Corporation status allows companies to embed sustainable principles into their company DNA. A Benefit Corporation’s directors and officers operate the business with the same authority as in a traditional corporation but are required to consider the impact of their decisions not only on shareholders, but also on society and the environment. To date, 28 states have passed legislation allowing for the creation of Benefit Corporations. Unfortunately, our home state of North Carolina has not yet passed such legislation. Therefore, as a North Carolina organization, we are unable to apply formally for Benefit Corporation status at the state level until legislation is passed. In the meantime, we are seeking to obtain B Corp certification. B Corps are certified by the nonprofit B Lab to meet rigorous standards of social and environmental performance, accountability, and transparency through a third-party audit and evaluation.
ATG: According to the Rubriq Website you are also trying to “create a system that will help match manuscripts to the best journals.” That sounds like a service being offered by American Journal Experts, a separate division of Research Square. Can you clarify that for us? Do Rubriq and AJE interact in some way?
JP: Rubriq and American Journal Experts (AJE) are author services within Research Square, along with Journal Guide, a free tool that helps authors identify journals that are publishing similar work and might be interested in the author’s manuscript. AJE customers can buy Rubriq reports directly from the AJE Website. In addition, AJE offers a service to their publication-ready editing customers in which the editor of the manuscript can use the tools on Journal Guide to provide the authors with a list of journals that might be interested in the subject matter of the paper.
In Rubriq, we also offer a Journal Recommendation service that uses some of these same tools but also utilizes feedback from the reviewers of the manuscript. This service provides the authors with a list of potential journals based not only on the subject matter of the paper but also on the perceived novelty and interest of the work. This helps focus the author’s publishing efforts on journals that are most likely to be receptive to the manuscript, in addition to helping them improve the work with the critical feedback from the reviewers. This combination of feedback will help them get their work published more quickly.
ATG: Do any of Research Square’s services offer any content editing? We are thinking specifically for foreign authors writing in English or vice versa?
JP: American Journal Experts offers language editing for text we receive from researchers, many of whom are international. We focus on language at the sentence level, and our editors do not comment on or edit the research content within the paper. We also ensure that we do not make any changes to a manuscript, such as paragraph reorganization or the removal of text, that would blur the line between authorship and editing.
Specifically, we offer three levels of language editing. Our Standard service focuses on grammar, punctuation, word choice, and phrasing, and our Premium level service builds on this with additional editing for style and consistency. Our Publication Ready service includes everything in our Premium service as well as a journal submission cover letter and help with responses to reviewers and journal selection. We do all of this by matching a researcher’s manuscript to one of the thousands of subject-expert editors in our network, all of whom come from the top research universities in the United States.
In addition, AJE offers translation for authors that would prefer to write their manuscript in Chinese, Portuguese, or Spanish. We also assist authors by formatting their manuscript and figures for submission to a specific journal and even by creating new figures from sketches or raw data.
ATG: Getting back to your peer review service, we understand that your reviewers use a standardized scorecard. What evaluation criteria are included in that scorecard? What type of rating system do you use?
JP: We have several scorecards that are tailored to the needs of manuscripts in a variety of fields. In general, the scorecards are broken down into three large categories: Quality of Research, Quality of Presentation, and Novelty & Interest. Within these categories, we have worked with researchers to identify the critical elements of the papers within those fields to design the separate scorecards. You can see an example of one of our scorecards (Life Sciences) here: https://secure.rubriq.com/sample/scorecard.
When a reviewer is filling out the scorecard, they can note the common deficiencies, and the system will suggest a rating based on a ten-point scale that the reviewer is then free to adjust to their own preference. Once the items are evaluated by each of the reviewers, the R-score is then calculated based on the averages for each of these categories. The upper limit of the R-score is determined by the novelty and interest score. A well-executed study with limited interest may have high quality scores but a low overall score because the novelty and interest value is low. For those who are interested, we have a white paper available on our Website that details more of the science and development of the Rubiq Scorecard (http://www.rubriq.com/img/rubriq-whitepaper.pdf).
However, authors and journal editors have access to more granular quantitative data about a manuscript in addition to the R-score. The reviewer averages for each of the categories and each of the items within these categories are also displayed. In this way, the author or journal editor can easily identify and focus on the aspects of the paper that may need some attention before publication.
ATG: Aside from the use of this scorecard what guarantees the quality control that scholars, publishers, and librarians expect from peer review? What qualifications are required of your reviewers? We also understand that your reviewers are paid. Can you tell us about that?
JP: The reviewers for a manuscript are selected by our team of doctorate-level Peer Review Coordinators, and each reviewer must have a doctorate-level degree (or hold a professorship), have an active research appointment, and be actively publishing papers themselves. The team then pairs the manuscripts with qualified reviewers based on alignment of the topic and methods of the manuscript with the potential reviewer’s own published work.
Once a reviewer has completed a review, the Peer Review Coordinators managing that manuscript will then read the review to ensure that the review contains meaningful, actionable feedback. For an author, that means that the review identified specific areas for improvement, and for a journal editor, that means that the reviewer offered meaningful justification of his or her scores. On the rare occasion that a review is found to be unactionable, a new reviewer for the paper is identified and the unactionable review is replaced.
We do offer compensation for our reviewers to recognize the time and effort they put into these reviews. This can take the form of direct compensation, contribution towards a charity such as AuthorAID, or can be used for editing, formatting, figure formatting, or even Rubriq reviews of the reviewer’s own manuscripts though AJE. However, based on the feedback we have received, the compensation is only part of the reason that many of the reviewers work with us. The ability to read interesting work in their field, help their fellow researchers improve their manuscripts, and support a system that aims to streamline the publishing process are also highly ranked motivations.
ATG: Do you have a list of reviewers along with their credentials that prospective authors would have access to? How many reviewers do you have working for you? Do the reviewers change, and, if so, how often?
JP: We currently have over three thousand reviewers who work with us, and that number is growing every day. If we receive a manuscript and do not have three reviewers that are a great subject match for the work in our system already, then we recruit new reviewers for that submission. In this way, we are always bringing new reviewers into Rubriq and giving authors the best feedback on their work. We do not have reviewers ‘change’, really — we simply keep adding expertise to our network of researchers.
We protect the confidentiality of our relationship with reviewers, so we cannot provide a list of our current reviewers.
ATG: You’ve recently announced the addition of Sound Research Stamps to your scorecards. What are they, and how do they impact the overall score that a perspective publication gets?
JP: Sound Research Stamps are earned by manuscripts based on the reviewer’s responses to one simple question: “Disregarding any consideration of novelty, does this work represent technically sound research?” This is the core question asked by many of the broad-scope, sound science and sound research journals that aim to publish good work with no thought towards the potential impact. Based on the answers to this question, a manuscript can earn a “Sound Research Certified” stamp that indicates that the work is suitable for publication with or without minor revisions, or a “Sound Research Potential” stamp that indicates that the work is solid research, but will require some additional work before it is ready for publication.
These stamps are an addition that we made to the Rubriq Report to complement the R-score. Because the R-score takes the Novelty & Interest scores provided by the reviewers into account, it can make our reports slightly more complicated for an author targeting a sound research journal or an editor working with a sound research journal to use. These stamps add a clear statement about the current state of a manuscript within the context of sound research publishing environment. However, they are independent of the R-score, and the answers to the sound research question do not influence the R-score at all.
ATG: The scorecard is designed to evaluate manuscripts reporting original scientific research. Does that mean scholars in the humanities and social sciences need to look elsewhere for such a service?
JP: Not at all. Although we launched with a focus on the biomedical sciences, we have scorecards for manuscripts both in the humanities and social sciences, in addition to physical science, engineering and material science, math and computer science, and clinical case reports. We have reviewed papers across this entire spectrum both directly with authors and with our journal partners.
ATG: Speaking of scorecards, how is Rubriq doing? Do you have any statistics on the success of Rubriq? For example, can you tell us what percentage of Rubriq manuscripts have been accepted for publication?
JP: So far we have assisted the authors of over 900 manuscripts, with the majority (just over 700) of those reviews performed in 2014. (This makes our peer review operation larger than 90% of the world’s journals, according to our conservative estimates.) Because some of the feedback that our reviewers provide requires further experimentation and the publication process can still take some time (even with our assistance), we are still waiting to see the percent of published manuscripts stabilize.
The feedback that we have received from authors, reviewers, and our publishing partners has been quite positive. Many authors have commented on the thorough nature of the reviews and the depth of knowledge of our reviewers, and the reviewers themselves consistently give us high marks and compliments on our scorecards and our system in general. Our publishing partners also appreciate the quality of the review and the credentials of the reviewers, and both editors and the authors they serve appreciate the speed of our service.
ATG: You also say that Rubriq’s independent peer review “does not require that the traditional process goes away — it just makes it more efficient.” How so? Related to that, another of your goals is to speed up the overall publication process. How does Rubriq accomplish this?
JP: There have recently been many innovations and new models in publishing, such as post-publication peer review. While we are certainly introducing innovations around peer review, what we are doing is still aligned with the traditional concept of pre-publication peer review, and our goal is to improve rather than eliminate the traditional peer review process.
We are innovating and bringing efficiency to the process in three different areas: speed, quality, and journal-independent feedback. An author can expect to get feedback on their manuscript within two weeks from Rubriq, while it can take many journals that length of time to simply decide if they will even review a manuscript. The structure of our Scorecards increases the quality of the reviews by guiding the reviewers through the assessment of every aspect of the manuscript. Our doctoral-level Peer Review Coordinators then ensure that quality by replacing any reviews that do not meet our standards. In addition, the journal-independent, double-blind nature of our evaluations allows the authors to receive honest feedback of the work outside of the context of one specific journal, which will not only help them improve the work but also understand the potential of the manuscript in the broader context of the literature. All combined, we provide fast, high-quality reviews that can help guide the author to publish the work in the right venue quickly rather than the current “trial and error” method. Expediting this process helps to reduce the risk of an author’s work becoming less novel or obsolete because someone else with similar findings gets published first.
ATG: You mention that Rubriq is still aligned with the traditional concept of pre-publication peer review. Were there any specific publisher models that you drew on for inspiration in developing your approach?
JP: No, there really weren’t any specific publisher models that we drew from. Early on we debated the merits of many of the new, emergent peer review models that are currently being experimented with as well as the traditional model. However, in the end we believed that the traditional, blinded peer review model provided the most credible, honest feedback that the author could acquire from their field. Given that some studies have shown that the reviews that a paper receives can be influenced by the gender, nationality, or other aspects of the authors of the paper, we decided to take the additional step of blinding the reviewers to the identity of the authors.
While post-publication peer review has been a hot topic, we think that replacing pre-publication with post-publication peer review fundamentally undermines the relationship between society and science. The general public wants to have some certainty around the veracity of published content. Researchers also seek out trusted journals to designate the importance of their work in order to establish their own reputations. We do not believe a post-publication peer review model will replace the trusted, reputation-building pre-publication peer review model.
ATG: You operate on an author-pay model. What type costs should an interested author expect to incur? What can an author expect for his/her investment?
JP: We currently offer our standard product, which is a double-blind review of a manuscript by three reviewers, for $600. We perform this review in 14 days and return a Rubriq Report that aggregates the reviewer comments and scores. This Report contains comments from each reviewer on each aspect of the manuscript, and it helps an author understand the strengths of the manuscript as well as areas that may need improvement before it can be published. You can see an example Report here: https://secure.rubriq.com/author/submission/report/id/SAMPLE11. In addition, our standard product also includes an iThenticate report that will assist the author in identifying any similarities between the text of their paper and the published literature.
If an author would also like some assistance with selecting a target journal for their work, then for an additional $50 we will find journals publishing similar work and rank those journals by likelihood of acceptance based on the feedback about the novelty and interest potential as noted by the reviewers. The authors then receive a Journal Recommendation report with these journals and information about them such as interest in considering a Rubriq Report, publication times, publication fees, and acceptance rates.
We also have a new product that will allow authors with manuscripts containing statistics to have a biostatistician review that specific aspect of their paper using a scorecard we developed with a team of biostatisticians. This product utilizes a single reviewer and costs $250, or it can be added onto our standard Rubriq review for a total cost of $800.
ATG: You also talk about journals joining the Rubriq network? What is that all about? Which journal publishers have joined? Are there any open access journals that are part of the network?
JP: There are several ways that journals can work with us. The simplest way is for a journal editor to let us know that they are open to considering a Rubriq Report if an author submits the report with their manuscript. We currently have over 500 journals that have indicated their willingness to receive a Rubriq Report, with some journals from all of the major publishers being represented.
We also can work directly with journals to assist them with their peer review process. We have been working with international open access publisher Bloomsbury Qatar Foundation to facilitate the peer review process for a wide variety of their journals, and we are the sole source of peer review for QScience Connect, QScience’s broad-scope, sound research journal. We have been serving in a similar capacity for Veterinaria México OA as they have been rebuilding their journal.
In addition, we are currently in conversations with some of the leading established sound research journals, and we may have some exciting news soon about new ways that Rubriq can help researchers.
ATG: We couldn’t help but notice that all of the journals you mention are science oriented. Are you working with any humanities or social science journals?
JP: Our relationship with QScience includes QScience Connect, which does consider work in the humanities and social science areas, as well as QScience’s International Review of Law. However, we are always open to exploring relationships with any other journals either inside or outside of the sciences that believe that they we may be able to help them better serve their authors.
ATG: Introducing a new service like Rubriq must demand a lot of your time. But to stay sharp you need to recharge your batteries once in a while. Are there any activities that you particularly enjoy that help you kick back and relax?
JP: I’m a pretty mellow person, so no cliff-diving for me. I really enjoy reading and watching movies, and I seem to be on a mission to turn my house into a combination of antique scientific equipment museum and a public aquarium. When the urge to create strikes, I have a woodworking shop set up in my garage and I’m trying desperately to get back to creative writing.
ATG: We really appreciate you taking the time to talk with us about Rubriq, Research Square, and your other projects. We’ve enjoyed learning about them.