Column Editor: Darrell W. Gunter (President & CEO, Gunter Media Group)
This new era of COVID-19 has reminded us of the importance of good quality scholarly research. Our guest of this issue of ATG is Dr. Leslie McIntosh, the CEO of Ripeta (www.ripeta.com).
DG: Leslie, thank you for making yourself available for this interview to discuss the Ripeta business and how it is going to improve and enhance the quality of scholarly research.
LM: Thank you so much for having me, Darrell. It’s an honor to be here.
DG: So if you could just give our audience a little bit of background about your education, experience, et cetera. How did you come to start Ripeta?
LM: So my educational background is a little interesting. The fun part right now is that my Ph.D. is actually in Epidemiology. As my stepmom put it, “I didn’t even know what that was before you got your Ph.D. and now I hear about it everywhere.” It was a data-driven Ph.D. supported by my master’s in Biostatistics and Epidemiology.
One perhaps surprising piece about my educational background is that I never graduated from High School. I always like to throw that out there in case there is anyone out there who needs a little motivation to keep going — keep going.
Upon completing my Ph.D., I went into biomedical data and working with a lot of electronic health record data. And I talk about that training because that influenced why I started Ripeta.
What I noticed over ten years in working with a lot of clinical data, with so many great researchers, it was really hard to reproduce the work that they were doing. And I worked with clinical researchers at a wonderful medical school, the Washington University School of Medicine in St. Louis. I noticed that while they were good at doing their research, it was really hard to follow back from that “gold” of a research article and trace it back to where things started. So that’s really where Ripeta began. It was looking at how we make the reporting process reproducible, and it’s grown from there.
DG: So what is the problem that Ripeta is solving with its services? And also, if you could explain your services as well, that would be helpful.
LM: We apply artificial intelligence to scientific writing, checking the scientific paper to see if things that should be there are actually in there. Then we produce a report — like a credit report for scientific articles. We actually are not checking if it’s good scientific research yet, we’re saying if it’s hygienic science. So that’s like when you have a kid who comes in and the parent checks to see whether the clothes are clean. The parent isn’t judging them on whether the outfit looks good. They’re just trying to see — did the kid make it to the clean clothes stage?
This speeds up scientific reviews. And if we can speed up certain parts, like making sure that the quality of the paper is better, then let’s do that. So it’s making better science easier. And that’s what we’re trying to do at Ripeta. For example, we check for the study hypothesis, using algorithms. Are the data there that should be there? Not all data can be shared, but have you at least cited where you got your data from or where you put your data? What about the analysis? And even the best researcher forgets some of these things.
Even in the paper we just wrote about the quality of COVID articles we took some preprints, we almost forgot to cite the software that we were using. Because it happens. Because you’re trying to get through things pretty quickly. We put it through our process, and our report told us we forgot something.
And I think in this time of COVID-19, it’s a great time to be able to see why this is needed. Because scientific papers are coming out and scientists are working quickly, which lends itself to possible omissions. And if we can prevent some mistakes, then let’s prevent those.
DG: And what are the key areas that Ripeta looks at to make sure that the paper is going to get a good score?
LM: As I just mentioned, we’re looking for things right now such as the study purpose — is it clearly stated? And you may think that that’s a no-brainer. But it’s not always stated clearly where it should be in a paper. It should be at the end of the background or an introduction so that it leads right into the method section. We want to see if somebody’s sharing the data, where are they sharing it? If they’re not sharing it, do they give a reason for not sharing it? Are they sharing their code, which is something new but is very important for the computational reproducibility of a paper? And do they have their statistical analysis section in there and described? Those are some of the things that we’re looking for. But the goal is to make the paper better and also to help some researchers, who may be newer to the writing process, to know what should be in a paper and making sure those items are in there.
DG: And why is reproducibility so important to the scholarly publishing industry? I have heard about reproducibility, reproducibility, reproducibility. Why is that so important?
LM: Well, it is the main tenet of science. The scientific method is to be able to reproduce the work that you’re doing. Yet as the technology within science has become much more sophisticated, research reproducibility has also become more of a challenge. Part of that, to give an example, is just computational power. The amazing things that we can do with computers has vastly increased what we can do with science. It also has vastly complicated how to record what we’re doing in science. So every time you add a new machine, you want to make sure that it’s not the machine that’s creating the difference. You want to see that whatever you’re testing in the science is making the difference.
For instance, I like to bake as a hobby. And if I’m cooking something — or you watch any cooking show, you know, you want to make sure that everyone has a fair shot at winning on whatever show you’re watching, which means that they need to pretty much have the same oven. Or the same stove, or the same utensils.
We have not done a good job in science of keeping up with categorizing those machines, putting those into the papers. And it does make a difference. What happens if you and I are using different ovens, and one is set to Celsius and one set Fahrenheit, and we don’t know when we’re setting it both at 200? Well, there’s a big difference in the outcome of our cakes. I think one might be edible and the other might not at that temperature. And reproducibility, science is built on that. You assume that you will get one group’s research and that you can build off of it. There is that assumption of reproducibility, that you assume and you build off of it, when you move to future research.
DG: You know, recently, at the STM Annual U.S. Conference, they had a debate about the efficacy of preprints. And the gentleman who — Kent Anderson, who is well known, he was against the preprints. He had talked about the errors that are in the preprints. As I was listening to this debate, I was thinking about the Ripeta tool and how if those preprints had gone through the Ripeta scoring system, they would have made those adjustments, and the paper — the preprints would have been better. What has been your experience thus far in working with the preprints? Do you find that the folks who are utilizing preprints are very open to using the Ripeta tool?
LM: That’s an interesting question. I think that there are two parts to this. One is that the preprint is one of our sweet spots right now, of where we’re finding a niche for Ripeta. There are preprint services such as bioRxiv and medRxiv that have limited pre-checking. Then there are other services coming in, such as ResearchSquare, that are providing a little bit more augmentation to the pre-peer review process, if you will. And that’s where it’s nice for something like Ripeta to come in and do some really quick quality checks. The authors can go back and fix things before they ever go out to peer review. But we still need to recognize that these articles are what one would consider a prototype; these are not fully minted scientific articles.
The other part though, is pre-print services reduce the places for a researcher to go to check their manuscripts. When I’ve spoken with researchers, they’re looking for just one place to go to check their manuscripts because there is a fatigue with apps at this point.
DG: And what has been the result — I guess, what has been the response from the researchers? What have they said about the Ripeta tool and the Ripeta score?
LM: After listening to feedback, our product for researchers has evolved over the last couple of years. We’ve scaled back to refine the checks and thought about looking across scientific guidelines. What are the minimal, but most effective, items that we could automate and check to improve the research? Researchers seem to be very responsive to that — particularly newer researchers — where you quickly run it through. You give it back, and they have some items that they could fix rather quickly.
The other newer development is to give some contextual feedback. So what if something doesn’t meet the criteria? We can give them automatic feedback to suggest a correction. Again, researchers want ways to make science better but also not have all the burden put on them. So there is a balance that we’re trying to find so we can work with them.
DG: Well, you know, that’s very interesting because right now, I guess, there’s like 20 thousand — a gazillion publishers, right? And of course, everybody wants to publish in the top tier journals, but unfortunately that’s not going to happen, but they have to submit their manuscript. And there are so many different manuscript systems out there. Is there a tool where the author can go directly to Ripeta to check what they’re doing before they submit it to the publisher?
LM: We have that in beta right now, where an author could go and upload a paper, test it out, get a report. Otherwise, it is done through an platform integration. And what we’re doing is, if an author just wants to check one paper, then it’s free to them to go and do, and get a report, and be able to make those changes.
DG: That’s nice! Yes.
LM: So we’re coming out with a new version but we’re tweaking some of the algorithms, and we’ll make it better for the researchers. We have focused much more on the API development that we are now getting back to the updating the original report.
DG: Wonderful. Boy, that’s exciting! That’s exciting. And I take it, you’re working with — okay, publishers and special vendors. Have you talked to any of the manuscript companies?
LM: We have worked with publishers and some societies, as well, to test this out. From their perspective, this is some of the feedback that we’re getting from publishers, is the need to get our feedback to the authors, honestly, before they get to the full submission.
DG: Which is great.
LM: So for anyone — which for anyone who’s never sub- mitted a scientific manuscript, it’s not the easiest process there is. You think as a researcher, that you’re done. You’ve got your research. You’ve written your paper, and then — and then it’s a long process to get that paper into the submission status.
DG: And then you find out, it’s the Olympics. And it’s like a decathlon with 10 more events.
LM: Yes. Yes, that’s exactly it. And I understand why it’s like that right now, but that’s the process, and I do think people are trying to make that easier as well. These are exactly the types of things that we’re trying to do, to reduce the burden on the researcher and make the research shine.
DG: You know, just like you did for Ripeta, did it come — I guess it came directly out of your experience about seeing a void. Tell us a little bit about that experience. Of you saying, “Wait a minute, this is a business that we should be doing.”
LM: Yes, I think there are couple of things that drove me to start the company. One is a very selfish part. When I ran the center for biomedical informatics at Washington University, I helped researchers use electronic medical data for legitimate in-house research to reuse — understand important things, like why are people getting so many CT scans? And can we reduce that number?
LM: Working with anesthesiologists to look at the anesthesia given, and some of the outcomes that happen after some surgeries. And then what I would notice is that these wonderful clinical researchers would write a paper. And I wasn’t an author, which was kind of okay because I didn’t write the paper. But our work, our center, and our data weren’t cited either. One of my physician students had just finished a long day of the clinic, and I had brought my complaints up in class. And she said, “Well, why don’t you give me the citation, and I’ll put it in?” And I was kind of taken aback and a little upset. But then it seemed like she was right; that’s what kind of started the business. I was like, okay, I’ll fix this. And here we are.
LM: So kudos to her. I should go back and thank her for making that comment.
DG: So, let’s switch gears here. And let’s talk about this great paper, this three-page paper that you have published, called “Trusting science in a fast-paced world.” What prompt-ed you to write this great paper? Which is really needed right now, when we have folks in the world talking about different drugs for different things that people are dying from, but any-way — I digress.
LM: So that paper started coming along last year. I remember last summer thinking about, how do we trust science? And we, as a country — I think not just the country, but as the world — have been challenging science, and there’s so much information. So I was trying to figure out the things that make me trust or not trust in science, and this is as a scientist. I had the fortunate opportunity to be a keynote speaker at STM in December in London, which is where I first presented the idea.
How do we look at trusting science? Three things to start with: the journal, peer-review, and the scientist. Well, we do look at journals. We do look at whether it’s a high impact journal or not, a lot of us within science. Although let us just admit that some journals are not high impact but have Nobel Prize winners coming from them. So let us not dismiss those very important places that allow science to flourish.
We also have peer review — a peer review process to check the science, and then we have researchers. But we have cracks in each one of those. One of the things that we’ve struggled with as a society with this information age is also the misinformation age. And that’s really what I wanted to get to, is how do we figure out what information to trust and what not to trust?
We also have the peer review process, but there are challenges with that. Because not all peer reviewers know everything. And like we say in the article Sarah Nathan wrote with me, is that it’s kind of like being on a nonprofit board. You know you’re not going to have all of the people cover all of the expertise that you need. But by having that board, you have some security measure there. You have some vetting process there. And what happens is sometimes either you have scientists, unfortunately, who skipped the peer review process unethically or you may not have had the opportunity for peer-review such as the case with pre-prints. In this last case, non-scientists pull that information and think that it has already been vetted, yet there has been no vetting process.
So as we move through this, we need to think about what makes science trustworthy. And it’s different in each society. Science is a very consensus-building place. We challenge each other on ideas, quite heatedly sometimes. And we still many times will go out for coffee or tea with that person. And that’s how science works. It doesn’t work on one paper alone.
So the whole point of this was to start thinking and talking about how we trust in science because it’s very different than, say, in the courtroom. Right? Scientists, as the saying goes, we always ask questions we never know the answer to. And in a courtroom, you never ask a question that you don’t already know the answer to. And these two are converging, and it becomes very important during this time of COVID-19 to understand when things could be good science, but they’re early science. That was the paper.
DG: Very, very good point. Early science. Let’s talk about early science. What are some of the opportunities but also the challenges of early science?
LM: You don’t know if it’s real or not. I mean, to just start there. For example, there’s the very kind of fringe idea that ulcers could be caused by a bacterium, which just sounded crazy because it challenged the common way of thinking within science. So you have to prove yourself in early science. But the other things.
One, not only do you have to prove your idea, you have to have funding to continue to try to prove those ideas. You need different people to challenge those assumptions because we test all those assumptions from different angles to see if something should move through the scientific ecosystem.
I’ll give you an example of early epidemiology. It was believed that alcohol consumption caused lung cancer. And yes, at the time, you knew that people in bars tended to smoke and drink together, that those two came together. It makes sense. You know afterward that it was the tobacco or the toxins within cigarettes that were causing the lung cancer, but there was that confounder of alcohol. So early science is not clear.
And there’s a lot of testing, a lot of playing with different scenarios and hypotheses within early science, which is the part that takes time. It’s that part I’m not sure how much we can speed up. We’re working on it, but that’s the part — back to kind of why I started Ripeta – that is going to take time. Clinical trials take time. So there are some ways to speed up some things. But if we can speed up other parts, like checks in a paper, let’s do it.
DG: That’s best practices. It’s nothing like you have in a best practice where you are submitting your paper with all of the information that’s needed to do reproducibility. I think this is such an — a very exciting tool. You know, in one sense, it sounds very simple, but it doesn’t happen. So if everyone can adopt the Ripeta tool, we’re going to have much better science coming out. And also, it will save the editors and the peer reviewers some time as well.
LM: Well, I hope so. Thank you. I appreciate that.
DG: So, you know, believe it or not, we are running down to the final stretches of our interview. And I want to make sure that folks hear about your leadership style, the type of culture that you want to have at Ripeta.
LM: Thank you for asking that. I think my leadership style is definitely one where I try to cultivate the best out of my entire team, which means that everybody needs to have a voice, needs to bring their ideas to the table, and needs to respect one another. And truthfully, I have a rather hands-off approach on many things, except for when it comes to the data part because I still love the data. But I like for everybody to be able to shine and provide their diverse perspectives, and I like to bring that into the team and cultivate a lot of leaders. So it’s not just dependent on me; it is dependent on the team. And I try to be as transparent and honest with them as possible and expect the same out of this team.
DG: So one other question I’d like to get in before we have to close is, what’s one of your favorite quotes that you want to share with our audience that defines leadership?
LM: So I am going to go back to some stoic philosophy and Seneca because I find some inspiration from there. Seneca said that — this was 2,000 years ago, so things don’t change a lot — he said, “It is a rough road that leads to the heights of greatness.” And I remind myself of that. I look for other great leaders in very interesting places, including my small Twitter group that I follow, of these phenomenal people who are not necessarily representative of great leaders, but I think they’re just phenomenal people who will inspire me to do something fun or positive (see @Afro_Herper who runs #FindThatLizard) during this COVID epidemic because it just raises people’s spirits. I just like to look at that and look to that greatness during this time.
DG: Well, before we go, if you could take 30 seconds to share with our audience, one remaining thought that you have about Ripeta, and what you want to achieve.
LM: What I want to achieve with Ripeta is to let people know that to make science better, we have to make better science easier. And that’s what we’re out to do.
DG: Well, that is excellent. Leslie thank you for joining us on The Innovator’s SAGA!
LM: Thank you so much, Darrell, for having me. I really appreciate it.