Hi there, my name is Tom Gregory and I'm an Associate Professor of Female Public Medicine and Reconstructive Surgery at Oregon Health and Science University in Portland, Oregon. And we're here today to talk about the NIH grant review process, especially as it pertains to somebody who might be trying to put together an application that might be going to the NIH. You've been part of some of these webinars and what we want to talk about is why you might be presenting or putting together your grant the way you are. And so I think that understanding what happens when you submit will help you understand why it is that you're doing what you're doing. So the objectives will be to sort of be a nuts and bolts discussion to understand the process after grant submission. And like I said, it'll help you understand the rationale for why the grant application is being written and edited the way we are suggesting as part of these webinars. So what happens? So the submission of a grant process is the following. So you, the grant applicant, or the principal investigator, are thinking about your research idea and you're initiating the project, but it's the institution of the principal investigator who will actually submit the application. So as you're working on your application, you'll get it put together and the institution will review it and make sure that all the T's are crossed and the I's are dotted and that you're going to be able to do what you say you're going to do. And then it's submitted to the NIH, for example. And then hopefully, with luck and with perseverance, the NIH will be able to receive funds and then the hard part starts and you'll be able to conduct the research that you talked about. But there's a big piece that is kind of missing there of understanding, and that's this black box that is the NIH, and that's what we're going to kind of talk about here now. So what's in that black box? The NIH is one of eight health agencies that are part of the U.S. Department of Health and Human Services, and the NIH is actually composed of 27 separate institutes or centers. And for the most part, those of us in female pelvic medicine and reconstructive surgery will be interfacing with the ones here in yellow, either the National Institute on Aging, the National Institute of Child Health and Human Development, Kennedy, the National Institute of Diabetes and Digestive and Kidney Diseases, and the Center for Scientific Review, which is actually a separate part, and we'll talk about how those are different here in a moment. Although it's possible that you might be finding your research fits into one of these other institutes, and I encourage you to kind of understand what each of these institutes' foci are so that you can make sure that your work is being considered in the right location. So there are something called a scientific review group, and there are different types of scientific review groups, and it depends, again, on whether it's being conducted through the Center for Scientific Review or the institutes themselves. So the Center for Scientific Review has several regular study sections, and I'm actually on one of the Center for Scientific Review's regular study sections. It's called now UGPP, and they also will have special study sections that are sort of focusing on a particular topic that is outside of the regular study sections, and then they also have special emphasis panels kind of put together in a similar fashion. And in general, the scientific review groups at the Center for Scientific Review will review research project grants, which are the typical ones that you may know about, the R01s, the R21s, and things like that. But they also look at academic research enhancement awards, postdoc fellowships, F32s, small business innovation research, SBIRs, and then some shared instrumentation grants. Now the institutes, for example, NIDDK or NICHD, also have special, or excuse me, sitting scientific review groups that function very, very similarly to the ones at the CSR. They also have contract review committees. The institutes, they'll look at things like program project grants, PE grants, centers, institutional training grants, such as like a K12, contracts, potentially RFA, so a particular one-time funding announcement that only the one institute wants to be able to fund. And then career awards, for example, K08s and K23s are examples. And then some other smaller grants might be instituted, initiated by the institute and then reviewed by the scientific review groups at those institutes. So what is the system of the peer review that then happens with the review groups? So let's imagine we're sending our grant to the Center for Scientific Review. So the CSR will receive your grant and then assign it to a specific study section, and by virtue of what the application is about, to one of the institutes or centers of the NIH. Then the study section will, in a group, evaluate your application for, and this is important for scientific merit. Once that happens, then the institute will evaluate your grant for program relevance. What I mean by program relevance is, does it fit in with what the institute has sort of declared as what they want to focus on in the short and long term? And you can get access to those program ideas from the institute's websites directly. Once that happens, then there's something called an advisory council and other boards, which then, based on the evaluation by the study section and whether or not it fits within the institute's program relevance, will actually recommend an action such as fund or not fund. And then finally, the director of the institute involved will take that final action based on the recommendation from the advisory council and suggest that money be allocated or not. And so the institute director is the final person. Okay, so who assigns the applications to the different locations, the different scientific review groups? Well, there's something called a referral officer, who typically is a professional scientist, often PhDs, most of whom also serve as what's called a scientific review officer of study sections. And you have the ability to help influence the assignment to the different review groups in your application through something called a cover letter, which hopefully, by virtue of you doing some research beforehand, you might recognize that there is a particular study section, which you think, based on its focus or based on the makeup of the people that are there, might be the most appropriate. And you can get access to study section rosters and overall interests because at the institutes as well as the Center for Scientific Review on the website, you can determine the different review groups and study sections and find out what their standing rosters are and then go from there. Also, you can, in your cover letter, indicate the institute or institutes you might get dual funding or opportunity for that, that might be likely to be interested in funding your work. So that's, you know, you can begin just by your cover letter to try to help understand and direct where your work should go. Once you have had your assignment to a study section, that information is actually sent back to you. You'll get an email or a letter or through the, what's called the ERA Commons website, that information will be sent to you. You'll get an assignment of a number for your grant. You'll get the information of the name, address, and telephone number of the scientific review officer of the review group. And again, that's somebody who you can connect with in contact. And then an assigned institute, again, not necessarily the CSR, but the institute like NICHD or NIDDK, and a telephone number for that separate contact. Okay. So the study section scientific review officer, his or her job is to make sure that your grant gets due diligence and that there is appropriate expertise on the entire panel to review the entire panel of grants that will be looked at and received. That person will pull together a group of reviewers and will distribute the grants among these different reviewers. Now I mentioned that there are standing members of a study section, but also pretty regularly there are multiple ad hoc reviewers recruited depending on the makeup of the grants to be reviewed to assure that everybody gets their fair shake and fair review. Now I just want to make sure that you understand that the scientific review officer at the Center for Scientific Review or even your institute is not necessarily, is not the program officer you may have talked to as you're thinking about your grant and prior to submitting it. These are different people. So there's a program officer and then a scientific review officer, different people. Once your grant is submitted, you're going to be talking potentially to the scientific review officer to understand the mechanics of what has to happen in case there are issues. Okay, so you have sent it in, it has been looked at by the SRO and the group has been assembled and then the grants themselves will be forwarded to the various reviewers. Hopefully beforehand the scientific review officer will have looked for evidence of conflict of interest and appropriate assignment, but occasionally that then has to be reviewed by the actual reviewers and we do, we try to make sure that there is absolutely no conflict of interest or even the appearance of conflict of interest. We have to sign multiple things before and after the meeting to assure that that's the case. Generally, each reviewer in the group will have anywhere from 8 to 12 grants to review in full as a primary, secondary, or tertiary or reader reviewer. While I did mention that the scientific review officer tries to make sure that there's expertise in the entire group and for each grant essentially, not every reviewer of the primary, secondary, or tertiary reviewer will be necessarily an expert in your field or topic. I really think that this is important for you to understand because that's where your grant writing ability will help the persons who are not necessarily fully familiar with all the details in your particular niche or field. You have to make sure that your grant is being written to all the different types of people who might be reading your grant. It has to be clear as well as detailed. That's a trick that hopefully over the course of this webinar we are helping you understand and learn how to do. I mentioned that each reviewer will have 8 to 12 grants to review in full as a primary, secondary, or tertiary, but all the reviewers will receive access to all the grants. That can be anywhere from 60 to 75 in total. You can imagine there's a ton of things that have to be done for these review meetings. The clearer and easier and more interesting and precise that your grant can be will be the more likely to be understood and cared for in the review sessions. Let's talk about the Center for Scientific Review because that's the one I know better than the others, but it's sort of the clearinghouse for most of the applications. CSR actually reviews up to 115,000 applications per year and growing. It uses almost 18,000 external reviewers to do that. The study sections provide judgment. Again, this is something that's often unclear. The study sections provide judgment of the scientific merit, what is good and bad about the science of it, but the study sections themselves, while they give scores to the grants, are not the people that determine the funding. I mentioned that in one of the earlier slides that talked about the fact that the advisory council and the institutes themselves make the final judgment based on not only program relevance but also the scores. A combination of all that is done by not the study section but by other people. Study section membership is typically about 16 members, and I mentioned that there are regular members as well as ad hoc people. The term is for three to four years, and they generally set that up so that there are staggered appointments, so not everybody is starting fresh and ending, so there's always more experienced people on the study sections. And I know that they're trying to have some of the regular members be there but also get sort of more junior folks to have maybe a smaller load but to make sure that there's an increase in the number of reviewers over the course of the future. In general, it seems that PhDs predominate, but constituents will change based on the grants to be reviewed, and there are, again, by the scientific review officer's efforts, the appropriate balance of PhD basic scientists, clinician scientists, clinicians who are involved in clinical research to make sure that there is a good review of the grants. And I know this to be the case that, in general, there's a significant desire to support your grant at the meeting. Your reviewers are actually there to support you, so everything that you can do while you're writing your grant to make it easy for them to support your grant, then that will be fantastic. And, again, I know that this is part of the webinar that we are trying to show you how to put together your grant so that it is very digestible but precise. You know, there's a criteria for selection in the study section. Part of it is that there has to have been a demonstrated scientific expertise in the past, hopefully mature judgment, balanced perspective, open-minded, and objectivity. Again, we are hoping that by the complement of the study section that we're giving everybody's application a fair and balanced review. Of course, there's hopefully personal integrity. It's really important regarding confidentiality because the study section is going to look at multiple different grants and it wouldn't be fair to talk about them openly to other folks. Significant appropriate representation of women minority, like I said, clinical versus basic science. And that, hopefully, that person who's on the study section can articulate opinions in a clear and relatively concise manner because of the sheer number of grants that have to be reviewed. Okay. So, do you now understand sort of what the study section complement and what it comprises? What happens now particular to your grant? All right. So, your grant is in the hands of a reviewer and that reviewer and the two other reviewers will do a review using established criteria that you perhaps are aware of. And these scores for each criteria are uploaded or entered essentially blinded to the other reviewers prior to the meeting for the initial discussion. So, hopefully, you know, each person gets to put in their information before hearing about what other people think. Then, there is the meeting. And, generally, the standing meetings happen about three times per year and they're frequently two days. Occasionally, if there's a fewer number of applications, they can be one day. And that meeting occurs with the members of the study section, the chair of the study section, the scientific review officer. And then, relevant institute representatives to listen to the discussion. And on the study section that I am on, typically, that is NICHD and NIDDK representatives that either are on the phone or sitting there in the room listening to the discussion trying to help frame their discussions based on what they hear. I mentioned conflict of interest and here are the things that sort of are conflict of interest bullets. If you have worked with one of the key personnel on one of the grants in the past three years or currently, then you have conflict. If there's a potential or even a look of potential financial gain, then there's conflict. If you're obviously a close relative of the key personnel, if there is actually just a PI from your institution and you don't have any necessarily working relationship with them at all, you are in conflict. If it's known that you have a significant scientific disagreement with one of the applicants, of course, if the PI was your student or a major dissertation advisor, there's conflict. If it's known that you are applying for a job at the PI's institution or vice versa. So these are sort of the main conflicts that would have you recused from not only reviewing the application and from sitting in on the discussion so you're asked to leave. Okay. So here we're getting sort of more to the crux of things. So each application, like I mentioned, is assigned three to four reviewers. And again, the primary and the secondary reviewers will essentially write a full written critique based on each of the criteria, which I'll talk about in a minute. The tertiary reviewer or the discussant or the reader, there's a lot of different terms for it, will submit a review, but they won't necessarily submit information about each of the criteria, but rather a summary paragraph that describes why they feel great about your grant or have some concerns about it. So that's, like I said, three to four assigned reviewers will do this at a time. So here are the review criteria, which, again, you have kind of written your grant to focus on this. So there is significance, the approach, innovation, investigators, and environment are sort of the main five review criteria, and they are scored, each of them separately, anywhere from one to nine, one being exceptional and nine being poor. Also as part of your application, and this is really important, that there is an evaluation of safety and appropriateness for human and animal study. Although that's not necessarily scored in the same fashion, if you don't demonstrate that there is concern for human or animal safety, including, for example, things like how many subjects or how many animals you need to use to demonstrate what you want to do, then there can be, it may be sent back to you for incompleteness. And then finally, so you've got all these five different criteria, there is a final, what's called overall impact, which is sort of the main score that is finally used, and that impact, again, is scored from one to nine. And the way that that's thought of is that your application will exert a sustained and powerful influence on the research fields involved. And it does get a little bit confusing as to how that is applied, and so we'll talk about that in a few minutes. Okay, so another thing that's really important to know is how these applications get reviewed. Previously, all the applications from all the different types of applicants would be scored and then discussed in order of scoring. But in an attempt to make sure that especially new investigators were considered in a more beneficial manner, they began batching reviews several years ago. And so the conduct of the meeting will be that there will be essentially two or three sessions that are focused on the different types of applications. So at some point in the meeting, the new investigators will be reviewed and their R01 applications will be reviewed. So if you have never had significant funding from NIH, then you might be considered a new investigator and all your applications will be scored and compared to other new investigators. So that's one batch. Another one is established investigators. Their R01 is either a new, excuse me, an R01 that's a different type or a renewal, or something that is called an R21, which is a smaller grant. But if you're submitting an R21, but you're a new investigator, it's actually not, you're not going to be separated into the new investigators section because it's an R21, not an R01. And then all the other mechanisms, R03s, et cetera, will be batched separately and talked about separately. And I mention that this is important because generally we don't read and discuss, we don't discuss all the 75 grants. Really, we only have time to talk about the top, let's say, 40 to 50 percent based on initial scores. And so you can imagine that it's likely that maybe new investigators might have scores in general that are lower than established investigators just because of experience in grant writing, but that at least the top 50 percent or so of the new investigators will be discussed and talked about at the meeting rather than being relegated down to a level that might be a cutoff score for other applications. So it's actually quite beneficial. So, again, the top 50 percent or so of initial scores will be discussed. The remainder will be what's called not discussed. As I remembered, as I said before, all reviewers will have previously submitted their criterion scores and their overall impact score electronically, blinded essentially to the other reviewers. As the discussion begins, everybody will say, okay, I gave it an overall impact score of 3. And then discuss what's going on and then it's possible that they might have initially submitted a 3 and then read some of the reviews prior to the meeting from the other reviewers. And then they may have done a little bit of changing either up or down based on the reviews that they read prior to the meeting. But this is all prior to the actual discussion. Okay, so you, your grant is in front of the primary reviewer. They're at the meeting and they're getting ready to talk about it. And this is where I think it's really important for you to understand. The primary reviewer will essentially summarize your project in five minutes or less, in five or six sentences. So the better you have done in conveying what it is you're trying to do and how you're going to do it, the better likely you are to have your primary reviewer be able to re-summarize that on your behalf. So it's a short little review of your project, not in significant detail. To be honest, most of the group will get the gist of your study, of your plan, from the primary reviewer's summary. They may be sitting there listening to the summary and reading, let's say, the abstract of your grant application while they're listening to this primary reviewer. So again, it's important that that primary reviewer can convey what it is that you're trying to do. That person will then point out strengths and weaknesses of your grant as essentially bullet points in each of the five criteria. So they'll just go through and talk about the significance of the application, the innovation, the investigator, the environment, and such. And talk about the weaknesses and the strengths. Then, so the primary reviewer has done his or her review. The other reviewers will then essentially point out anything that is either complementary or contradictory to the points raised by the primary reviewer and things that maybe the other reviewers had not addressed. Then, it's sort of open for discussion from all the rest of the reviewers. And these points and subsequent discussions can be quite influential to adjustments and they can be both up or down of initial scores. So the opportunity to have a discussion of your application often is quite beneficial. I've seen grant scores shift quite significantly during these discussions. So following the discussion, the chair of the meeting, and that's one of the members, not the scientific review officer, will try to, after having listened to the discussion, summarize the salient points that might influence the final score. And then all the assigned reviewers will restate their scores, which I will mention can often change up or down. So after the assigned reviewers state their scores, the remainder of the group will then vote based on the scores that they heard from the primary reviewers. In general, unless they contribute significantly to the discussion and state otherwise, all the reviewers will score within the range set by the assigned reviewers. So, for example, if the primary reviewer scored a 2, secondary reviewer scored a 3, and the tertiary reviewer scored a 4, essentially all the remainder of the group, the other 15 people or so, will score between a 2 and a 4. If somebody wants to score higher or lower than that range, they need to pipe up and say that they're going to score outside the range, hopefully having put in their 2 cents worth during that discussion to clarify why that would be the case. And from that, you'll get the final impact score of your grant. Okay, so for the most of you listening to this, it may not be as relevant anymore, but it used to be that grants were scored in a range between 1 and 5, and everything was really quite bunched up. And in an attempt to spread the scores out, the NIH reoriented the scoring to be a 1 to 9 system as shown here. Essentially, 1 is exceptional type of grant and 9 is poor. How do you characterize that, though? And this is what this and the next couple of slides are going to try to help you understand. So, you know, 1, 2, or 3 is thought to be a more high-impact application, whereas 7, 8, and 9 is a lower impact. And you can imagine that as you go from 1 to 9, the weaknesses begin to outweigh the strengths of the application. So that's sort of a general understanding of how that might be. To sort of clarify even more specifically, one of the things that we'll do at the meeting is have this matrix out for us to sort of think about. An exceptionally high-impact grant will be generally really, really strong with essentially no weaknesses, whereas something like a 3 will be quite strong with only some minor weaknesses. And so some people will try to clarify as they talk about the strengths and weaknesses whether or not they think it's a minor weakness, moderate weakness, or major weakness, and therefore help them guide their scoring. This is what has been kind of used for the last year and a half to two years. And, you know, it turns out that even some of our well-known, well-respected, well-established investigators will, for any given grant, receive scores in the yellow range. And so the idea is that a medium to good score is in the 4s to 5s. And so it's quite common for us to see grants from established investigators scored in this range. So just because you're an established investigator doesn't mean you're obviously going to get or automatically going to get a higher score. And another way to try to help us understand this is this slide in which, you know, trying to help us understand the likelihood for a project to sustain, to exert a sustained or powerful influence on the research field. So you can imagine that the application may have some strengths and some weaknesses. How could you then convert that to an overall impact score? So high impact are applications that are addressing a problem of high importance or high interest in the field. And it may have some or no technical weaknesses. A medium application will be something that might be addressing a problem of high importance in the field, but weaknesses in the criteria bring down the overall impact to medium. So the applications that may be addressing a problem of moderate importance, but it's possible that that application technically has no significant weaknesses in the approach. So if it's not a really high importance problem, then it may be relegated to a more medium application. And then, of course, the low is perhaps of low or no importance with some or no technical weaknesses as well. So it's really, there is definitely some stringent criteria as well as some subjective things that are, that's why the discussion is important. Okay, your application has been submitted, reviewed, and talked about. After the meeting, the scientific review officer will summarize the review of the discussed grant applications in a paragraph or two, and then forward that summary to the institute and applicant as sort of a document. And that's usually posted on the Commons website that you'll get access to. Priority scores. So we've just told you that the scores were scored from 1 to 9, but the priority score itself is essentially an average of all the scores multiplied by 10. And then those priority scores are then turned into sort of a percentile, which is tabulated from not only a result of that current meeting, but also two previous review meetings of that review group. The National Advisory Council is, like I mentioned, is a different group, and they take a look at the percentiles. And, you know, these are scientific and public representatives, about 12 to 15 of them. They meet also three times a year to advise the institute on programs and priorities and review these research applications. And so their application, like I said, their work is based on not only the scientific merit, but also the relevance to the institute's programs and priorities. So those impact scores, let's say it's a 24 or a 32, are translated into percentiles, basically looking at the distribution of the scores and making funding decisions. Percentiles will allow an important comparison between study sections. So I might be in a study section that tends to score in the 2s and 3s frequently, whereas another study section might tend to score between 4s and 5s. It would be really unfair if the study section that only scores in the 2s to 3s had all their grants funded and the study section that scored between 4s and 5s didn't. So percentiles allow these types of comparisons between study sections' priority scores. Again, I mentioned that there is a kind of a three-group or three-study-section running average that helps them identify what these percentiles should be. Unfortunately, and this actually may be old news and worsening at the moment, but currently most institutes are funding at the 8th to 15th percentile, and I think that number is certainly shrinking. That means that, of course, only the very few applications are receiving funding. And a lot of times they are truly bunched up, but they have to have some criteria on which to make decisions based on the money that they have. The institute, as I mentioned, the awarding of the grants are occurring. They are awarding the grants. They will act on the recommendation of the review group and the advisory council, again, based on scientific merit, programmatic considerations, and availability of funds. If everything is hunky-dory, then that money will be sent to your institution, which will dole out the money to you, the investigator for your project. Hooray, hooray! I did want to mention one other thing here. Here we go. You can actually get a lot more information than I have presented here and kind of fill in the gaps by looking at a couple of websites. So the first one here is from the Office of Extramural Research NIH, just generally about grants and funding. You can go through all these different bullets and kind of focus on a few of the pieces. So I encourage you to go to this website up here, grants.nih.gov, slash grants, slash about, underscore grants, dot A-T-T-M. Another very important website, and this will kind of be helpful to you after you've listened to this webinar, is go ahead and go to this website, in which the peer review process is sort of revealed even more clearly. So they actually did a little movie that put together a fake study section to talk about some grants, and so you can see the way in which people interact and how that happens. It's a great movie. I encourage you to take a look at it. So I hope that this clarifies for you the black box that is the application process. It takes a lot of time to understand the alphabet soup of the NIH and what different institutes are, and I've been doing this now for 12 or so years. I still am learning about the different types of grants, the different players and the different places, and if there are any questions, I'm happy to receive questions from the scholars or by e-mail. You can find my e-mail on the OGS website, and I hopefully can kind of answer some of them. So thank you very much for your time this morning, and good luck to you.

NIH grant review process

research project

peer review

grant writing

scoring process

criteria assessment

funding decisions

scientific merit