Establishing a Threshold for Application Inclusion in Peer Review Committee (PRC) Discussions

Research Question:

What threshold would capture the largest percentage of applications that were ultimately approved following discussion at peer review committee (PRC) meeting, while minimizing the percentage of applications that would not be funded without PRC discussion?

Dataset:

Long description
  • Ten OOGP Competitions (2010-2015)
  • 24,101 Applications
  • 3,974 Funded Applications
  • 16.48% Success Rate

Method:

Applications initial ranking percentiles within PRCs prior to face-to-face meetings was compared to success of the application following the discussions.

For all applications above a given ranking percentile, the percentage of ultimately approved applications found within the group was calculated.

Results:

Ninety-eight percent of all applications that were ultimately approved following PRC discussion had an initial ranking of 60 or above, while two percent of applications that would have received funding if included in PRC discussion are lost.

Figure 1. Application Rankings before Peer Review Committee Meetings and Application Success in Ten OOGP Competitions (2010-2015)

Figure 1 long description
Pre-Meeting Ranking Percentiles If a threshold for inclusion in PRC discussions is set at a specific percentile:
% of Applications Ultimately Approved after PRC discussion that are included % of Applications Ultimately Approved after PRC discussion that are lost
0 100% 0%
10th 99.95% 0.05%
20th 99.87% 0.13%
30th 99.77% 0.23%
40th 99.52% 0.48%
50th 99.02% 0.98%
60th 98.01% 1.99%
70th 94.56% 5.44%
80th 84.07% 15.93%
90th 55.54% 44.46%
100th 10.95% 89.05%

The blue graph line represents the percentage of applications that were funded after PRC discussions that are not captured within a bracket from the given percentile to the top ranked application.

Conclusion:

The analysis demonstrates that using a threshold for inclusion on pre-meeting ranking percentiles would have better focused PRC application discussion meetings on those applications with a higher likelihood of receiving funding.

For a breakdown by OOGP competition of the above analysis see the Competition Breakdowns page.

Note: A similar analysis was conducted to determine if there was a threshold above which applications always get funded and do not need to be discussed by PRCs; however, there was no ranking percentile above which 100% of the applications were consistently funded. For analysis please see below.

Research Question:

Is there a threshold on pre-meeting rankings above which applications always get funded and do not need to be discussed by PRCs?

Figure 2 long description
Pre-Meeting Ranking Percentile Percentage of Applications Ultimately Approved after PRC Discussion
0 16.49%
10th 18.06%
20th 20.29%
30th 23.08%
40th 26.83%
50th 31.60%
60th 39.23%
70th 49.98%
80th 65.36%
90th 82.47%
100th 95.39%

The green graph line represents the percentage of applications that received funding after PRC discussions ranked above an identified percentile.

Date modified: