The Accurator (Accuracy Calculator) is a simple web tool designed to help you evaluate the influence of genetic risk scores on prediction of outcomes. There are two tools for different purposes.

The “Accuracy” tool lets you input the known Specificity and Sensitivity of a test, and explore the effect of changing the ratio of cases to controls on Positive and Negative Predictive Values. A fourth input, the total sample size, allows you to see the numbers of individuals expected in each cell.

The “Relative Risk” tool instead computes Sensitivity, Specificity, Positive and Negative Predictive Value, and Accuracy, given input of the prevalence of the condition in the population that was sampled, the relative risk for those above a specified cutoff of the genetic risk score, and the proportion of people above the cutoff.

The lower half of the output then reports the number of events that would be prevented by targeting the called-positive individuals. It also allows you to specify different response-to-treatment rates in the called-positive and called-negative groups, assuming your predictor is designed to identify responders. This tool also computes the Number Needed to Treat (NNT), which is 1 over the difference between absolute risk with and without treatment, as a percentage. NNT is a measure of how many people benefit from being treated, with smaller numbers implying more responders. The sentence at the bottom of the table summarizes the results.

For example, suppose that 5% of people in a population have a condition, and that the top 20% for a genetic risk predictor identifies people on average twice as likely to get the condition as the remaining 80%. In this case, the Precision (Positive Predictive Value) is 8%, and the Negative Predictive Value is 96%: that is, 8% of the 20% with the high risk score will develop the condition compared with just 4% of the remainder. The Sensitivity and Specificity are 33% and 81% respectively, and the Accuracy of the test is 78%. Now, without treatment, 5% of the people develop the condition (eg. 500 of 10,000), 167 of whom are called positive. If 50% of these high risk individuals respond to intervention, compared with just 20% of the remainder, then 150 events will be prevented overall (83 from the high risk group, and 67 from the low risk). This means that targeting 20% of the population prevents just over half the incidents which could be prevented if everyone were treated (and 17% of all incidents). The NNT is reduced almost three-fold, from 67 to 24. By contrast, if half the events are prevented in both the high and low risk groups, it turns out that targeting 20% prevents a third of the incidents that could be prevented, and has a more modest effect on improving the NNT, since the treatment is more effective overall.

Definitions:

Sensitivity (also called Recall, or True Positive Rate) is the proportion of cases who are identified by the test. It is the ratio of called positive cases (true positives) to called negative cases.

Specificity (also called the True Negative Rate) is the proportion of controls correctly identified as such. It is the ratio of called negative controls (true negatives) to called positive controls.

Accuracy is the percentage of correct diagnoses. It is the sum of the true positives plus true negatives divided by the total sample size.

Positive Predictive Value (also called the Precision) is the proportion of people called positive who are true positives. For the called positive group, it can be thought of as the prevalence in that group.

Negative Predictive Value is the proportion of people correctly called controls, namely those among the called negative group who do not develop the condition.

The Relative Risk in this application is the ratio of prevalence of the condition in the called positive to the called negative.