An Introduction to Classification Metrics
Sensitivity and Specificity
Sensitivity
What is the probability of actually having Covid-19 given that you have tested positive? ‘Sensitivity’ or ‘Recall’ answers this question. The sensitivity of a test is defined as the ratio of people who test positive to those who actually have the disease.
A ‘sensitive’ test would measure how receptive the test is to the presence of the virus in an infected person. If a test is highly sensitive, it is capable of correctly identifying people who are infected. Therefore, a negative result on a highly sensitive test would most likely mean that the individual is healthy.
Consider the RT-PCR and Rapid Antigen tests. Although the sensitivity of the RT-PCR test varies according to the sampling technique, it is said to be a lot more sensitive than the Rapid Antigen test making it a more reliable test. However, keep in mind that this increased sensitivity usually comes at the expense of more false-positives (known as specificity).
Specificity
ICMR has stated in its advisory that the Rapid Antigen test’s sensitivity was between 50.6% and 84% in 2 different labs, and the specificity was found to be around 99%. ICMR recommends that people who get negative results on the Rapid Antigen kit must be retested using RT-PCR, given the Antigen test’s low sensitivity. So what do these values indicate?
The ‘specificity’ or ‘true negative rate’ of a test is the ratio of people who test negative to those who actually do not have that disease.
In other words, specificity measures how effective the test is when used on negative individuals.
However, a positive result would not mean they definitely are infected because a highly specific test does not factor in how common the disease is (prevalence). As prevalence increases, the chances of a positive test being correct increase. Specificity only defines how specifically the test can find a healthy person to be negative.
Type 1 and 2 Errors
In statistical hypothesis testing, Anytime we make a decision on a binary classification problem using statistics there are four possible outcomes, with two representing correct decisions and two representing errors. In order to come up with a significance testing, we need to make a null and alternate hypothesis on some population. Null hypothesis tends to be the status quo and alternate is something new. To decide whether to reject the null hypothesis, we take a sample of the population and use it to calculate a statistic . prob of getting that statistic given that null hypothesis - p value if p value is less than significance level alpha then we reject our null hypothesis.
A type 1 error is also known as a false positive and occurs when a researcher incorrectly rejects a true null hypothesis. This means that your report that your findings are significant when in fact they have occurred by chance.
A type II error is also known as a false negative and occurs when a researcher fails to reject a null hypothesis which is really false. Here a researcher concludes there is not a significant effect, when actually there really is.
The consequences of making a type I error mean that changes or interventions are made which are unnecessary, and thus waste time, resources, etc.
Type II errors typically lead to the preservation of the status quo (i.e. interventions remain the same) when change is needed.
