Relative importance of Recall, Precision, F-Measure, Informedness, and Markedness metrics to evaluate security tools in Business Critical, Heightened Critical, Best Effort, and Minimum Effort scenarios, according to the declared preferences and familiarity with measures of experts in the domain

Abstract

[EN] The benchmarking of security tools is endeavored to determine which tools are more suitable to detect system vulnerabilities or intrusions. The analysis process is usually oversimplified by employing just a single metric out of the large set of those available. Accordingly, the decision may be biased by not considering relevant information provided by neglected metrics. This work proposes a novel approach to take into account several metrics, different scenarios, and the advice of multiple experts. The proposal relies on experts quantifying the relative importance of each pair of metrics towards the requirements of a given scenario. Their judgments are aggregated using group decision making techniques, and pondered according to the familiarity of experts with the metrics and scenario, to compute a set of weights accounting for the relative importance of each metric. Then, weight-based multi-criteria-decision-making techniques can be used to rank the benchmarked tools. This dataset contains raw data obtained from 21 experts, who declared their familiarity with considered metrics and their preference for each metric in the considered scenarios. Processed data include the consistency ratio of resulting pairwise comparison matrices so inconsistent matrices are rejected - weight = 0.00), the relative contribution of each expert according to their declared familiarity with metrics and computed CRs, and the contribution (weight) of each metric towards each considered scenario.