The road safety barriers are today designed and installed in compliance with the European standards for Road Restraint Systems (EN 1317), which lays down common requirements for the testing and certification in all EU countries. The introduction of the European Union (EU) regulation for safety barriers, which is based on performance, has encouraged European road agencies to perform an upgrade of the old barriers installed before 2000, with the expectation that there will be safety benefits at the retrofitted sites. Due to the high cost of such treatments, a benefit-cost analysis (BCA) is often used for site selection and ranking and to justify the investment. To this aim a crash modification factor (CMF) has to be applied and errors in the estimation of benefits are directly reflected in the reliability of BCA. Despite the benefits of empirical Bayes before-after (EB-BA) analysis or similar rigorous methods are well-known in the scientific world, these approaches are not always the standard for estimating the effectiveness of safety treatments. To this aim, the differences between the EB-BA and a naive comparison of observed crashes before and after the treatment are presented in the paper. Crash modification factors for total and target crashes are estimated by performing an EB—BA based on data from a motorway in Italy. As expected the results suggest a strong safety benefit for the ran-off-road crashes by reducing the number of severe crashes (fatal and injury). The statistical significance of results obtained by the EB-BA approach show that the retrofits are still cost-effective. The comparison pointed out as selection bias effects can overestimate the safety benefit of the retrofits when a naive approach is used to estimate the CMF and how those can significantly affect a benefit-cost analysis.