- -

On the effect of calibration in classifier combination

RiuNet: Repositorio Institucional de la Universidad Politécnica de Valencia

Compartir/Enviar a

Citas

Estadísticas

  • Estadisticas de Uso

On the effect of calibration in classifier combination

Mostrar el registro sencillo del ítem

Ficheros en el ítem

[Cerrado]
Nombre: Bella;Ferri;José ...
Tamaño: 805.4Kb
Formato: PDF
Descripción: Versión editorial
Solicitar una copia al autor
dc.contributor.author Bella Sanjuán, Antonio es_ES
dc.contributor.author Ferri Ramírez, César es_ES
dc.contributor.author José Hernández-Orallo es_ES
dc.contributor.author Ramírez Quintana, María José es_ES
dc.date.accessioned 2014-06-09T12:12:01Z
dc.date.issued 2012-06
dc.identifier.issn 0924-669X
dc.identifier.uri http://hdl.handle.net/10251/38005
dc.description.abstract A general approach to classifier combination considers each model as a probabilistic classifier which outputs a class membership posterior probability. In this general scenario, it is not only the quality and diversity of the models which are relevant, but the level of calibration of their estimated probabilities as well. In this paper, we study the role of calibration before and after classifier combination, focusing on evaluation measures such as MSE and AUC, which better account for good probability estimation than other evaluation measures. We present a series of findings that allow us to recommend several layouts for the use of calibration in classifier combination. We also empirically analyse a new non-monotonic calibration method that obtains better results for classifier combination than other monotonic calibration methods. es_ES
dc.description.sponsorship We thank the anonymous reviewers for their comments, which have helped to improve this paper significantly. This work was supported by the MEC/MINECO projects CONSOLIDER-INGENIO CSD2007-00022, COST action IC0801 and TIN 2010-21062-C02-02, GVA project PROMETEO/2008/051, and the RE-FRAME project granted by the European Coordinated Research on Long-term Challenges in Information and Communication Sciences & Technologies ERA-Net (CHIST-ERA), and funded by the Ministerio de Economia y Competitividad in Spain. en_EN
dc.format.extent 20 es_ES
dc.language Inglés es_ES
dc.publisher Springer Verlag (Germany) es_ES
dc.relation.ispartof Applied Intelligence es_ES
dc.rights Reserva de todos los derechos es_ES
dc.subject Classi¿er combination es_ES
dc.subject Classifier calibration es_ES
dc.subject Classifier diversity es_ES
dc.subject Probability estimation es_ES
dc.subject Calibration measures es_ES
dc.subject Separability measures es_ES
dc.subject.classification LENGUAJES Y SISTEMAS INFORMATICOS es_ES
dc.title On the effect of calibration in classifier combination es_ES
dc.type Artículo es_ES
dc.embargo.lift 10000-01-01
dc.embargo.terms forever es_ES
dc.identifier.doi 10.1007/s10489-012-0388-2
dc.relation.projectID info:eu-repo/grantAgreement/MEC//CSD2007-00022/ES/Agreement Technologies/ es_ES
dc.relation.projectID info:eu-repo/grantAgreement/COST//IC0801/EU/Agreement Technologies/ es_ES
dc.relation.projectID info:eu-repo/grantAgreement/Generalitat Valenciana//PROMETEO08%2F2008%2F051/ES/Advances on Agreement Technologies for Computational Entities (atforce)/ es_ES
dc.relation.projectID info:eu-repo/grantAgreement/MICINN//TIN2010-21062-C02-02/ES/SWEETLOGICS-UPV/ es_ES
dc.rights.accessRights Cerrado es_ES
dc.contributor.affiliation Universitat Politècnica de València. Departamento de Sistemas Informáticos y Computación - Departament de Sistemes Informàtics i Computació es_ES
dc.description.bibliographicCitation Bella Sanjuán, A.; Ferri Ramírez, C.; José Hernández-Orallo; Ramírez Quintana, MJ. (2012). On the effect of calibration in classifier combination. Applied Intelligence. 38(4):566-585. https://doi.org/10.1007/s10489-012-0388-2 es_ES
dc.description.accrualMethod S es_ES
dc.relation.publisherversion http://dx.doi.org/10.1007/s10489-012-0388-2 es_ES
dc.description.upvformatpinicio 566 es_ES
dc.description.upvformatpfin 585 es_ES
dc.type.version info:eu-repo/semantics/publishedVersion es_ES
dc.description.volume 38 es_ES
dc.description.issue 4 es_ES
dc.relation.senia 238203
dc.contributor.funder Generalitat Valenciana es_ES
dc.contributor.funder European Cooperation in Science and Technology es_ES
dc.contributor.funder Ministerio de Educación y Ciencia es_ES
dc.contributor.funder Ministerio de Ciencia e Innovación es_ES
dc.description.references Amemiya T (1973) Regression analysis when the dependent variable is truncated normal. Econometrica 41(6):997–1016 es_ES
dc.description.references Ayer M, Brunk H, Ewing G, Reid W, Silverman E (1955) An empirical distribution function for sampling with incomplete information. Ann Math Stat 5:641–647 es_ES
dc.description.references Bella A, Ferri C, Hernandez-Orallo J, Ramirez-Quintana M (2009) Calibration of machine learning models. In: Handbook of research on machine learning applications. IGI Global, Hershey, pp 128–146 es_ES
dc.description.references Bella A, Ferri C, Hernández-Orallo J, Ramírez-Quintana M (2009) Similarity-binning averaging: a generalisation of binning calibration. In: Intelligent data engineering and automated learning—IDEAL 2009. Lecture notes in computer science, vol 5788. Springer, Berlin/Heidelberg, pp 341–349 es_ES
dc.description.references Bennett PN (2006) Building reliable metaclassifiers for text learning. PhD thesis, Carnegie Mellon University es_ES
dc.description.references Bennett PN, Dumais ST, Horvitz E (2005) The combination of text classifiers using reliability indicators. Inf Retr 8(1):67–98 es_ES
dc.description.references Blake C, Merz C (1998) UCI repository of machine learning databases. http://www.ics.uci.edu/~mlearn/MLRepository.html es_ES
dc.description.references Breiman L (1996) Bagging predictors. Mach Learn 24:123–140 es_ES
dc.description.references Brier G (1950) Verification of forecasts expressed in terms of probabilities. Mon Weather Rev 78:1–3 es_ES
dc.description.references Brümmer N (2010) Measuring, refining and calibrating speaker and language information extracted from speech. PhD thesis, University of Stellenbosch es_ES
dc.description.references Canuto A, Santos A, Vargas R (2011) Ensembles of artmap-based neural networks: an experimental study. Appl Intell 35:1–17 es_ES
dc.description.references Caruana R, Munson A, Mizil AN (2006) Getting the most out of ensemble selection. In: ICDM ’06: proceedings of the sixth international conference on data mining. IEEE Computer Society, Washington, pp 828–833 es_ES
dc.description.references Caruana R, Niculescu-Mizil A (2004) Data mining in metric space: an empirical analysis of supervised learning performance criteria. In: Proceedings of the tenth ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’04. ACM Press, New York, pp 69–78 es_ES
dc.description.references Cohen I, Goldszmidt M (2004) Properties and benefits of calibrated classifiers. In: Proceedings of the 8th European conference on principles and practice of knowledge discovery in databases, PKDD ’04. Springer, Berlin, pp 125–136 es_ES
dc.description.references Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30 es_ES
dc.description.references Dietterich TG (2000) Ensemble methods in machine learning. In: Proceedings of the first international workshop on multiple classifier systems, MCS ’00. Springer, London, pp 1–15 es_ES
dc.description.references Dietterich TG (2000) An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization. Mach Learn 40:139–157 es_ES
dc.description.references Fahim M, Fatima I, Lee S, Lee Y (2012) Eem: evolutionary ensembles model for activity recognition in smart homes. Appl Intell, 1–11. doi: 10.1007/s10489-012-0359-7 es_ES
dc.description.references Ferri C, Flach P, Hernández-Orallo J (2004) Delegating classifiers. In: Proceedings of the twenty-first international conference on machine learning, ICML ’04. ACM Press, New York, pp 37–45 es_ES
dc.description.references Ferri C, Hernández-Orallo J, Modroiu R (2009) An experimental comparison of performance measures for classification. Pattern Recognit Lett 30:27–38 es_ES
dc.description.references Ferri C, Hernández-Orallo J, Salido M (2003) Volume under the ROC surface for multi-class problems. Exact computation and evaluation of approximations. In: Proceedings of 14th European conference on machine learning, pp 108–120 es_ES
dc.description.references Flach P, Blockeel H, Ferri C, Hernández-Orallo J, Struyf J (2003) Decision support for data mining: an introduction to ROC analysis and its applications. In: Data mining and decision support: integration and collaboration. Kluwer Academic, Boston, pp 81–90 es_ES
dc.description.references Freund Y, Schapire RE (1996) Experiments with a new boosting algorithm. In: International conference on machine learning, pp 148–156 es_ES
dc.description.references Gama J, Brazdil P (2000) Cascade generalization. Mach Learn 41:315–343 es_ES
dc.description.references Garczarek U (2002) Classification rules in standardized partition spaces. PhD thesis, Universitat Dortmund es_ES
dc.description.references Gebel M (2009) Multivariate calibration of classifier scores into the probability space. PhD thesis, University of Dortmund es_ES
dc.description.references Hand DJ, Till RJ (2001) A simple generalisation of the area under the ROC curve for multiple class classification problems. Mach Learn 45:171–186 es_ES
dc.description.references Hoeting JA, Madigan D, Raftery AE, Volinsky CT (1999) Bayesian model averaging: a tutorial. Stat Sci 14(4):382–417 es_ES
dc.description.references Khor K, Ting C, Phon-Amnuaisuk S (2012) A cascaded classifier approach for improving detection rates on rare attack categories in network intrusion detection. Appl Intell 36:320–329 es_ES
dc.description.references Kuncheva LI (2002) A theoretical study on six classifier fusion strategies. IEEE Trans Pattern Anal Mach Intell 24:281–286 es_ES
dc.description.references Kuncheva LI (2004) Combining pattern classifiers: methods and algorithms. Wiley-Interscience, New York es_ES
dc.description.references Kuncheva LI (2005) Diversity in multiple classifier systems. Inf Fusion 6(1):3–4 es_ES
dc.description.references Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51:181–207 es_ES
dc.description.references Lee H, Kim E, Pedrycz W (2012) A new selective neural network ensemble with negative correlation. Appl Intell, 1–11. doi: 10.1007/s10489-012-0342-3 es_ES
dc.description.references Maudes J, Rodríguez J, García-Osorio C, Pardo C (2011) Random projections for linear svm ensembles. Appl Intell 34:347–359 es_ES
dc.description.references Murphy AH (1972) Scalar and vector partitions of the probability score: part II. n-State situation. J Appl Meteorol 11:1182–1192 es_ES
dc.description.references Platt JC (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in large margin classifiers. MIT Press, Boston, pp 61–74 es_ES
dc.description.references Raftery AE, Gneiting T, Balabdaoui F, Polakowski M (2005) Using Bayesian model averaging to calibrate forecast ensembles. Monthly Weather Rev, p 133 es_ES
dc.description.references Rifkin R, Klautau A (2004) In defense of one-vs-all classification. J Mach Learn Res 5:101–141 es_ES
dc.description.references Robertson T, Wright FT, Dykstra RL (1988) Order restricted statistical inference. Wiley, New York es_ES
dc.description.references Souza L, Pozo A, Rosa J, Neto A (2010) Applying correlation to enhance boosting technique using genetic programming as base learner. Appl Intell 33:291–301 es_ES
dc.description.references Tulyakov S, Jaeger S, Govindaraju V, Doermann D (2008) Review of classifier combination methods. In: Marinai HFS (ed) Studies in computational intelligence: machine learning in document analysis and recognition. Springer, Berlin, pp 361–386 es_ES
dc.description.references Verma B, Hassan S (2011) Hybrid ensemble approach for classification. Appl Intell 34:258–278 es_ES
dc.description.references Wang C, Hunter A (2010) A low variance error boosting algorithm. Appl Intell 33:357–369 es_ES
dc.description.references Witten IH, Frank E (2002) Data mining: practical machine learning tools and techniques with java implementations. SIGMOD Rec 31:76–77 es_ES
dc.description.references Wolpert DH (1992) Stacked generalization. Neural Netw 5:241–259 es_ES
dc.description.references Zadrozny B, Elkan C (2002) Transforming classifier scores into accurate multiclass probability estimates. In: Proceedings of the eighth ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’02. ACM Press, New York, pp 694–699 es_ES


Este ítem aparece en la(s) siguiente(s) colección(ones)

Mostrar el registro sencillo del ítem