Estrategias metodológicas para la construcción de indicadores compuestos en la gestión universitaria

Abstract

Over the past few years, higher education institutions have been submitted to various changes largely conditioned by the demands of society. The change from a university for the elite to a university for the masses, the increasing demand for the improvement in quality, the globalization of knowledge, the competitiveness at national and international levels, the development of internal improvement procedures, the changes in relationships with administration and society in general; are some of the unavoidable factors of growing importance in the field of higher education in Europe and around the world.

Accountability, to build trust between society and universities, and the demand for transparency, for the design and implementation of plans for improvement and decision-making, are key aspects for meeting society¿s needs. Nevertheless, it is a fact that universities offer less openness than desired, although in recent years there has been a common effort to change this situation.

Spanish universities have developed several initiatives to assess the quality of university activities. In particular, one of the relevant challenges to ensure quality in the Spanish university system is the assessment of the university teaching staff activity. The importance of the assessment activities is ever increasing. Such activities can help ensure the quality of educational processes affecting students, the university community and society in general. The evaluation results will allow the design and implementation of plans for improvement for the performance of university teaching staff at universities. It is therefore important to have measurement tools to assess the university teaching staff activity in three dimensions: teaching, research and management. However, commonly the assessment instrument used does not meet the need for the transparency which society demands.

The aim of this research is to design methodological strategies for building composite indicators to assess and measure multidimensional concepts related to higher education management. To that end, a thorough review of the statistical techniques that can be applied in the development of composite indicators is performed. Furthermore, the importance of the indicators as a basic tool for suitable quality management and accountability in universities is studied.

After reviewing a summary of the use of indicators in higher education, this study focuses on the assessment of the academic activities of the Universitat Politècnica de València teaching staff. First of all, an exploratory analysis of all indicators proposed by the university, for their teaching staff assessment, is performed. To do this, statistical techniques of dimensionality reduction, and classification techniques are applied. Subsequently, different methodologies for constructing a composite indicator that measures teaching activity are proposed, according to the normalization and weighting techniques applied. Lastly, the analytical characteristics of each of the methodologies proposed are studied, using uncertainty and global sensitivity analysis techniques to evaluate the quality and robustness of the proposed models, according to the goals set by the university. The uncertainty analysis is performed using a quasi-random simulation process and applying a non-compensatory aggregation technique on the results, known as the Copeland method. A nonparametric approach, known as the State Dependent Parameter, based on the Kalman filter and smoothing techniques, is performed to apply global sensitivity analysis. The uncertainty and sensitivity analysis are considered necessary techniques to provide validity and methodological support for the composite indicators proposed.

Finally, the generalization of the composite indicator referring to teaching activity is studied, and a composite indicator is designed to evaluate the academic activity of each teacher with respect to the rest. This is based on the three main dimensions of university teachers: teaching, research and university management. In order to do this, a non-parametric weighting and aggregation technique called Benefit of the Doubt is applied. This technique measures the relative effectiveness in the development of academic activities of each member of university teaching staff, according to their academic profile.

The methodological techniques proposed in this study are applicable in any field of research, and they provide a robust and transparent construction scheme of composite indicators.