- -

Research portfolios in science policy: moving from financial returns to societal benefits

RiuNet: Institutional repository of the Polithecnic University of Valencia

Share/Send to

Cited by


Research portfolios in science policy: moving from financial returns to societal benefits

Show full item record

Wallace, M.; Rafols García, I. (2015). Research portfolios in science policy: moving from financial returns to societal benefits. Minerva. 53(2):89-115. https://doi.org/10.1007/s11024-015-9271-8

Por favor, use este identificador para citar o enlazar este ítem: http://hdl.handle.net/10251/84529

Files in this item

Item Metadata

Title: Research portfolios in science policy: moving from financial returns to societal benefits
Author: Wallace, Matthew Rafols García, Ismael
UPV Unit: Universitat Politècnica de València. Instituto de Gestión de la Innovación y del Conocimiento - Institut de Gestió de la Innovació i del Coneixement
Issued date:
[EN] Funding agencies and large public scientific institutions are increasingly using the term "research portfolio" as a means of characterizing their research. While portfolios have long been used as a heuristic for ...[+]
Subjects: Science policy , Research portfolio , Prioritisation , Research landscape , Societal challenges
Copyrigths: Reserva de todos los derechos
Minerva. (issn: 0026-4695 ) (eissn: 1573-1871 )
DOI: 10.1007/s11024-015-9271-8
Springer Verlag (Germany)
Publisher version: http://doi.org/10.1007/s11024-015-9271-8
Project ID:
We thank Tommaso Ciarli, Jochen Glaser, Jordi Molas-Gallart, Richard Wooley, and two anonymous referees for their insightful comments and suggestions. We acknowledge support from the UK Economic and Social Research Council ...[+]
Type: Artículo


Agarwal, Pankaj, and David B Searls. 2009. Can literature analysis identify innovation drivers in drug discovery? Nature Reviews. Drug Discovery 8. Nature Publishing Group: 865–78. doi: 10.1038/nrd2973 .

Altman, Edward I., and Anthony Saunders. 1998. Credit risk measurement: Developments over the last 20 years. Journal of Banking and Finance 21: 1721–1742.

Arnold, Erik. 2004. Evaluating research and innovation policy: A systems world needs systems evaluations. Research Evaluation 13: 3–17. [+]
Agarwal, Pankaj, and David B Searls. 2009. Can literature analysis identify innovation drivers in drug discovery? Nature Reviews. Drug Discovery 8. Nature Publishing Group: 865–78. doi: 10.1038/nrd2973 .

Altman, Edward I., and Anthony Saunders. 1998. Credit risk measurement: Developments over the last 20 years. Journal of Banking and Finance 21: 1721–1742.

Arnold, Erik. 2004. Evaluating research and innovation policy: A systems world needs systems evaluations. Research Evaluation 13: 3–17.

Awerbuch, Shimon. 2006. Portfolio-Based Electricity Generation Planning: Policy Implications For Renewables And Energy Security. Mitigation and Adaptation Strategies for Global Change 11: 693–710. doi: 10.1007/s11027-006-4754-4 .

Bazilian, Morgan, and Fabien Roque (eds.). 2008. No Analytical Methods for Energy Diversity and Security: Portfolio Optimization in the Energy Sector: A Tribute to the Work of Dr. Shimon Awerbuch. Amsterdam: Elsevier.

Bernard Cohen, I. 1993. Analogy, Homology, and Metaphor in the Interactions between the Natural Sciences and the Social Sciences, Especially Economics. In Non-natural Social Science: Reflecting on the Enterprise of More Heat than Light, ed. Neil de Marchi, 7–44. Durham: Duke University Press.

Boaz, Annette, Siobhan Fitzpatrick, and Ben Shaw. 2008. Assessing the impact of research on policy: A review of the literature for a project on bridging research and policy through outcome evaluation Final report with references and appendices, February 2008.

Boyack, Kevin W., and Paul Jordan. 2011. Metrics associated with NIH funding: A high-level view. JAMIA 18: 423–431. doi: 10.1136/amiajnl-2011-000213 .

Bozeman, Barry, and Juan Rogers. 2001. Strategic management of government-sponsored R&D portfolios. Environment and Planning C: Government and Policy 19: 413–442. doi: 10.1068/c1v .

Bozeman, Barry, and Daniel Sarewitz. 2005. Public values and public failure in US science policy. Science and Public Policy 32(2): 119–136.

Bozeman, Barry, and Daniel Sarewitz. 2011. Public Value Mapping and Science Policy Evaluation. Minerva 49(1): 1–23. doi: 10.1007/s11024-011-9161-7 .

Brooks, Harvey. 1978. The Problem of Research Priorities. Daedelus 107: 171–190.

Buxton, Martin, Leonie Sundmacher, Jorge Mestre-Ferrandiz, Liz Allen, Nick Black, David Cox, Helen Munn, Briony Rayfield, Eddy Nason, and Jon Sussex. 2008. Medical Research: What's it worth? Estimating the economic benefits from medical research in the UK. London: Health Economics Research Group, Office of Health Economics, RAND Europe.

Calvert, Jane. 2006. What’s Special about Basic Research? Science, Technology & Human Values 31: 199–220. doi: 10.1177/0162243905283642 .

Chalmers, Iain, M.B. Bracken, and Ben Djulbegovic. 2014. How to increase value and reduce waste when research priorities are set. The Lancet: 7–16.

Chien, Chen–Fu. 2002. A portfolio–evaluation framework for selecting R&D projects. R&D Management 32: 359–368. doi: 10.1111/1467-9310.00266 .

Cozzens, Susan. 1997. The knowledge pool: Measurement challenges in evaluating fundamental research programs. Evaluation and Program Planning 20: 77–89. doi: 10.1016/S0149-7189(96)00038-9 .

Cozzens, Susan, and Michelle Snoeck. 2010. Knowledge to Policy Contributing to the Measurement of Social, Health, and Environmental Benefits. Paper prepared for the Workshop on the Science of Science Measurement: 1–39.

Dasgupta, Partha, and Eric Maskin. 2012. The Simple Economics of Research Portfolios. The Economic Journal 97: 581–595.

Devinney, Timothy M., and David W. Stewart. 1988. Rethinking the Product Portfolio: A Generalized Investment Model. Management Science 34: 1080–1095. doi: 10.1287/mnsc.34.9.1080 .

Dietz, James S., and Juan D. Rogers. 2012. Meanings and policy implications of “transformative research”: Frontiers, hot science, evolution, and investment risk. Minerva 50(1): 21–44. doi: 10.1007/s11024-012-9190-x .

Dolby, Kevin, Jimmy Whitworth, Marta Tufet, Suzi Morris, Jessica Burnett, Lily Ickowitz-Seidler, Annie Sanderson, Dave Carr, and Jo Scott. 2012. Malaria 1990–2009. London: Wellcome Trust.

Eikenberry, Angela M., and Jodie Drapal Kluver. 2004. The Marketization of the Nonprofit Sector: Civil Society at Risk. Public Administration Review 64: 132–140.

Ely, Adrian, Patrick Van Zwanenberg, and Andrew Stirling. 2014. Broadening out and opening up technology assessment: Approaches to enhance international development, co-ordination and democratisation. Research Policy 43: 505–518. doi: 10.1016/j.respol.2013.09.004 .

Ernst, Holger. 1998. Patent portfolios for strategic R&D planning. Journal of Engineering and Technology Management 15: 279–308. doi: 10.1016/S0923-4748(98)00018-6 .

European Commission. 2005. Impact assessment and ex ante evaluation. Brussels.

Evans, James A., Jae-Mahn Shim, and John P. Ioannidis. 2014. Attention to local health burden and the global disparity of health research. PloS One 9: e90147. doi: 10.1371/journal.pone.0090147 .

Feller, Irwin. 2012. Performance measures as forms of evidence for science and technology policy decisions. The Journal of Technology Transfer 38: 565–576. doi: 10.1007/s10961-012-9264-9 .

Fernandez, Eduardo, Edy Lopez, Gustavo Mazcorro, Rafael Olmedo, and Carlos Coello Coello. 2013. Application of the non-outranked sorting genetic algorithm to public project portfolio selection. Information Sciences 228: 131–149. doi: 10.1016/j.ins.2012.11.018 .

Fisher, Erik, Catherine P. Slade, Derrick Anderson, and Barry Bozeman. 2010. The public value of nanotechnology? Scientometrics 85: 29–39. doi: 10.1007/s11192-010-0237-1 .

Foray, Dominique, David C. Mowery, and Richard R. Nelson. 2012. Public R&D and social challenges: What lessons from mission R&D programs? Research Policy 41: 1697–1702. doi: 10.1016/j.respol.2012.07.011 .

Freeman, Christopher. 1991. Innovation, Changes of Techno-Economic Paradigm and Biological Analogies in Economics. Revue économique 42: 211. doi: 10.2307/3502005 .

Frodeman, Robert, and Adam Briggle. 2012. The dedisciplining of peer review. Minerva 50(1): 3–19. doi: 10.1007/s11024-012-9192-8 .

Garfinkel, Michele S., Daniel Sarewitz, and Alan L. Porter. 2006. A societal outcomes map for health research and policy. American Journal of Public Health 96: 441–446. doi: 10.2105/AJPH.2005.063495 .

Geels, Frank W. 2004. From sectoral systems of innovation to socio-technical systems. Research Policy 33: 897–920. doi: 10.1016/j.respol.2004.01.015 .

Georghiou, Luke. 1998. Issues in the Evaluation of Innovation and Technology Policy. Evaluation 4: 37–51. doi: 10.1177/13563899822208374 .

Ghiselin, Michael T. 1978. The Economy of the Body. The American Economic Review 68: 233–237.

Gläser, Jochen. 2012. Jochen Gläser on the possibility of a sociological middle-range theory linking science. TUTS-WP-1-2012. Technical University Technology Studies Working Papers. Berlin.

Golec, Joseph H. 1996. The effects of mutual fund managers’ characteristics on their portfolio performance, risk and fees. Financial Services Review 5: 133–147. doi: 10.1016/S1057-0810(96)90006-2 .

Guthrie, Susan, Benoit Guerin, Helen Wu, Sharif Ismail, and Steven Wooding. 2013. Alternatives to Peer Review in Research Project Funding.

Haak, Laurel L., Will Ferriss, Kevin Wright, Michael E. Pollard, Kirk Barden, Matt A. Probus, Michael Tartakovsky, and Charles J. Hackett. 2012. The electronic Scientific Portfolio Assistant: Integrating scientific knowledge databases to support program impact assessment. Science and Public Policy 39: 464–475. doi: 10.1093/scipol/scs030 .

Hage, Jerald, Gretchen B. Jordan, and Jonathan Mote. 2007. A theory-based innovation systems framework for evaluating diverse portfolios of research, part two: Macro indicators and policy interventions. Science and Public Policy 34: 731–741. doi: 10.3152/030234207X265385 .

Hammerstein, Peter, and Edward H. Hagen. 2005. The second wave of evolutionary economics in biology. Trends in Ecology & Evolution 20: 604–609. doi: 10.1016/j.tree.2005.07.012 .

Hanney, Stephen R. 2003. The utilisation of health research in policy-making: Concepts, examples and methods of assessment. Health Research 28: 1–28.

Hanney, Stephen R., Iain Frame, Jonathan Grant, Martin Buxton, Tracey Young, and Grant Lewison. 2005. Using categorisations of citations when assessing the outcomes from health research. Scientometrics 65: 357–379.

Hausmann, Ricardo, César A. Hidalgo, Sebastián Bustos, Michele Coscia, Alexander Simoes, and Muhammed A. Yildirim. 2013. The atlas of economic complexity: Mapping paths to prosperity. Cambridge: Massachusetts Institute of Technology and Centre for International Development, Harvard University.

Hicks, Diana. 2014. “What are grand challenges?” The selected works of Diana Hicks. http://works.bepress.com/diana_hicks/38 . (unpublished).

Holbrook, J. Britt, and Robert Frodeman. 2011. Peer review and the ex ante assessment of societal impacts. Research Evaluation 20: 239–246. doi: 10.3152/095820211X12941371876788 .

Ioannidis, John P. 2011. Fund people not projects. Nature 477: 529–531.

Ioannidis, John P. 2014. How to Make More Published Research True. PLoS Medicine 11: e1001747. doi: 10.1371/journal.pmed.1001747 .

Ismail, Sharif, Jan Tiessen, and Steven Wooding. 2010. Strengthening Research Portfolio Evaluation at the Medical Research Council.

Jordan, Gretchen B., Jerald Hage, and Jonathon Mote. 2008. A theories-based systemic framework for evaluating diverse portfolios of scientific work, part 1: Micro and meso indicators. New Directions for Evaluation 2008: 7–24.

Kay, Luciano, Nils Newman, Jan Youtie, Alan L. Porter, and Ismael Rafols. 2014. Patent Overlay Mapping: Visualizing Technological Distance. Journal of the American Society for Information Science and Technology 65: 2432–2443. doi: 10.1002/asi.23146 .

Kuehn, Bridget M. 2012. US Reviews High-Risk Research Portfolio. JAMA 307: 1682.

Kuhn, Thomas S. 1979. Metaphor in science. In Metaphor and Thought, ed. A. Ortony. Cambridge: Cambridge University Press.

Lakoff, George, and Mark Johnson. 1980. Metaphors we live by. Chicago: University of Chicago Press.

Largent, Mark A., and Julia Lane. 2012. STAR METRICS and the Science of Science Policy. Review of Policy Research 29: 431–438. doi: 10.1111/j.1541-1338.2012.00567.x .

Laudel, Grit, and Jochen Gläser. 2014. Beyond breakthrough research: Epistemic properties of research and their consequences for research funding. Research Policy 43: 1204–1216. doi: 10.1016/j.respol.2014.02.006 .

Liggins, Charlene, Lisa Pryor, and Marie A. Bernard. 2010. Challenges and Opportunities in Advancing Models of Care for Older Adults: An Assessment of the National Institute on Aging Research Portfolio. Journal of the American Geriatrics Society 58: 2345–2349. doi: 10.1111/j.1532-5415.2010.03157.x .

Linton, Jonathan D., Steven T. Walsh, and Joseph Morabito. 2002. Analysis, ranking and selection of R&D projects in a portfolio. R&D Management 32: 139–148. doi: 10.1111/1467-9310.00246 .

Luo, Lieh-Ming. 2011. Optimal diversification for R&D project portfolios. Scientometrics 91: 219–229. doi: 10.1007/s11192-011-0537-0 .

Marburger, John. 2005. Presentation to the Annual Meeting of the Consortium of Social Science Associations (Washington, D.C.).

Markowitz, Harry. 1952. Portfolio Selection. Journal of Finance 7: 77–91.

Marres, Noortje, and Esther Weltevrede. 2013. SCRAPING THE SOCIAL? Journal of Cultural Economy 6:313–335. doi: 10.1080/17530350.2013.772070 .

Martin, Ben R. 2011. The Research Excellence Framework and the “impact agenda”: Are we creating a Frankenstein monster? Research Evaluation 20: 247–254. doi: 10.3152/095820211X13118583635693 .

McGeary, Michael, and Philip M. Smith. 1996. The R&D portfolio: A concept for allocating science and technology funds. Science 274: 1484–1485.

Meador, Kimford J., Jacqueline French, David W. Loring, and Page B. Pennell. 2011. Disparities in NIH funding for epilepsy research. Neurology 77: 1305–1307.

Mirowski, Philip. 1991. More Heat than Light: Economics as Social Physics, Physics as Nature’s Economics. Cambridge: Cambridge University Press.

Molas-Gallart, Jordi, and Puay Tang. 2011. Tracing “productive interactions” to identify social impacts: An example from the social sciences. Research Evaluation 20: 219–226. doi: 10.3152/095820211X12941371876706 .

Moravcsik, Michael J. 1984. Life in a multidimensional world. Scientometrics 6: 75–85. doi: 10.1007/BF02021280 .

Moravcsik, Michael J. 1988. The limits of science and the scientific method. Research Policy 17: 293–299.

Mowery, David C. 2012. Defense-related R&D as a model for “Grand Challenges” technology policies. Research Policy 41: 1703–1715. doi: 10.1016/j.respol.2012.03.027 .

National Research Council. 2005. A prospective evaluation of applied energy research and development at DOE (Phase One). Washington.

National Research Council. 2012. A Review of NASA Human Research Program’s Scientific Merit Processes: Letter Report. Washington: The National Academies Press.

National Science Board. 2001. Federal Research Resources: A Process for Setting Priorities. National Science Foundation.

Nicholson, Joshua M., and John P. Ioannidis. 2012. Research grants: Conform and be funded. Nature 492: 34–36.

Perlitz, Manfred, Thorsten Peske, and Randolf Schrank. 1999. Real options valuation: The new frontier in R&D project evaluation? R&D Management 29: 255–270. doi: 10.1111/1467-9310.00135 .

Porter, Theodore M. 1995. Trust in numbers: The pursuit of objectivity in science and public life. Princeton: Princeton University Press.

Portfolio Review Group. 2014. Report of the Portfolio Review Group: 2012-2013 University of California Systemwide Research Portfolio: Cycle 1 Programs Findings and Recommendations. University of California.

Rafols, Ismael, Alan L. Porter, and Loet Leydesdorff. 2010. Science overlay maps: A new tool for research policy and library management. Journal of the American Society for Information Science and Technology 61: 1871–1887. doi: 10.1002/asi.21368 .

Rafols, Ismael, Loet Leydesdorff, Alice O’Hare, Paul Nightingale, and Andy Stirling. 2012. How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management. Research Policy 41: 1262–1282. doi: 10.1016/j.respol.2012.03.015 .

Reid, W.V., D. Chen, L. Goldfarb, H. Hackmann, Y.T. Lee, K. Mokhele, E. Ostrom, K. Raivio, H.J. Schellnhuber, and A. Whyte. 2010. Earth system science for global sustainability: Grand challenges. Science 330: 916–917.

Robertson, G. Philip, Vivien G. Allen, George Boody, Emery R. Boose, Nancy G. Creamer, E. Laurie, James R. Gosz, et al. 2008. Long-term Agricultural Research: A research, education, and extension imperative. BioScience 58: 640–645.

Røttingen, John-Arne, Sadie Regmi, Mari Eide, Alison J. Young, Roderik F. Viergever, Christine Ardal, Javier Guzman, Danny Edwards, Stephen Matlin, and Robert F. Terry. 2013. Mapping of available health research and development data: What’s there, what’s missing, and what role is there for a global observatory? Lancet 382: 1286–1307. doi: 10.1016/S0140-6736(13)61046-6 .

Ruegg, Rosalie T. 2007. Quantitative portfolio evaluation of US federal research and development programs. Science and Public Policy 34: 723–730. doi: 10.3152/030234207X259021 .

Salter, Ammon J., and Ben R. Martin. 2001. The economic benefits of publicly funded basic research: A critical review. Research Policy 30: 509–532. doi: 10.1016/S0048-7333(00)00091-3 .

Sarewitz, Daniel. 1996. Frontiers of Illusion: Science, Technology and the Politics of Progress. Philadelphia: Temple University Press.

Sarewitz, Daniel, and Roger A. Pielke Jr. 2007. The neglected heart of science policy: Reconciling supply of and demand for science. Environmental Science & Policy 10: 5–16. doi: 10.1016/j.envsci.2006.10.001 .

Schwenk, Charles R. 1988. The Cognitive Perspective on Strategic Decision Making. Journal of Management Studies 25: 41–55. doi: 10.1111/j.1467-6486.1988.tb00021.x .

Scientific Management Review Board. 2013. Draft Report on Approaches to Assess the Value of Biomedical Research Supported by NIH. National Institutes of Health.

Skupin, André, Joseph R. Biberstine, and Katy Börner. 2013. Visualizing the topical structure of the medical sciences: A self-organizing map approach. PloS One 8: e58779. doi: 10.1371/journal.pone.0058779 .

Smith, Richard. 1988. Peering into the bowels of the MRC. I: Setting priorities. British Medical Journal (Clinical research ed.) 296: 484–488.

Smith, Keith. 2000. Innovation as a Systemic Phenomenon: Rethinking the Role of Policy. Enterprise and Innovation Management Studies 1: 73–102. doi: 10.1080/146324400363536 .

Snellen, Ignatius Th.M. 1983. Social Merit as a Criterion of Scientific Choice: Its Application in Dutch Science Policy. Minerva 21: 16–36.

Souder, William E., and Tomislav Mandakovic. 1986. R&D Project Selection Models. Research Management 29: 36–42.

Spaapen, Jack, and Leonie van Drooge. 2011. Introducing “productive interactions” in social impact assessment. Research Evaluation 20: 211–218. doi: 10.3152/095820211X12941371876742 .

Sponberg, Adrienne F. 2005. Streamlining the federal water research portfolio. BioScience 55.

Srivastava, Christina Viola, Nathaniel Deshmukh Towery, and Brian Zuckerman. 2007. Challenges and opportunities for research portfolio analysis, management, and evaluation. Research Evaluation 16: 152–156. doi: 10.3152/095820207X236385 .

Stilgoe, Jack. 2014. Against excellence. The Guardian, December 19. http://www.theguardian.com/science/political-science/2014/dec/19/against-excellence .

Stirling, Andy. 2007. A general framework for analysing diversity in science, technology and society. Journal of the Royal Society Interface 4: 707–719. doi: 10.1098/rsif.2007.0213 .

Stirling, Andy, and Ian Scoones. 2009. From Risk Assessment to Knowledge Mapping: Science, Precaution, and Participation in Disease Ecology. Ecology and Society 14: 14.

Stummer, Christian, and Kurt Heidenberger. 2003. Interactive R&D Portfolio Analysis With Project Interdependencies and Time Profiles of Multiple Objectives. IEEE Transactions on Engineering Management 50(2): 175–183.

Swedish Presidency of the European Union. 2009. The Lund Declaration. European Union.

Van Bekkum, Sjoerd, Enrico Pennings, and Han Smit. 2009. A real options perspective on R&D portfolio diversification. Research Policy 38: 1150–1158.

Vonortas, Nicholas S., and Chintal A. Desai. 2007. “Real options” framework to assess public research investments. Science and Public Policy 34: 699–708. doi: 10.3152/030234207X259012 .

Waltman, Ludo, and Nees Jan van Eck. 2012. A new methodology for constructing a publication-level classification system of science. Journal of the American Society for Information Science and Technology 63: 2378–2392.

Waltman, Ludo, Nees Jan Van Eck, and Ed C. M. Noyons. 2009. A unified approach to mapping and clustering of bibliometric networks: 1–11.

Weinberg, Alvin M. 1963. The Criteria for Scientific Choice. Minerva 1: 159–171.

Woolf, Steven H. 2008. The meaning of translational research and why it matters. JAMA 299: 211–213. doi: 10.1001/jama.2007.26 .

Wulf, William A. 1998. Balancing the research portfolio. Science (New York, N.Y.) 281: 1803.


This item appears in the following Collection(s)

Show full item record