Designing an Impact‑Oriented Model of Research and Technology Evaluation: An Experience of I.R.Iran

Katayoun Falahat, Monir Baradaran Eftekhari, Shaghayegh Haghjooy Javanmard, Elham Ghalenoee, Hanieh Shakeri

Abstract


Background: Research impact assessment is already being institutionalized in health research and innovation systems. In developing countries, there are many different research assessment models which have focused more on research output in academic levels and less on impact. Objective: The aim of this study is designing an Iranian impact‑oriented model of research and technology evaluation. Method: This is a mixed study. In the quantitative part, by reviewing the literature, a list of research impact indicators that existed were gathered, reviewed, and scored by participants on importance, relevance, and measurability via a 5‑point Likert scale. All indicators with a mean score equal to or greater than 3.5 entered the qualitative part, which were discussed in depth by engaging key stakeholders regarding their validity and feasibility through focus groups, interviews, and expert panels. Results: The Iranian research impact evaluation model was developed with four main pillars (including input and process, output, outcome, and impact), four areas (stewardship, advancing knowledge and translation, technology, and impact), and 30 indicators through key stakeholders participation in the Iranian health research system. Conclusions: This model has been introduced as the first model designed to evaluate the impact of health research and can be one of the most important tools for allocating limited funding resources while maximizing the desired impact of research in the community.

Keywords


Evaluation; health; impact assessment; research; technology

Full Text:

PDF

References


Holmes R. The THES university rankings: Are they really world class? Asian J Univ Educ 2006;2:1 14.

Dominique AP, Lorente LM, Zorita JC, Casado ES. Model for estimating Academic Ranking of World Universities (Shanghai Ranking) scores. Rev Esp Doc Cient 2018;41:4.

Dastani M, Panahi S, Sattari M, Webometrics analysis of Iranian Universities about medical sciences’ websites between September 2016 and March 2017. Acta Inform Med 2019;3:7 12.

Matsuo T, Sirilli G, Gault F. Frascati Manual. Proposed Standard Practice for Surveys on Research and Experimental Development. Frascati Manual 2002.

Gault F. The Oslo Manual, in Handbook of Innovation Indicators and Measurement. Edward Elgar Publishing; 2013. p. 41 59.

Kunttu L, Dan S, Kalliomäki H, Kuusisto J. Assessing Evolving Innovation Space in Oslo Manuals. The International Society for Professional Innovation Management (ISPIM). ISPIM Conference Proceedings, 2021. p. 1 12.

Jianghua Q. Translation as Knowledge Production, Management, and Utilization: Towards Transknowletology. Contemporary Foreign Language Studies 2022;457:60 71. doi: 10.3969/j.issn. 1674 8921.2022.01.004.

Bornmann L. Measuring the societal impact of research: Research is less and less assessed on scientific impact alone we should aim to quantify the increasingly important contributions of science to society. EMBO Rep 2012;13:673 6.

Greenhalgh T, Raftery J, Hanney S, Glover M. Research impact: A narrative review. BMC Med 2016;14:1 16.

Adam P, Ovseiko PV, Grant J, Graham KE, Boukhris OF, Dowd AM, et al. ISRIA statement: Ten point guidelines for an effective process of research impact assessment. Health Res Policy Syst 2018;16:1 6.

Penfield T, Baker MJ, Scoble R, Wykes MC. Assessment, evaluations, and definitions of research impact: A review. Res Eval 2014;23:21 32.

Peykari N, Djalalinia S, Owlia P, Habibi E, Falahat K, Ghanei M, et al. Health research system evaluation in IR of Iran. Arch Iran Med 2012;15:394 9.

Akhondzadeh S, Ebadifar A, Baradaran Eftekhari M, Falahat K. Medical science and research in Iran. Arch Iran Med 2017;20:665 72.

Eftekhari MB, Sobhani Z, Eltemasi M, Ghalenoee E, Falahat K, Habibi E, et al. Research ranking of Iranian universities of medical sciences based on international indicators: An experience from I.R of Iran. Arch Iran Med 2017;20:673 9.

Harvey N, Holmes CA. Nominal group technique: An effective method for obtaining group consensus. Int J Nurs Pract 2012;18:188 94.

Lawshe CH. A quantitative approach to content validity. Pers Psychol 1975;28:563 75.

Madadizadeh F, Bahariniya S. Tutorial on how to calculating content validity of scales in medical research. Perioper Care Oper Room Manag 2023;31:100315. doi: S2405603023000109.

Hariri FZ, Moghaddam Banaem L, Siah Bazi S, Saki Malehi A, Montazeri A. The Iranian version of the Premenstrual Symptoms Screening Tool (PSST): A validation study. Arch Womens Ment Health 2013;16:531 7.

Torrance, H., The research excellence framework in the United Kingdom: Processes, consequences, and incentives to engage. Qual Inq 2020;26:771 9.

Graham KE, Chorzempa HL, Valentine PA, Magnan J. Evaluating health research impact: Development and implementation of the Alberta Innovates–Health solutions impact framework. Res Eval 2012;21:354 67.

Buykx P, Humphreys J, Wakerman J, Perkins D, Lyle D, McGrail M, et al. ‘Making evidence count’: A framework to monitor the impact of health services research. Aust J rural Health 2012;20:51 8.

Cruz Rivera S, Kyte DG, Aiyegbusi OL, Keeley TJ, Calvert MJ Assessing the impact of healthcare research: A systematic review of methodological frameworks. PLoS Med 2017;14:e1002370. doi: 10.1371/journal.pmed.1002370.

Weiss AP. Measuring the impact of medical research: Moving from outputs to outcomes. Am J Psychiatry 2007;164:206 14.

Stern N. Building on success and learning from experience: An independent review of the Research Excellence Framework. 2016. Available from: https://assets.publishing.service.gov.uk/ media/5a803df4e5274a2e8ab4f03d/ind169refsternreview.pdf. [Last accessed on 2024 Sep 07]. 25. Eftekhari MB, Ebadifar A, Ghalenoee E, Falahat K. Peer review research evaluation in Iran: Strengths, weaknesses and suggestions. Iran J Pub Health 2021;50:1260 5.

Milat AJ, Bauman AE, Redman S. A narrative review of research impact assessment models and methods. Health Res Policy Syst 2015;13. doi: 10.1186/s12961 015 0003 1.

Hansen J, Azzopardi Muscat N, Keskimäki I, Lindahl AK, Pfaff H, Wismar M, et al. Measuring and improving the societal impact of health care research. Eurohealth 2013;19:32 5.