Designing a Framework for Evaluating the Scientific Productions

Ali Norouz, Parastoo Parsaei‑ Mohammadi, Ehsan Geraei, Firoozeh Zare‑Farashbandi

Abstract


Background: Evaluation of scientific productions to attract, maintain, and promote faculty members is one of the necessary processes of academic societies and is of interest to policy makers in the field of higher education. This study aimed to provide a uniform and native framework for evaluating the scientific productions of researchers in the fields of science and health in Iran. Methods: The current research used the single‑stage fuzzy Delphi technique and AHP. The research community comprised 50 top scientific experts and researchers from the country. The data collection tool was a researcher‑made checklist obtained from the review of literature and laws and regulations of Iranian universities and research institutions. Results: Data analysis led to the identification, prioritization and weighting of eleven criteria and 124 items for the intended framework, which are, respectively, author’s authority (scientific leaders) (15 items); database used in calculating the index (3 items); Innovation and technological impact (18 items); Gaining rank in national and international festivals related to the specialized field (6 items); Citations (7 items); subject area (3 items); level of cooperation (15 items); Types of scientific productions (54 items); scientific age (3 items); Evaluating, refereeing and monitoring of research, technology and innovation activities (2 items) Author`s role and position (1 item). Conclusion: The final framework obtained for evaluating Iranian researchers has 11 criteria and 124 items that can be used to compile an author‑centered and native scientometrics index that leads to the same evaluation of health and science researchers.

Keywords


Author‑oriented indicator; researcher evaluation; scientific production evaluation; scientometrics indicators

Full Text:

PDF

References


Osareh F, Parsaei Mohammadi P, Farajpahlou A, Rahimi FA. A comparative study of criteria and indicators of local, regional, and National University Ranking Systems. J Sci Res 2023;12:54 67.

Zakiani S, Ghaffari S, Mohseni M. Evaluating the scientific outputs of information management researchers in WoS. Casp J Sci 2019;6:60 8.

Noroozi Chakoli A, Rezaei M. Scientometrics, international special indexes, scientific productivity evaluation. Iran J Inf Process Manag 2014;30:3 39.

Alonso S, Cabrerizo FJ, Herrera Viedma E, Herrera F. hg index: A new index to characterize the scientific output of researchers based on the h and g indices. Scientometrics 2010;82:391 400.

Salman M, Ahmed MM, Afzal MT. Assessment of author ranking indices based on multi authorship. Scientometrics 2021;126:4153 72.

Rawat S, Meena S. Publish or perish: Where are we heading? J Res Med Sci 2014;19:87 9.

Morales E, McKiernan EC, Niles MT, Schimanski L, Alperin JP. How faculty define quality, prestige, and impact of academic journals. PLOS One 2021;16:e0257340.

Aksnes DW, Langfeldt L, Wouters P. Citations, citation indicators, and research quality: An overview of basic concepts and theories. Sage Open 2019;9:2158244019829575.

Hicks D, Wouters P. The Leiden Manifesto for research metrics. Use these ten principles to guide research evaluation. Nature 2015;520:9 11.

Wouters P, Thelwall M, Kousha K, Waltman L, de Rijcke S, Rushforth A, et al. The metric tide: Literature review, Supplementary Report I to the Independent Review of the Role of Metrics in Research Assessment and Management. London: HEFCE; 2015.

Portfolio A. Guidelines for Good Evaluation Practice with the Wildgaard LE. Guidelines for Good Evaluation Practice with the ACUMEN portfolio. Leiden University, 2014. p. 116.

Watermeyer R, Hedgecoe A. Selling ‘impact’: Peer reviewer projections of what is needed and what counts in REF impact case studies. A retrospective analysis. J Educ Policy 2016;31:651 65.

Supreme Council of Cultural Revolution of Iran. Employment regulations for faculty members of universities and higher education, research and technology institutions. Iran: 2011. (In Persian)

Supreme Council of Cultural Revolution of Iran. Regulations for promotion of academic staff members of governmental and non governmental institutions of higher education, research and technology, 7453 (2016). (In Persian)

Raheel M, Ayaz S, Afzal MT. Evaluation of h index, its variants and extensions based on publication age and citation intensity in civil engineering. Scientometrics 2018;114:1107 27.

Ameer M, Afzal MT. Evaluation of h index and its qualitative and quantitative variants in Neuroscience. Scientometrics 2019;121:653 73.

Shamsi M, Nourmohammadi H. Identify the most important indicators and models in the evaluation of science and technology in knowledge based companies in Iran. Sci Res J 2018;4:1 16.

Habibi A, Sarafrazi A, Izadyar S. Delphi technique theoretical framework in qualitative research. Int J Eng Sci 2014;3:8 13.

Rahdary A, Nasr M. Challenges of think tanks in Iran. JMDP 2017;30:23 54.

Hirsch JE. An index to quantify an individual’s scientific research output. Proc Natl Acad Sci 2005;102:16569 72.

Sidiropoulos A, Katsaros D, Manolopoulos Y. Generalized Hirsch h index for disclosing latent facts in citation networks. Scientometrics 2007;72:253 80.

Jin B, Liang L, Rousseau R, Egghe L. The R and AR indices: Complementing the h index. Chin Sci Bull 2007;52:855 63.

Belikov AV, Belikov VV. A citation based, author and age normalized, logarithmic index for evaluation of individual researchers independently of publication counts. F1000Research 2015;4:884.

Gunthe SS, Gettu R. A new index for assessing faculty research performance in higher educational institutions of emerging economies such as India. Scientometrics. 2022;127:4959 76.

Kaur J, Radicchi F, Menczer F. Universality of scholarly impact metrics. J Informetr 2013;7:924 32.

Hirsch JE. hα: An index to quantify an individual’s scientific leadership. Scientometrics 2019;118:673 86.

Egghe L, Rousseau R. An h index weighted by citation impact. Inf Process Manage 2008;44:770 80.

Fu HZ, Ho YS. Top cited articles in adsorption research using Y index. Res Eval 2014;23:12 20.

Bihari A, Tripathi S. EM index: A new measure to evaluate the scientific impact of scientists. Scientometrics 2017;112:659 77.

Iranian Islamic Council Research Center. Executive regulations of the law to support knowledge based companies and institutions and c ommercialize innovations and inventions, 141602 (2012). (In Persian)

van Eck NJ, Waltman L. Generalizing the h and g indices. J Informetr 2008;2:263 71.

Mohammed S, Morgan A, Nyantakyi E. On the influence of uncited publications on a researcher’s h index. Scientometrics 2020;122:1791 9.

Kosmulski M. A new Hirsch type index saves time and works equally well as the original h index. International Society for Scientometrics and Informetrics (ISSI) 2005 2006;2:4 6.

Galam S. Tailor based allocations for multiple authorship: A fractional gh index. Scientometrics 2011;89:365 79.

Xu F, Liu WB, Mingers J. New journal classification methods based on the global h index. Inf Process Manag 2015;51:50 61.

Bianchi F, Grimaldo F, Squazzoni F. The F3 index. Valuing reviewers for scholarly journals. J Inf 2019;13:78 86.

Bornmann L. Do we need the E index in addition to the h index and its variants? J Am Soc Inf Sci Technol 2011;62:1433 4.

Batista PD, Campiteli MG, Kinouchi O. Is it possible to compare researchers with different scientific interests? Scientometrics 2006;68:179 89.