J Korean Med Sci.  2021 Mar;36(11):e74. 10.3346/jkms.2021.36.e74.

Article-Level Metrics

• Gasparyan AY
• Yessirkepov M
• Voronov AA
• Maksaev AA
• Kitas GD

• KMID: 2514050
• DOI: http://doi.org/10.3346/jkms.2021.36.e74

Abstract

In the era of digitization and Open Access, article-level metrics are increasingly employed to distinguish influential research works and adjust research management strategies. Tagging individual articles with digital object identifiers allows exposing them to numerous channels of scholarly communication and quantifying related activities. The aim of this article was to overview currently available article-level metrics and highlight their advantages and limitations. Article views and downloads, citations, and social media metrics are increasingly employed by publishers to move away from the dominance and inappropriate use of journal metrics. Quantitative article metrics are complementary to one another and often require qualitative expert evaluations. Expert evaluations may help to avoid manipulations with indiscriminate social media activities that artificially boost altmetrics. Values of article metrics should be interpreted in view of confounders such as patterns of citation and social media activities across countries and academic disciplines.

Keyword

Access to Information; Open Access Publishing; Bibliometrics; Bibliography as Topic; Periodicals as Topic

Reference

1. Tregoning J. How will you judge me if not by impact factor? Nature. 2018; 558(7710):345. PMID: 29921857.

2. Hatch A, Curry S. Changing how we evaluate research is difficult, but not impossible. Elife. 2020; 9:e58654. PMID: 32782065.

3. Bornmann L, Haunschild R. Alternative article-level metrics: the use of alternative metrics in research evaluation. EMBO Rep. 2018; 19(12):e47260. PMID: 30425125.

4. Gasparyan AY, Yessirkepov M, Voronov AA, Koroleva AM, Kitas GD. Comprehensive approach to open access publishing: platforms and tools. J Korean Med Sci. 2019; 34(27):e184. PMID: 31293109.

5. Citrome L. Moving forward with article level metrics: introducing altmetrics. Int J Clin Pract. 2015; 69(8):811. PMID: 26223556.

6. Chandrashekhar Y, Shaw L. Journal editors and altmetrics: moth to the flame? JACC Cardiovasc Imaging. 2019; 12(9):1899–1902. PMID: 31488257.

7. Chavda J, Patel A. Measuring research impact: bibliometrics, social media, altmetrics, and the BJGP. Br J Gen Pract. 2016; 66(642):e59–e61. PMID: 26719483.

8. Hammarfelt B. Using altmetrics for assessing research impact in the humanities. Scientometrics. 2014; 101(2):1419–1430.

9. Dardas LA, Woodward A, Scott J, Xu H, Sawair FA. Measuring the social impact of nursing research: an insight into altmetrics. J Adv Nurs. 2019; 75(7):1394–1405. PMID: 30507052.

10. Gorraiz J, Melero-Fuentes D, Gumpenberger C, Valderrama-Zurián JC. Availability of digital object identifiers (DOIs) in Web of Science and Scopus. J Informetrics. 2016; 10(1):98–109.

11. Erfanmanesh M. Highly-alted articles in library and information science. Webology. 2017; 14(2):66–77.

12. Salahshoori F, Abedini Z. Investigating the social media presence of articles in altmetrics field indexed in Scopus database: an altmetrics study. Libr Philos Pract. 2019; 2779.

13. Boudry C, Chartron G. Availability of digital object identifiers in publications archived by PubMed. Scientometrics. 2017; 110(3):1453–1469.

14. Butler JS, Kaye ID, Sebastian AS, Wagner SC, Morrissey PB, Schroeder GD, et al. The evolution of current research impact metrics: from bibliometrics to altmetrics? Clin Spine Surg. 2017; 30(5):226–228. PMID: 28338492.

15. Knowlton SE, Paganoni S, Niehaus W, Verduzco-Gutierrez M, Sharma R, Iaccarino MA, et al. Measuring the impact of research using conventional and alternative metrics. Am J Phys Med Rehabil. 2019; 98(4):331–338. PMID: 30300231.

16. Haustein S, Peters I, Bar-Ilan J, Priem J, Shema H, Terliesner J. Coverage and adoption of altmetrics sources in the bibliometric community. Scientometrics. 2014; 101(2):1145–1163.

17. Eldakar MAM. Who reads international Egyptian academic articles? An altmetrics analysis of Mendeley readership categories. Scientometrics. 2019; 121(1):105–135.

18. Kohn K. Effects of Publisher Interface and Google Scholar on HTML and PDF Clicks: Investigating Paths That Inflate Usage. J Acad Librariansh. 2018; 44(6):816–823.

19. Yan KK, Gerstein M. The spread of scientific information: insights from the web usage statistics in PLoS article-level metrics. PLoS One. 2011; 6(5):e19917. PMID: 21603617.

20. Fang Z, Guo X, Yang Y, Yang Z, Li Q, Hu Z, et al. Measuring global research activities using geographic data of scholarly article visits. Electron Libr. 2017; 35(4):822–838.

21. Moliterno DJ. The top papers of 2017: by subsequent citations and online views and downloads. JACC Cardiovasc Interv. 2018; 11(3):325–327. PMID: 29413251.

22. Chen B. Usage pattern comparison of the same scholarly articles between Web of Science (WoS) and Springer. Scientometrics. 2018; 115(1):519–537.

23. Duan Y, Xiong Z. Download patterns of journal papers and their influencing factors. Scientometrics. 2017; 112(3):1761–1775.

24. Davis PM. Public accessibility of biomedical articles from PubMed Central reduces journal readership--retrospective cohort analysis. FASEB J. 2013; 27(7):2536–2541. PMID: 23554455.

25. Wang X, Cui Y, Li Q, Guo X. Social media attention increases article visits: An investigation on article-level referral data of PeerJ . Front Res Metr Anal. 2017; 2:11.

26. Wang X, Fang Z, Guo X. Tracking the digital footprints to scholarly articles from social media. Scientometrics. 2016; 109(2):1365–1376.

27. Amath A, Ambacher K, Leddy JJ, Wood TJ, Ramnanan CJ. Comparing alternative and traditional dissemination metrics in medical education. Med Educ. 2017; 51(9):935–941. PMID: 28719136.

28. Singson M, Thiyagarajan S, Leeladharan M. Relationship between electronic journal downloads and citations in library consortia. Libr Rev. 2016; 65(6/7):429–444.

29. Vaughan L, Tang J, Yang R. Investigating disciplinary differences in the relationships between citations and downloads. Scientometrics. 2017; 111(3):1533–1545.

30. McGillivray B, Astell M. The relationship between usage and citations in an open access mega-journal. Scientometrics. 2019; 121(2):817–838.

31. Fenton JE, O'Connor A, Ullah I, Ahmed I, Shaikh M. Do citation classics in rhinology reflect utility rather than quality? Rhinology. 2005; 43(3):221–224. PMID: 16218517.

32. Gasparyan AY, Yessirkepov M, Voronov AA, Gerasimov AN, Kostyukova EI, Kitas GD. Preserving the integrity of citations and references by all stakeholders of science communication. J Korean Med Sci. 2015; 30(11):1545–1552. PMID: 26538996.

33. Gasparyan AY, Ayvazyan L, Kitas GD. Multidisciplinary bibliographic databases. J Korean Med Sci. 2013; 28(9):1270–1275. PMID: 24015029.

34. Powell AJ, Conlee EM, Chang DG. Three decades of citation classics: the most cited articles in the field of physical medicine and rehabilitation. PM R. 2014; 6(9):828–840. PMID: 25091931.

35. Bohl MA, Turner JD, Little AS, Nakaji P, Ponce FA. Assessing the relevancy of “citation classics” in neurosurgery. Part II: foundational papers in neurosurgery. World Neurosurg. 2017; 104:939–966. PMID: 28438655.

36. Tahamtan I, Bornmann L. What do citation counts measure? An updated review of studies on citations in scientific documents published between 2006 and 2018. Scientometrics. 2019; 121(3):1635–1684.

37. Ahlgren P, Colliander C, Sjögårde P. Exploring the relation between referencing practices and citation impact: a large-scale study based on Web of Science data. J Assoc Inf Sci Technol. 2018; 69(5):728–743.

38. Gates AJ, Ke Q, Varol O, Barabási AL. Nature's reach: narrow work has broad impact. Nature. 2019; 575(7781):32–34. PMID: 31695218.

39. Lazarev VS, Nazarovets SA. Don't dismiss citations to journals not published in English. Nature. 2018; 556(7700):174.

40. Neff M. Quest for publication metrics undermines regional research. Nature. 2018; 554(7691):169.

41. Bornmann L, Haunschild R. Citation score normalized by cited references (CSNCR): the introduction of a new citation impact indicator. J Informetrics. 2016; 10(3):875–887.

42. Bornmann L, Wohlrabe K. Normalisation of citation impact in economics. Scientometrics. 2019; 120(2):841–884.

43. Hicks D, Wouters P, Waltman L, de Rijcke S, Rafols I. Bibliometrics: the Leiden Manifesto for research metrics. Nature. 2015; 520(7548):429–431. PMID: 25903611.

44. Dunaiski M, Geldenhuys J, Visser W. On the interplay between normalisation, bias, and performance of paper impact metrics. J Informetrics. 2019; 13(1):270–290.

45. Hutchins BI, Yuan X, Anderson JM, Santangelo GM. Relative citation ratio (RCR): a new metric that uses citation rates to measure influence at the article level. PLoS Biol. 2016; 14(9):e1002541. PMID: 27599104.

46. Murphy LS, Kraus CK, Lotfipour S, Gottlieb M, Langabeer JR 2nd, Langdorf MI. Measuring scholarly productivity: a primer for junior faculty. Part III: understanding publication metrics. West J Emerg Med. 2018; 19(6):1003–1011. PMID: 30429933.

47. Naik G. The quiet rise of the NIH's hot new metric. Nature. 2016; 539(7628):150. PMID: 27830815.

48. Bornmann L, Haunshild R. Relative citation ratio (RCR): an empirical attempt to study a new field-normalized bibliometric indicator. J Assoc Inf Sci Technol. 2017; 68(4):1064–1067.

49. Purkayastha A, Palmaro E, Falk-Krzesinski HJ, Baas J. Comparison of two article-level, field-independent citation metrics: field-weighted citation impact (FWCI) and relative citation ratio (RCR). J Informetrics. 2019; 13(2):635–642.

50. Waltman L. NIH's new citation metric: a step forward in quantifying scientific impact? Updated 2015. Accessed February 5, 2021. https://www.cwts.nl/blog?article=n-q2u294.

51. Alshareef AM, Alhamid MF, El Saddik A. Toward citation recommender systems considering the article impact in the extended nearby citation network. Peer Peer Netw Appl. 2018; 12(5):1336–1345.

52. Vardell E, Swogger SE. F1000Prime: a faculty of 1000 tool. Med Ref Serv Q. 2014; 33(1):75–84. PMID: 24528266.

53. Akers KG. Electronic resources reviews: F1000prime: expert recommendations of journal articles in biology and medicine. Issues Sci Technol Librariansh. 2018; 90.

54. Du J, Tang X, Wu Y. The effects of research level and article type on the differences between citation metrics and F1000 recommendations. J Assoc Inf Sci Technol. 2016; 67(12):3008–3021.

55. Bornmann L, Haunschild R. Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000Prime data. PLoS One. 2018; 13(5):e0197133. PMID: 29791468.

56. Ortega JL. Altmetrics data providers: a meta-analysis review of the coverage of metrics and publication. Prof Inf. 2020; 29(1):e290107.

57. Haustein S, Costas R, Larivière V. Characterizing social media metrics of scholarly papers: the effect of document properties and collaboration patterns. PLoS One. 2015; 10(3):e0120495. PMID: 25780916.

58. Trueger NS, Thoma B, Hsu CH, Sullivan D, Peters L, Lin M. The altmetric score: a new measure for article-level dissemination and impact. Ann Emerg Med. 2015; 66(5):549–553. PMID: 26004769.

59. Ortega JL. Reliability and accuracy of altmetric providers: a comparison among Altmetric.com, PlumX and Crossref Event Data. Scientometrics. 2018; 116(3):2123–2138.

60. Lindsay JM. PlumX from plum analytics: not just altmetrics. J Electron Resour Med Libr. 2016; 13(1):8–17.

61. Azer SA, Azer S. Top-cited articles in medical professionalism: a bibliometric analysis versus altmetric scores. BMJ Open. 2019; 9(7):e029433.

62. Meschede C, Siebenlist T. Cross-metric compatability and inconsistencies of altmetrics. Scientometrics. 2018; 115(1):283–297.

63. Zahedi Z, Costas R, Wouters P. How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics. 2014; 101(2):1491–1513.

64. Pooladian A, Borrego Á. A longitudinal study of the bookmarking of library and information science literature in Mendeley. J Informetrics. 2016; 10(4):1135–1142.

65. Zahedi Z, Costas R, Wouters P. Mendeley readership as a filtering tool to identify highly cited publications. J Assoc Inf Sci Technol. 2017; 68(10):2511–2521.

66. Haunschild R, Bornmann L. Normalization of Mendeley reader counts for impact assessment. J Informetrics. 2016; 10(1):62–73.

67. Maggio LA, Meyer HS, Artino AR Jr. Beyond citation rates: a real-time impact analysis of health professions education research using altmetrics. Acad Med. 2017; 92(10):1449–1455. PMID: 28817430.

68. Zimba O, Radchenko O, Strilchuk L. Social media for research, education and practice in rheumatology. Rheumatol Int. 2020; 40(2):183–190. PMID: 31863133.

69. Park P, Macy M. The paradox of active users. Big Data Soc. 2015; 2(2):1–4.

70. Ortega JL. The presence of academic journals on Twitter and its relationship with dissemination (tweets) and research impact (citations). Aslib J Inf Manag. 2017; 69(6):674–687.

71. Khan MS, Shahadat A, Khan SU, Ahmed S, Doukky R, Michos ED, et al. The Kardashian index of cardiologists. celebrities or experts? JACC Case Rep. 2020; 2(2):330–332. PMID: 32292918.

72. Jamison AM, Broniatowski DA, Quinn SC. Malicious actors on twitter: a guide for public health researchers. Am J Public Health. 2019; 109(5):688–692. PMID: 30896994.

73. Pagoto S, Waring ME, Xu R. A call for a public health agenda for social media research. J Med Internet Res. 2019; 21(12):e16661. PMID: 31855185.

74. Ahmed S, Gupta L. Social media for medical journals. Cent Asian J Med Hypotheses Ethics. 2020; 1(1):26–32.

75. Ravikumar SS, Khonglam B. Tweets of an article and its citation: an altmetric study of most prolific authors. Libr Philos Pract. 2018; 1745.

76. Haustein S, Peters I, Sugimoto CR, Thelwall M, Larivière V. Tweeting biomedicine: an analysis of tweets and citations in the biomedical literature. J Assoc Inf Sci Technol. 2014; 65(4):656–669.

77. Bornmann L, Haunschild R. How to normalize Twitter counts? A first attempt based on journals in the Twitter index. Scientometrics. 2016; 107(3):1405–1422. PMID: 27239079.

78. Didegah F, Bowman TD, Holmberg K. On the differences between citations and altmetrics: an investigation of factors driving altmetrics versus citations for finnish articles. J Assoc Inf Sci Technol. 2018; 69(6):832–843.

79. Ortega JL. Blogs and news sources coverage in altmetrics data providers: a comparative analysis by country, language, and subject. Scientometrics. 2020; 122(1):555–572.

80. Asyyed Z, McGuire C, Samargandi O, Al-Youha S, Williams JG. The use of Twitter by plastic surgery journals. Plast Reconstr Surg. 2019; 143(5):1092e–1098e.

81. Pineda C, Pérez-Neri I, Sandoval H. Challenges for social media editors in rheumatology journals: an outlook. Clin Rheumatol. 2019; 38(6):1785–1789. PMID: 31093788.

82. Dharnidharka VR. A social media editor for pediatric transplantation. Pediatr Transplant. 2019; 23(1):e13343. PMID: 30635957.

83. Kochanek PM, Kudchadkar SR, Kissoon N. New developments for pediatric critical care medicine in 2019 and beyond. Pediatr Crit Care Med. 2019; 20(4):311.

84. Lopez M, Chan TM, Thoma B, Arora VM, Trueger NS. The social media editor at medical journals: responsibilities, goals, barriers, and facilitators. Acad Med. 2019; 94(5):701–707. PMID: 30334841.