Ann Lab Med.  2025 Jan;45(1):1-11. 10.3343/alm.2024.0258.

Toward High-Quality Real-World Laboratory Data in the Era of Healthcare Big Data

• Kim S
• Min WK

• KMID: 2563850
• DOI: http://doi.org/10.3343/alm.2024.0258

Abstract

With Industry 4.0, big data and artificial intelligence have become paramount in the field of medicine. Electronic health records, the primary source of medical data, are not collected for research purposes but represent real-world data; therefore, they have various constraints. Although structured, laboratory data often contain unstandardized terminology or missing information. The major challenge lies in the lack of standardization of test results in terms of metrology, which complicates comparisons across laboratories. In this review, we delve into the essential components necessary for integrating real-world laboratory data into high-quality big data, including the standardization of terminology, data formats, equations, and the harmonization and standardization of results. Moreover, we address the transference and adjustment of laboratory results, along with the certification for quality of laboratory data. By discussing these critical aspects, we seek to shed light on the challenges and opportunities inherent to utilizing real-world laboratory data within the framework of healthcare big data and artificial intelligence.

Keyword

Artificial intelligence; Big data; Data quality; Harmonization; Laboratory medicine; Real-world data; Standardization

Figure

• 

  Fig. 1 Improving patient outcomes through analytics performed on big data gathered from various sources.

• 

  Fig. 2 ‘Big data-to-big data loop’ of laboratory tests in the Industry 4.0 era.

• 

  Fig. 3 Essential components for building high-quality laboratory big data.

Reference

References

1. Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. 2017; Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol. 2:230–43. DOI: 10.1136/svn-2017-000101. PMID: 29507784. PMCID: PMC5829945.

2. Ehsani-Moghaddam B, Martin K, Queenan JA. 2021; Data quality in healthcare: A report of practical experience with the Canadian Primary Care Sentinel Surveillance Network data. Health Inf Manag. 50:88–92. DOI: 10.1177/1833358319887743. PMID: 31805788.

3. Daniel C, Serre P, Orlova N, Bréant S, Paris N, Griffon N. 2019; Initializing a hospital-wide data quality program. The AP-HP experience. Comput Methods Programs Biomed. 181:104804. DOI: 10.1016/j.cmpb.2018.10.016. PMID: 30497872.

4. Aerts H, Kalra D, Sáez C, Ramírez-Anguita JM, Mayer MA, Garcia-Gomez JM, et al. 2021; Quality of hospital electronic health record (EHR) data based on the International Consortium for Health Outcomes Measurement (ICHOM) in heart failure: pilot data quality assessment study. JMIR Med Inform. 9:e27842. DOI: 10.2196/27842. PMID: 34346902. PMCID: PMC8374665. PMID: b283a765906646d7ae661c0561f87815.

5. Liaw ST, Guo JGN, Ansari S, Jonnagaddala J, Godinho MA, Borelli AJ, et al. 2021; Quality assessment of real-world data repositories across the data life cycle: a literature review. J Am Med Inform Assoc. 28:1591–9. DOI: 10.1093/jamia/ocaa340. PMID: 33496785. PMCID: PMC8475229.

6. Wu J, Wang C, Toh S, Pisa FE, Bauer L. 2020; Use of real-world evidence in regulatory decisions for rare diseases in the United States-current status and future directions. Pharmacoepidemiol Drug Saf. 29:1213–8. DOI: 10.1002/pds.4962. PMID: 32003065.

7. U.S. FDA. 2016. Use of real-world evidence to support regulatory decision-making for medical devices. Guidance for industry and food and drug administration staff. FDA-2016-D-2153. U.S. Department of Health and Human Services Food and Drug Administration;Rockville, MD:

8. Blatter TU, Witte H, Nakas CT, Leichtle AB. 2022; Big data in laboratory medicine-FAIR quality for AI? Diagnostics (Basel). 12:1923. DOI: 10.3390/diagnostics12081923. PMID: 36010273. PMCID: PMC9406962. PMID: 92558140ec7749b48c2041d59d1d8e1d.

9. Ronzio L, Cabitza F, Barbaro A, Banfi G. 2021; Has the flood entered the basement? A systematic literature review about machine learning in laboratory medicine. Diagnostics (Basel). 11:372. DOI: 10.3390/diagnostics11020372. PMID: 33671623. PMCID: PMC7926482. PMID: e6fa6768811a40be82a853f67d4b95c9.

10. Drenkhahn C, Ingenerf J. 2020; The LOINC content model and its limitations of usage in the laboratory domain. Stud Health Technol Inform. 270:437–42. DOI: 10.3233/SHTI200198. PMID: 32570422.

11. Regenstrief Institute Inc. Logical Observation Identifiers Names and Codes (LOINC) version 2.77. https://loinc.org/. Updated on April 2024.

12. Cholan RA, Pappas G, Rehwoldt G, Sills AK, Korte ED, Appleton IK, et al. 2022; Encoding laboratory testing data: case studies of the national implementation of HHS requirements and related standards in five laboratories. J Am Med Inform Assoc. 29:1372–80. DOI: 10.1093/jamia/ocac072. PMID: 35639494. PMCID: PMC9277627.

13. Parr SK, Shotwell MS, Jeffery AD, Lasko TA, Matheny ME. 2018; Automated mapping of laboratory tests to LOINC codes using noisy labels in a national electronic health record system database. J Am Med Inform Assoc. 25:1292–300. DOI: 10.1093/jamia/ocy110. PMID: 30137378. PMCID: PMC7646911.

14. Liu CT, Wang LW, Hsu MH, Wen LL, Lai JS. 2007. A unified approach to adoption of laboratory LOINC in Taiwan. In : Healthcom 2007: Ubiquitous healthcare in aging societies - 2007 9th International Conference on e-Health Networking, Application and Services; p. 144–9. DOI: 10.1109/HEALTH.2007.381620.

15. Stram M, Seheult J, Sinard JH, Campbell WS, Carter AB, de Baca ME, et al. 2020; A survey of LOINC code selection practices among participants of the College of American Pathologists Coagulation (CGL) and Cardiac Markers (CRT) proficiency testing programs. Arch Pathol Lab Med. 144:586–96. DOI: 10.5858/arpa.2019-0276-OA. PMID: 31603714.

16. Lin MC, Vreeman DJ, McDonald CJ, Huff SM. 2010; Correctness of voluntary LOINC mapping for laboratory Tests in three Large Institutions. AMIA Annu Symp Proc. 2010:447–51. PMID: 21347018. PMCID: PMC3041457.

17. McDonald CJ, Baik SH, Zheng Z, Amos L, Luan X, Marsolo K, et al. 2023; Mis-mappings between a producer's quantitative test codes and LOINC codes and an algorithm for correcting them. J Am Med Inform Assoc. 30:301–7. DOI: 10.1093/jamia/ocac215. PMID: 36343113. PMCID: PMC9846663.

18. Bhargava A, Kim T, Quine DB, Hauser RG. 2020; A 20-year evaluation of LOINC in the United States' largest integrated health system. Arch Pathol Lab Med. 144:478–84. DOI: 10.5858/arpa.2019-0055-OA. PMID: 31469586.

19. Hardie RA, Moore D, Holzhauser D, Legg M, Georgiou A, Badrick T. 2018; Informatics External Quality Assurance (IEQA) Down Under: evaluation of a pilot implementation. J Lab Med. 42:297–304. DOI: 10.1515/labmed-2018-0050.

20. Rychert J. 2023; In support of interoperability: a laboratory perspective. Int J Lab Hematol. 45:436–41. DOI: 10.1111/ijlh.14113. PMID: 37337695.

21. The Royal College of Pathologists of Australasia. Pathology terminology and information standardisation projects. https://www.rcpa.edu.au/Library/Practising-Pathology/PTIS. Updated on April 2024.

22. Hauser RG, Gisriel S, El-Khoury J. 2022; The surprising absence of a laboratory result standard. Am J Clin Pathol. 157:642–3. DOI: 10.1093/ajcp/aqab198. PMID: 34871342.

23. Cho J, Jeong TD, Moon SY, Chung JW, Nam Y, Lee SG, et al. 2022; Current status of reporting units and unit sizes of quantitative test results of clinical chemistry in Korea. Lab Med Online. 12:292–303. DOI: 10.47429/lmo.2022.12.4.292.

24. Thygesen K, Alpert JS, Jaffe AS, Simoons ML, Chaitman BR, White HD, et al. 2012; Third universal definition of myocardial infarction. J Am Coll Cardiol. 60:1581–98. DOI: 10.1016/j.jacc.2012.08.001. PMID: 22958960.

25. Barth JH, Panteghini M, Bunk DM, Christenson RH, Katrukha A, Noble JE, et al. 2014; Recommendation to harmonize the units for reporting cardiac troponin results. Clin Chim Acta. 432:166. DOI: 10.1016/j.cca.2013.10.023. PMID: 24211729.

26. McKeeman GC, Auld PW. 2015; A national survey of troponin testing and recommendations for improved practice. Ann Clin Biochem. 52:527–42. DOI: 10.1177/0004563214568163. PMID: 25568139.

27. Secchiero S, Sciacovelli L, Plebani M. 2018; Harmonization of units and reference intervals of plasma proteins: state of the art from an external quality assessment scheme. Clin Chem Lab Med. 57:95–105. DOI: 10.1515/cclm-2017-1172. PMID: 29750639.

28. National Pathology Accreditation Advisory Council (NPAAC). 2022. Requirements for information communication and reporting. 5th ed. Australian Commission on Safety and Quality in Health Care;Australia:

29. Kidney Disease: Improving Global Outcomes CKDWG. 2024; KDIGO 2024 clinical practice guideline for the evaluation and management of chronic kidney disease. Kidney Int. 105:S117–S314. DOI: 10.1016/j.kint.2023.10.018. PMID: 38490803.

30. Jones GRD, Barker G, Goodall I, Schneider HG, Shephard MDS, Twigg SM. 2011; Change of HbA1c reporting to the new SI units. Med J Aust. 195:45–6. DOI: 10.5694/j.1326-5377.2011.tb03190.x. PMID: 21728944.

31. Sinnott M, Eley R, Steinle V, Boyde M, Trenning L, Dimeski G. 2014; Decimal numbers and safe interpretation of clinical pathology results. J Clin Pathol. 67:179–81. DOI: 10.1136/jclinpath-2013-201865. PMID: 24043714.

32. Coskun A. 2006; Westgard multirule for calculated laboratory tests. Clin Chem Lab Med. 44:1183–7. DOI: 10.1515/CCLM.2006.233. PMID: 17032128.

33. Kraut JA, Madias NE. 2007; Serum anion gap: its uses and limitations in clinical medicine. Clin J Am Soc Nephrol. 2:162–74. DOI: 10.2215/CJN.03020906. PMID: 17699401.

34. Hong J, Gu H, Lee J, Lee W, Chun S, Han KH, et al. 2023; Intuitive modification of the Friedewald formula for calculation of LDL-cholesterol. Ann Lab Med. 43:29–37. DOI: 10.3343/alm.2023.43.1.29. PMID: 36045054. PMCID: PMC9467839.

35. Martins J, Rossouw HM, Pillay TS. 2022; How should low-density lipoprotein cholesterol be calculated in 2022? Curr Opin Lipidol. 33:237–56. DOI: 10.1097/MOL.0000000000000833. PMID: 35942811.

36. Jeong TD, Hong J, Lee W, Chun S, Min WK. 2023; Accuracy of the new creatinine-based equations for estimating glomerular filtration rate in Koreans. Ann Lab Med. 43:244–52. DOI: 10.3343/alm.2023.43.3.244. PMID: 36544336. PMCID: PMC9791020.

37. Meeusen JW, Kasozi RN, Larson TS, Lieske JC. 2022; Clinical impact of the refit CKD-EPI 2021 creatinine-based eGFR equation. Clin Chem. 68:534–9. DOI: 10.1093/clinchem/hvab282. PMID: 35038721.

38. Inker LA, Eneanya ND, Coresh J, Tighiouart H, Wang D, Sang Y, et al. 2021; New creatinine- and cystatin C-based equations to estimate GFR without race. N Engl J Med. 385:1737–49. DOI: 10.1056/NEJMoa2102953. PMID: 34554658. PMCID: PMC8822996.

39. Miller WG, Kaufman HW, Levey AS, Straseski JA, Wilhelms KW, Yu HE, et al. 2022; National Kidney Foundation Laboratory Engagement Working Group recommendations for implementing the CKD-EPI 2021 race-free equations for estimated glomerular filtration rate: practical guidance for clinical laboratories. Clin Chem. 68:511–20. DOI: 10.1093/clinchem/hvab278. PMID: 34918062.

40. Pottel H, Delanaye P, Cavalier E. 2024; Exploring renal function assessment: creatinine, cystatin C, and estimated glomerular filtration rate focused on the European Kidney Function Consortium equation. Ann Lab Med. 44:135–43. DOI: 10.3343/alm.2023.0237. PMID: 37909162. PMCID: PMC10628758.

41. Lee HS, Bae GE, Lee JE, Park HD. 2023; Effect of two cystatin C reagents and four equations on glomerular filtration rate estimations after standardization. Ann Lab Med. 43:565–73. DOI: 10.3343/alm.2023.43.6.565. PMID: 37387489. PMCID: PMC10345172.

42. Jeong TD, Lee W, Chun S, Lee SK, Ryu JS, Min WK, et al. 2013; Comparison of the MDRD study and CKD-EPI equations for the estimation of the glomerular filtration rate in the Korean general population: the fifth Korea National Health and Nutrition Examination Survey (KNHANES V-1), 2010. Kidney Blood Press Res. 37:443–50. DOI: 10.1159/000355724. PMID: 24247487. PMID: 38d55316fea14a17b646022c9c694433.

43. Levey AS, Stevens LA, Schmid CH, Zhang YL, Castro AF 3rd, Feldman HI, et al. 2009; A new equation to estimate glomerular filtration rate. Ann Intern Med. 150:604–12. DOI: 10.7326/0003-4819-150-9-200905050-00006. PMID: 19414839. PMCID: PMC2763564.

44. Stevens LA, Li S, Kurella Tamura M, Chen SC, Vassalotti JA, Norris KC, et al. 2011; Comparison of the CKD Epidemiology Collaboration (CKD-EPI) and Modification of Diet in Renal Disease (MDRD) study equations: risk factors for and complications of CKD and mortality in the Kidney Early Evaluation Program (KEEP). Am J Kidney Dis. 57(S2):S9–16. DOI: 10.1053/j.ajkd.2010.11.007. PMID: 21338849. PMCID: PMC3298760.

45. Miller WG. 2017; Harmonization: its time has come. Clin Chem. 63:1184–6. DOI: 10.1373/clinchem.2017.274860. PMID: 28522443.

46. International Organization for Standardization. 2003. In vitro diagnostic medical devices-requirements for establishing metrological traceability of values assigned to calibrators, trueness control materials and human samples. International Organization for Standardization;Geneva: DOI: 10.3403/02874017u.

47. Joint Committee for Traceability in Laboratory Medicine. JCTLM Database: higher-order reference materials, methods and services v1.45. https://www.jctlmdb.org/. Updated on April 2024.

48. Lee S, Yu J, Cho CI, Cho EJ, Jeong TD, Kim S, et al. 2024; Impact of academia-government collaboration on laboratory medicine standardization in South Korea: analysis of eight years creatinine proficiency testing experience. Clin Chem Lab Med. 62:861–9. DOI: 10.1515/cclm-2023-1160. PMID: 37999449.

49. Kim S, Lee K, Park HD, Lee YW, Chun S, Min WK. 2021; Schemes and performance evaluation criteria of Korean Association of External Quality Assessment (KEQAS) for improving laboratory testing. Ann Lab Med. 41:230–9. DOI: 10.3343/alm.2021.41.2.230. PMID: 33063686. PMCID: PMC7591290.

50. Jeong TD, Cho EJ, Lee K, Lee W, Yun YM, Chun S, et al. 2021; Recent trends in creatinine assays in Korea: long-term accuracy-based proficiency testing survey data by the Korean Association of External Quality Assessment Service (2011-2019). Ann Lab Med. 41:372–9. DOI: 10.3343/alm.2021.41.4.372. PMID: 33536355. PMCID: PMC7884186.

51. Kim S. 2023; Laboratory data quality evaluation in the big data era. Ann Lab Med. 43:399–400. DOI: 10.3343/alm.2023.43.5.399. PMID: 37080739. PMCID: PMC10151286.

52. Kim S, Cho EJ, Jeong TD, Park HD, Yun YM, Lee K, et al. 2023; Proposed model for evaluating real-world laboratory results for big data research. Ann Lab Med. 43:104–7. DOI: 10.3343/alm.2023.43.1.104. PMID: 36045065. PMCID: PMC9467825.

53. Kim S, Jeong TD, Lee K, Chung JW, Cho EJ, Lee S, et al. 2024; Quantitative evaluation of the real-world harmonization status of laboratory test items using external quality assessment data. Ann Lab Med. 44:529–36. DOI: 10.3343/alm.2024.0082. PMID: 38919008. PMCID: PMC11375196.

54. van Rossum HH, Holdenrieder S, Ballieux B, Badrick TC, Yun YM, Zhang C, et al. 2024; Investigating the current harmonization status of tumor markers using global external quality assessment programs: a feasibility study. Clin Chem. 70:669–79. DOI: 10.1093/clinchem/hvae005. PMID: 38385453.

55. Ihde N, Marten P, Eleliemy A, Poerwawinata G, Silva P, Tolovski I, et al. Nambiar R, Poess M, editors. 2022. A survey of big data, high performance computing, and machine learning benchmarks. Technology Conference on Performance Evaluation and Bechnmarking. TPCTC 2021. Springer International Publishing;Cham: p. 98–118. DOI: 10.1007/978-3-030-94437-7_7.

56. Zhou L, Pan S, Wang J, Vasilakos AV. 2017; Machine learning on big data: opportunities and challenges. Neurocomputing. 237:350–61. DOI: 10.1016/j.neucom.2017.01.026.

57. Horowitz GL, Altaie S, Boyd JC, Ceriotti F, Garg U, Horn P, et al. 2010. Defining, establishing, and verifying reference intervals in the clinical laboratory. 3rd ed. Clinical and Laboratory Standards Institute;Wayne, PA: CLSI EP28-A3C.

58. Adeli K, Higgins V, Trajcevski K, White-Al Habeeb N. 2017; The Canadian laboratory initiative on pediatric reference intervals: a CALIPER white paper. Crit Rev Clin Lab Sci. 54:358–413. DOI: 10.1080/10408363.2017.1379945. PMID: 29017389.

59. Colantonio DA, Kyriakopoulou L, Chan MK, Daly CH, Brinc D, Venner AA, et al. 2012; Closing the gaps in pediatric laboratory reference intervals: a CALIPER database of 40 biochemical markers in a healthy and multiethnic population of children. Clin Chem. 58:854–68. DOI: 10.1373/clinchem.2011.177741. PMID: 22371482.

60. Estey MP, Cohen AH, Colantonio DA, Chan MK, Marvasti TB, Randell E, et al. 2013; CLSI-based transference of the CALIPER database of pediatric reference intervals from Abbott to Beckman, Ortho, Roche and Siemens clinical chemistry assays: direct validation using reference samples from the CALIPER cohort. Clin Biochem. 46:1197–219. DOI: 10.1016/j.clinbiochem.2013.04.001. PMID: 23578738.

61. Abou El Hassan M, Stoianov A, Araújo PAT, Sadeghieh T, Chan MK, Chen Y, et al. 2015; CLSI-based transference of CALIPER pediatric reference intervals to Beckman Coulter AU biochemical assays. Clin Biochem. 48:1151–9. DOI: 10.1016/j.clinbiochem.2015.05.002. PMID: 25979809.

62. Araújo PAT, Thomas D, Sadeghieh T, Bevilacqua V, Chan MK, Chen Y, et al. 2015; CLSI-based transference of the CALIPER database of pediatric reference intervals to Beckman Coulter DxC biochemical assays. Clin Biochem. 48:870–80. DOI: 10.1016/j.clinbiochem.2015.06.002. PMID: 26070714.

63. Higgins V, Chan MK, Nieuwesteeg M, Hoffman BR, Bromberg IL, Gornall D, et al. 2016; Transference of CALIPER pediatric reference intervals to biochemical assays on the Roche cobas 6000 and the Roche Modular P. Clin Biochem. 49:139–49. DOI: 10.1016/j.clinbiochem.2015.08.018. PMID: 26297116.

64. Budd JR, Durham AP, Gwise TE, Hawkins DM, Holland M, Iriarte B, et al. 2018. Measurement procedure comparison and bias estimation using patient samples. 3rd ed. Clinical and Laboratory Standards Institute;Wayne, PA: CLSI EP09c. DOI: 10.1515/cclm-2024-0595.

65. Gong Y, Liu G, Xue Y, Li R, Meng L. 2023; A survey on dataset quality in machine learning. Inf Softw Technol. 162:107268. DOI: 10.1016/j.infsof.2023.107268.

66. Fenza G, Gallo M, Loia V, Orciuoli F, Herrera-Viedma E. 2021; Data set quality in machine learning: consistency measure based on group decision making. Appl Soft Comput. 106:107366. DOI: 10.1016/j.asoc.2021.107366.

67. Thompson S, Chesher D. 2018; Lot-to-lot variation. Clin Biochem Rev. 39:51–60. PMID: 30473592. PMCID: PMC6223607.

68. Algeciras-Schimnich A, Bruns DE, Boyd JC, Bryant SC, La Fortune KA, Grebe SKG. 2013; Failure of current laboratory protocols to detect lot-to-lot reagent differences: findings and possible solutions. Clin Chem. 59:1187–94. DOI: 10.1373/clinchem.2013.205070. PMID: 23592508.

69. Thaler MA, Iakoubov R, Bietenbeck A, Luppa PB. 2015; Clinically relevant lot-to-lot reagent difference in a commercial immunoturbidimetric assay for glycated hemoglobin A1c. Clin Biochem. 48:1167–70. DOI: 10.1016/j.clinbiochem.2015.07.018. PMID: 26187005.

70. Kim JH, Cho Y, Lee SG, Yun YM. 2019; Report of Korean Association of External Quality Assessment Service on the accuracy-based lipid proficiency testing (2016-2018). J Lab Med Qual Assur. 41:121–9. DOI: 10.15263/jlmqa.2019.41.3.121.

71. Böttcher S, van der Velden VHJ, Villamor N, Ritgen M, Flores-Montero J, Escobar HM, et al. 2019; Lot-to-lot stability of antibody reagents for flow cytometry. J Immunol Methods. 475:112294. DOI: 10.1016/j.jim.2017.03.018. PMID: 28365329.

72. Kitchen AD, Newham JA. 2010; Lot release testing of serological infectious disease assays used for donor and donation screening. Vox Sang. 98:508–16. DOI: 10.1111/j.1423-0410.2009.01305.x. PMID: 20070648.

73. Yetley EA, Pfeiffer CM, Schleicher RL, Phinney KW, Lacher DA, Christakos S, et al. 2010; NHANES monitoring of serum 25-hydroxyvitamin D: a roundtable summary. J Nutr. 140:2030S–45S. DOI: 10.3945/jn.110.121483. PMID: 20881084. PMCID: PMC2955879.

74. Selvin E, Juraschek SP, Eckfeldt J, Levey AS, Inker LA, Coresh J. 2013; Calibration of cystatin C in the National Health and Nutrition Examination Surveys (NHANES). Am J Kidney Dis. 61:353–4. DOI: 10.1053/j.ajkd.2012.09.013. PMID: 23177702. PMCID: PMC3703771.

75. Selvin E, Manzi J, Stevens LA, Van Lente F, Lacher DA, Levey AS, et al. 2007; Calibration of serum creatinine in the National Health and Nutrition Examination Surveys (NHANES) 1988-1994, 1999-2004. Am J Kidney Dis. 50:918–26. DOI: 10.1053/j.ajkd.2007.08.020. PMID: 18037092.

76. Yun YM, Song J, Ji M, Kim JH, Kim Y, Park T, et al. 2017; Calibration of high-density lipoprotein cholesterol values from the Korea National Health and Nutrition Examination Survey data, 2008 to 2015. Ann Lab Med. 37:1–8. DOI: 10.3343/alm.2017.37.1.1. PMID: 27834059. PMCID: PMC5107612.

77. Katzman BM, Ness KM, Algeciras-Schimnich A. 2017; Evaluation of the CLSI EP26-A protocol for detection of reagent lot-to-lot differences. Clin Biochem. 50:768–71. DOI: 10.1016/j.clinbiochem.2017.03.012. PMID: 28322754.

78. CLSI. 2013. User evaluation of between-reagent lot variation. Clinical and Laboratory Standards Institute;Wayne, PA: EP26-A.

79. CLSI. 2012. Verification of comparability of patient results within one health care system. Clinical and Laboratory Standards Institute;Wayne, PA: EP31-A-IR.

80. Loh TP, Markus C, Tan CH, Tran MTC, Sethi SK, Lim CY. 2023; Lot-to-lot variation and verification. Clin Chem Lab Med. 61:769–76. DOI: 10.1515/cclm-2022-1126. PMID: 36420533.

81. Cho EJ, Jeong TD, Kim S, Park HD, Yun YM, Chun S, et al. 2023; A new strategy for evaluating the quality of laboratory results for big data research: using external quality assessment survey data (2010-2020). Ann Lab Med. 43:425–33. DOI: 10.3343/alm.2023.43.5.425. PMID: 37080743. PMCID: PMC10151270.