Healthc Inform Res.  2015 Oct;21(4):230-238. 10.4258/hir.2015.21.4.230.

Evaluation Framework for Telemedicine Using the Logical Framework Approach and a Fishbone Diagram

• Chang H

• KMID: 2166871
• DOI: http://doi.org/10.4258/hir.2015.21.4.230

Abstract

OBJECTIVES
Technological advances using telemedicine and telehealth are growing in healthcare fields, but the evaluation framework for them is inconsistent and limited. This paper suggests a comprehensive evaluation framework for telemedicine system implementation and will support related stakeholders' decision-making by promoting general understanding, and resolving arguments and controversies.
METHODS
This study focused on developing a comprehensive evaluation framework by summarizing themes across the range of evaluation techniques and organized foundational evaluation frameworks generally applicable through studies and cases of diverse telemedicine. Evaluation factors related to aspects of information technology; the evaluation of satisfaction of service providers and consumers, cost, quality, and information security are organized using the fishbone diagram.
RESULTS
It was not easy to develop a monitoring and evaluation framework for telemedicine since evaluation frameworks for telemedicine are very complex with many potential inputs, activities, outputs, outcomes, and stakeholders. A conceptual framework was developed that incorporates the key dimensions that need to be considered in the evaluation of telehealth implementation for a formal structured approach to the evaluation of a service. The suggested framework consists of six major dimensions and the subsequent branches for each dimension.
CONCLUSIONS
To implement telemedicine and telehealth services, stakeholders should make decisions based on sufficient evidence in quality and safety measured by the comprehensive evaluation framework. Further work would be valuable in applying more comprehensive evaluations to verify and improve the comprehensive framework across a variety of contexts with more factors and participant group dimensions.

Keyword

Telemedicine; Program Evaluation; Quality of Healthcare; Patient Safety

MeSH Terms

Delivery of Health Care
Logic*
Patient Safety
Program Evaluation
Quality of Health Care
Telemedicine*

Figure

• 

  Figure 1 Logical framework approach for telemedicine implementation.

• 

  Figure 2 Fishbone diagram for a comprehensive evaluation framework for telemedicine implementation.

Reference

1. Schwamm LH. Telehealth: seven strategies to successfully implement disruptive technology and transform health care. Health Aff (Millwood). 2014; 33(2):200–206.

2. Akalu R, Rossos PG, Chan CT. The role of law and policy in tele-monitoring. J Telemed Telecare. 2006; 12(7):325–327.

3. Brooks E, Turvey C, Augusterfer EF. Provider barriers to telemental health: obstacles overcome, obstacles remaining. Telemed J E Health. 2013; 19(6):433–437.

4. Kim JE, Song YM, Park JH, Lee JR. Attitude of Korean primary care family physicians towards telehealth. Korean J Fam Med. 2011; 32(6):341–351.

5. u-Health Forum Korea. Current status of telemedicine [Internet]. Seoul, Korea: u-Health Forum Korea;2013. cited at 2015 Sep 7. Available from: http://www.u-health.or.kr/library/file_down.php?file_path=../upload/pds/&data_file=230_1387075618.pdf&file_name=원격진료도입현황(UHA).pdf.

6. Hein MA. Telemedicine: an important force in the transformation of healthcare [Internet]. Washington (DC): US Department of Commerce;2009. cited at 2015 Sep 7. Available from: http://ita.doc.gov/td/health/telemedicine_2009.pdf.

7. van Dyk L. A review of telehealth service implementation frameworks. Int J Environ Res Public Health. 2014; 11(2):1279–1298.

8. Field MJ. Institute of Medicine. Telemedicine: a guide to assessing telecommunications for health care. Washington (DC): National Academies Press;1996.

9. Lustig TA. Institute of Medicine. The role of telehealth in an evolving health care environment: workshop summary. Washington (DC): National Academies Press;2012.

10. Poultney N. A comprehensive evaluation framework for telehealth services [Internet]. Adelaide, South Australia: Australasian Telehealth Society;2014. cited at 2015 Sep 8. Available from: http://event.icebergevents.com.au/uploads/contentFiles/files/2014-SFT/Nathan-Poultney.pdf.

11. Hebert M. Telehealth success: evaluation framework development. Stud Health Technol Inform. 2001; 84(Pt 2):1145–1149.

12. Wootton R, Vladzymyrskyy A, Zolfo M, Bonnardot L. Experience with low-cost telemedicine in three different settings. Recommendations based on a proposed framework for network performance evaluation. Glob Health Action. 2011 Dec 6 [Epub]. DOI: 10.3402/gha.v4i0.7214.

13. Chaula JA, Yngstrom L, Kowalski S. A framework for evaluation of information systems security. In : Proceedings of the ISSA 2005 new knowledge today conference; 2005 Jun 29-Jul 1; Sandton, South Africa. p. 1–11.

14. Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care: trends in evaluation research 1982-2002. Stud Health Technol Inform. 2004; 107(Pt 2):1289–1294.

15. Donabedian A. The evaluation of medical care programs. Bull N Y Acad Med. 1968; 44(2):117–124.

16. Delone WH, McLean ER. The DeLone and McLean model of information systems success: a ten-year update. J Manag Inf Syst. 2003; 19(4):9–30.

17. King WR, Rodriguez JI. Evaluating management information systems. MIS Q. 1978; 2(3):43–51.

18. Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 1989; 13(3):319–340.

19. Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: toward a unified view. MIS Q. 2003; 27(3):425–478.

20. Shaw NT. 'CHEATS': a generic information communication technology (ICT) evaluation framework. Comput Biol Med. 2002; 32(3):209–220.

21. Mammen A, Weeks R. Electronic Medical Record (EMR) technology acceptance by healthcare professionals in South Africa. In : Proceedings of 2014 Portland International Conference on Management of Engineering & Technology (PICMET); 2014 Jul 27-31; Kanazawa, Japan. p. 3539–3548.

22. Ohinmaa A, Reponen J. A model for the assessment of telemedicine and a plan for testing of the model within five specialities. Helsinki: Finnish Office for Health Care Technology Assessment;1997.

23. Bashshur R, Shannon G, Krupinski E, Grigsby J. The taxonomy of telemedicine. Telemed J E Health. 2011; 17(6):484–494.

24. Dattakumar A. A unified approach for the evaluation of telehealth implementations in Australia. Melbourne, Australia: Institute for a Broadband-Enabled Society;2013.

25. Van Dyk L. The development of a telemedicine service maturity model [dissertation]. Stellenbosch, South Africa: Stellenbosch University;2013.

26. Nepal S, Li J, Jang-Jaccard J, Alem L. A framework for telehealth program evaluation. Telemed J E Health. 2014; 20(4):393–404.

27. Bergmo TS. Can economic evaluation in telemedicine be trusted? A systematic review of the literature. Cost Eff Resour Alloc. 2009; 7:18.

28. Hersh WR, Hickam DH, Severance SM, Dana TL, Krages KP, Helfand M. Telemedicine for the medicare population: update. Evid Rep Technol Assess (Full Rep). 2006; (131):1–41.

29. Zanni GR. Telemedicine: sorting out the benefits and obstacles. Consult Pharm. 2011; 26(11):810–812. 814821–824.

30. Goldberg LR. Assessing quality of telehealth: home heart failure care comparing patient-driven technology models. In : Proceedings of Agency for Healthcare Research and Quality (AHRQ) 2007 Annual Conference; 2007 Sep 26-28; Bethesda, MD.

31. Grain H, Schaper LK. Health informatics: digital health service delivery: the future is now! Amsterdam: IOS Press;2013.

32. Cruz-Cunha MM. Handbook of research on developments in e-health and telemedicine: technological and social perspectives. Hershey (PA): IGI Global;2010.

33. Tanriverdi H, Iacono CS. Knowledge barriers to diffusion of telemedicine. In : Proceedings of the International Conference on Information Systems; 1998 Aug 14-16; Helsinki, Finland. p. 39–50.

34. Whitten PS, Allen A. Analysis of telemedicine from an organizational perspective. Telemed J. 1995; 1(3):203–213.

35. Adler-Milstein J, Kvedar J, Bates DW. Telehealth among US hospitals: several factors, including state reimbursement and licensure policies, influence adoption. Health Aff (Millwood). 2014; 33(2):207–215.

36. The World Bank. The LogFrame handbook: a logical framework approach to project cycle management [Internet]. Washington (DC): The World Bank;2000. cited at 2015 Oct 10. Available from: http://www.afdb.org/fileadmin/uploads/afdb/Documents/Evaluation-Reports-_Shared-With-OPEV_/00158077-EN-WBLOGICALFRAMEWORK-HANDBOOK.PDF.

37. Tague NR. The quality toolbox. 2nd ed. Milwaukee (WI): American Society for Quality;2005. p. 247–249.