Restor Dent Endod.  2024 Feb;49(1):e3. 10.5395/rde.2024.49.e3.

Sample size determination for conducting a pilot study to assess reliability of a questionnaire

• Bujang MA
• Omar ED
• Foo DHP
• Hon YK

• KMID: 2554131
• DOI: http://doi.org/10.5395/rde.2024.49.e3

Abstract

This article is a narrative review that discusses the recommended sample size requirements to design a pilot study to assess the reliability of a questionnaire. A list of various sample size tables that are based on the kappa agreement test, intra-class correlation test and Cronbach’s alpha test has been compiled together. For all calculations, type I error (alpha) was set at a maximum value of 0.05, and power was set at a minimum value of 80.0%. For the kappa agreement test, intra-class correlation test, and Cronbach’s alpha test, the recommended minimum sample size requirement based on the ideal effect sizes shall be at least 15, 22, and 24 subjects respectively. By making allowances for a non-response rate of 20.0%, a minimum sample size of 30 respondents will be sufficient to assess the reliability of the questionnaire. The clear guideline of minimum sample size requirement for the pilot study to assess the reliability of a questionnaire is discussed and this will ease researchers in preparation for the pilot study. This study provides justification for a minimum requirement of a sample size of 30 respondents specifically to test the reliability of a questionnaire.

Keyword

Pilot study; Questionnaire; Reliability; Sample size

Reference

1. Moore CG, Carter RE, Nietert PJ, Stewart PW. Recommendations for planning pilot studies in clinical and translational research. Clin Transl Sci. 2011; 4:332–337. PMID: 22029804.

2. Jalaludin M, Fuziah M, Hong J, Mohamad Adam B, Jamaiyah H. Reliability and validity of the revised Summary of Diabetes Self-Care Activities (SDSCA) for Malaysian children and adolescents. Malays Fam Physician. 2012; 7:10–20. PMID: 25606251.

3. Yunus A, Seet W, Mohamad Adam B, Haniff J. Validation of the Malay version of Berlin questionaire to identify Malaysian patients for obstructive sleep apnea. Malays Fam Physician. 2013; 8:5–11. PMID: 25606261.

4. Cohen JA. Coefficient of agreement for nominal scales. Educ Psychol Meas. 1960; 20:37–46.

5. Feldman RS, Douglass CW, Loftus ER, Kapur KK, Chauncey HH. Interexaminer agreement in the measurement of periodontal disease. J Periodontal Res. 1982; 17:80–89. PMID: 6211539.

6. Lemasney J, O’Mullane D, Coleman M. Effect of fluoridation on dental health in 5- and 11-year-old Irish schoolchildren. Community Dent Oral Epidemiol. 1984; 12:218–222. PMID: 6590172.

7. Mann J, Goultschin J, Call RL. Assessment of inter-examiner agreement in scoring periodontal disease. J Periodontal Res. 1985; 20:86–90. PMID: 3156239.

8. Bujang MA, Baharum N. A simplified guide to determination of sample size requirements for estimating the value of intraclass correlation coefficient: a review. Arch Orofac Sci. 2017; 12:1–11.

9. Bartko JJ. On various intraclass correlation reliability coefficients. Psychol Bull. 1976; 83:762–765.

10. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951; 16:297–334.

11. Leontitsis A, Pagge J. A simulation approach on Cronbach’s alpha statistical significance. Math Comput Simul. 2007; 73:336–340.

12. Flack VF, Afifi AA, Lachenbruch PA, Schouten HJA. Sample size determinations for the two rater kappa statistic. Psychometrika. 1988; 53:321–325.

13. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977; 33:159–174. PMID: 843571.

14. Fleiss JL. Statistical methods for rates and proportions. 2nd ed. New York, NY: John Wiley;1981.

15. Bujang MA, Baharum N. Guidelines of the minimum sample size requirements for Kappa agreement test. Epidemiol Biostat Public Health. 2017; 14:e12267-1–ee12267-10.

16. Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med. 1998; 17:101–110. PMID: 9463853.

17. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol. 1994; 6:284–290.

18. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016; 15:155–163. PMID: 27330520.

19. Bonett DG. Sample size requirements for testing and estimating coefficient alpha. J Educ Behav Stat. 2002; 27:335–340.

20. Nunnally JC. Psychometric theory. New York, NY: McGraw-Hill;1967.

21. Bujang MA, Omar ED, Baharum NA. A review on sample size determination for Cronbach’s alpha test: a simple guide for researchers. Malays J Med Sci. 2018; 25:85–99. PMID: 30914882.

22. Hertzog MA. Considerations in determining sample size for pilot studies. Res Nurs Health. 2008; 31:180–191. PMID: 18183564.

23. Johanson GA, Brooks G. Initial scale development: sample size for pilot studies. Educ Psychol Meas. 2010; 70:394–400.

24. Bujang MA. A step-by-step process on sample size determination for medical research. Malays J Med Sci. 2021; 28:15–27.

25. In : Bujang MA, Ab-Ghani P, Soelar SA, Zulkifli NA, editors. Sample size guideline for exploratory factor analysis when using small sample: taking into considerations of different measurement scales. 2012 International Conference on Statistics in Science, Business and Engineering (ICSSBE); 2012 Sep 10-12; Langkawi, Malaysia. Piscataway, NJ: IEEE;2012. 12. p. 1–5.

26. Bujang MA, Adnan TH. Requirements for minimum sample size for sensitivity and specificity analysis. J Clin Diagn Res. 2016; 10:YE01–YE06.

27. Bujang MA, Ghani PA, Soelar SA, Zulkifli NA, Omar ED. Invalid skewed responses contributes to invalid factor solution in exploratory factor analysis: a validation approach using real-life data. J Behav Health. 2019; 8:152–160.

28. Bujang MA. An elaboration on sample size planning for performing a one-sample sensitivity and specificity analysis by basing on calculations on a specified 95% confidence interval width. Diagnostics (Basel). 2023; 13:1390. PMID: 37189491.

29. Bujang MA, Mastura I, Bariyyah K, Baharum N, Othman SH, Mat-Lazim SS, et al. Validation of the summary of diabetes self-care activities (SDSCA) in Malay language for Malaysian adults. Malays J Public Health Med. 2016; 16:227–234.

30. Bujang MA, Lai WH, Ratnasingam S, Tiong XT, Hon YK, Yap EP, et al. Development of a quality-of-life instrument to measure current health outcomes: health-related quality of life with six domains (HRQ-6D). J Clin Med. 2023; 12:2816. PMID: 37109153.