ABSTRACT

Objective: To test the reliability, validity, and interpretability of the Brazilian version of the Clinical COPD Questionnaire (CCQ) in patients with COPD. Methods: Fifty patients with COPD completed the CCQ by interview on two occasions. At the first visit, the CCQ was administered twice, by two different raters, approximately 10 min apart; the patients also underwent spirometry and were administered the COPD Assessment Test, the modified Medical Research Council scale, and Saint George's Respiratory Questionnaire (SGRQ). At the second visit (1-2 weeks later), the CCQ was readministered. We tested the hypothesis that the CCQ total score would correlate positively with the total and domain SGRQ scores (r = 0.5). Results: Of the 50 patients, 30 (60%) were male. The mean age was 66 ± 8 years, and the mean FEV1 was 44.7 ± 17.9% of the predicted value. For all CCQ items, Cronbach's alpha coefficient (95% CI) was 0.93 (0.91-0.96). To analyze the interrater reliability and test-retest reliability of the CCQ, we calculated the two-way mixed effects model/single measure type intraclass correlation coefficient (0.97 [95% CI: 0.95-0.98] and 0.92 [95% CI: 0.86-0.95], respectively); the agreement standard error of measurement (0.65 for both); the smallest detectable change at the individual level (1.81 and 1.80, respectively) and group level (0.26 and 0.25, respectively); and the limits of agreement (-0.58 to 0.82 and -1.14 to 1.33, respectively). The CCQ total score correlated positively with all SGRQ scores (r = 0.70 for all). Conclusions: The Brazilian version of the CCQ showed an indeterminate measurement error, as well as satisfactory interrater/test-retest reliability and construct validity.

Keywords: Pulmonary disease, chronic obstructive; Health status; Patient reported outcome measures; Validation study.

RESUMO

Objetivo: Testar a confiabilidade, validade e interpretabilidade da versão brasileira do Clinical COPD Questionnaire (CCQ) em pacientes com DPOC. Métodos: Cinquenta pacientes com DPOC preencheram o CCQ por meio de entrevista em duas ocasiões. Na primeira visita, o CCQ foi aplicado duas vezes, por dois avaliadores, com intervalo de aproximadamente 10 min; os pacientes também foram submetidos a espirometria e aplicação do COPD Assessment Test, da escala modificada do Medical Research Council e do Saint George’s Respiratory Questionnaire (SGRQ). Na segunda visita (1-2 semanas depois), o CCQ foi reaplicado. Testamos a hipótese de que a pontuação total no CCQ se correlacionaria positivamente com a pontuação total e a pontuação nos domínios do SGRQ (r ≥ 0,5). Resultados: Dos 50 pacientes, 30 (60%) eram do sexo masculino. A média de idade foi de 66 ± 8 anos, e a média do VEF1 foi de 44,7 ± 17,9% do valor previsto. Para todos os itens do CCQ, o coeficiente alfa de Cronbach (IC95%) foi de 0,93 (0,91-0,96). Para analisar a confiabilidade interavaliadores e teste-reteste do CCQ, calculamos o coeficiente de correlação intraclasse de duas vias modelo de efeitos mistos para medidas únicas (0,97 [IC95%: 0,95-0,98] e 0,92 [IC95%: 0,86-0,95], respectivamente); erro-padrão de medida do tipo concordância (0,65 para ambas); a mínima mudança detectável individual (1,81 e 1,80, respectivamente) e no grupo (0,26 e 0,25, respectivamente); e os limites de concordância (−0,58 a 0,82 e −1,14 a 1,33, respectivamente). A pontuação total no CCQ correlacionou-se positivamente com todas as pontuações no SGRQ (r ≥ 0,70 para todas). Conclusões: A versão brasileira do CCQ apresentou erro de medida indeterminado, assim como confiabilidade interavaliadores/teste-reteste e validade de construto satisfatórias.

Palavras-chave: Doença pulmonar obstrutiva crônica; Nível de saúde; Medidas de resultados relatados pelo paciente; Estudo de validação.

 INTRODUCTION
 
Assessments of disease impact and clinical stability in patients with COPD should help physicians to make therapeutic decisions.(1) Patient-reported outcome measures (PROMs), such as the Clinical COPD Questionnaire (CCQ) and the COPD Assessment Test (CAT), are useful to assess disease impact cross-sectionally and clinical stability longitudinally.(1) In accordance with GOLD recommendations, the CCQ and the CAT are comprehensive and suitable PROMs for the assessment of symptoms in patients with COPD.(2)
 
Most COPD patients prefer CCQ to CAT, because the CCQ incorporates more details about daily respiratory problems than does the CAT and, therefore, reflects their health status better. Some patients also point out that the CCQ presents a system of response options which is easier to understand when compared to that of CAT.(3) Furthermore, the International Primary Care Respiratory Group(4) elected the CCQ as the best PROM to evaluate COPD patients in primary care. The CCQ was the only PROM that received top marks in the survey.
 
The selection of a suitable PROM for health status assessment should be based on the quality of its measurement properties—a PROM should be reliable and valid. There are numerous PROMs that can be used in order to measure health status. However, careful selection is of utmost importance to avoid the risk of imprecise or biased results, which could lead to wrong conclusions.(5) Measurement properties may differ between populations and therefore should be tested and considered appropriate for the specific population to be assessed.(6)
 
The CCQ was developed by van der Molen et al.(7) in 2003 with the purpose of promoting the evaluation of clinical control in patients with COPD. The domains selected as the most important for clinical control were functional state, symptoms, and mental state. (7) A Portuguese version of the CCQ for use in Brazil is available on the CCQ website, but its measurement properties have yet to be investigated. The present study aimed to analyze the internal consistency, reliability, measurement error, and construct validity, as well as the floor and ceiling effects, of the Brazilian version of the CCQ, when administered by interview, in patients with COPD.
 
METHODS
 
Patient selection
 
Patients referred to a public outpatient clinic specializing in COPD were invited to participate in the study. Inclusion criteria were as follows: having a confirmed diagnosis of COPD, being ≥ 40 years of age, being a smoker or a former smoker, having no other comorbidity (such as cardiovascular, neurological, orthopedic, rheumatic, or respiratory diseases other than COPD) that negatively impacted on the activities of daily life, and having a Mini-Mental State Examination(8) score ≥ 25 (literates) or ≥ 19 (illiterates). Exclusion criteria were having changes in clinical stability or disease impact in the month prior to the study or during data collection, assessed by closed questions, and not participating in the evaluations of the study. All patients who agreed to participate signed an informed consent form. This study was approved by the Human Research Ethics Committee of the Federal University of Santa Catarina (CAAE no. 33299214.8.0000.0121).
 
Study design
 
The study was conducted in a public outpatient clinic specializing in COPD in two visits in the morning period between 2017 and 2019. The selected PROMs—CAT, CCQ, modified Medical Research Council (mMRC) scale, and Saint George’s Respiratory Questionnaire (SGRQ)—were administered by interviews due to the low level of education of part of the sample. The PROMs were administered in a dedicated room where only the patient and the rater were present. The raters only read the instructions and items of the PROM and wrote down the choices of the patients. The raters are physiotherapists with experience in assessing health status in patients with COPD. At the first visit, spirometry was performed, and the CAT, the mMRC scale, and the SGRQ were administered. In addition, CCQ was first administered by rater 1 and then, approximately 10 min later, by rater 2, for interrater reliability analysis. At the second visit, between one and two weeks later, the CCQ was readministered by rater 1 for test-retest reliability analysis.(9)
 
Assessments
 
Lung function was assessed following the standards recommended by the American Thoracic Society/European Respiratory Society(10) using a spirometer (KoKo Sx 1000; nSpire Health Inc., Longmont, CO, USA). The reference values for post-bronchodilator spirometric variables were those established by Pereira et al.(11) The severity of airflow limitation was based on FEV1 and classified as GOLD I, II, III ou IV.(12)
 
The CAT(13) and the mMRC(14) scale scores, as well as the number of exacerbations within the last 12 months, regardless of hospital admissions, were used in order to classify the impact of COPD on health status and the risk of future events as GOLD A, B, C, or D.(12)
 
The SGRQ(15) was used in order to assess health-related quality of life. The questionnaire consists of 76 items distributed into three domains (symptoms, activity, and psychosocial impact). The total score ranges from 0 to 100, higher scores meaning poorer quality of life.
 
The CCQ(7) was used in order to evaluate clinical control. It consists of 10 items distributed into three domains (symptoms, mental state, and functional state). Total and domain scores range from 0 to 6, higher scores representing poorer control.
 
Statistical analysis
 
Data normality was analyzed by the Shapiro-Wilk test. The level of statistical significance was set at p < 0.05. The internal consistency of the CCQ items was analyzed by the Cronbach’s alpha coefficient (α) and the corresponding 95% CI.(16) To compare the scores between raters and between test and retest, the Student’s t-test or the Wilcoxon test was used according to data normality. Interrater and test-retest reliability of the CCQ scores were analyzed by the two-way mixed effects model/single measurement type intraclass correlation coefficient (ICC3,1) and the corresponding 95% CI.(17) For interrater reliability, test-retest reliability, and measurement error analyses, we calculated the agreement standard error of measurement (SEMagreement), the smallest detectable change at the individual level (SDCindividual) and group level (SDCgroup), and the limits of agreement (LoA).(9) To visualize the total score and the agreement between the CCQ measurements, Bland-Altman plots(18) were used. To analyze construct validity, the following hypothesis was used: the total CCQ score would positively correlate with the total and domain SGRQ scores, and the correlation coefficient (r) would be ≥ 0.5. The percentage of occurrence of minimum and maximum CCQ scores was used in order to analyze the floor and ceiling effects, respectively, which were classified as absent or present.(9)
 
RESULTS
 
Fifty patients with COPD were included in the study. The general characteristics of the sample are described in Table 1.
 
The median administration time of the CCQ by interview was 2.76 min (2.38-3.38 min). In the internal consistency analysis, the values of α (95% CI) for all of the CCQ items (CCQ total score) and for the symptoms (items 1, 2, 5, and 6), mental state (items 3 and 4), and functional state domains (items 7, 8, 9, and 10) were, respectively, 0.93 (0.91-0.96); 0.77 (0.66-0.85); 0.79 (0.64-0.87); and 0.94 (0.91-0.96).
 
Table 2 shows the CCQ total and domain scores in each administration. Differences between raters were observed for the total, the mental state domain, and the functional state domain scores, as well as for the mental state domain score between test and retest (p ≤ 0.05 for all). Table 2 also shows the analysis of interrater reliability, test-retest reliability, and measurement error. All ICC3,1 were ≥ 0.80, 95% CI ranges being broader between test and retest than between raters. The results of SEMagreement, SDCindividual, and SDCgroup were similar between raters and between test and retest for the CCQ total score, but they were lower between raters for the symptoms and mental state domains, as well as between test and retest for the functional state domain. Figure 1 also presents the measurement error by the LoA ranges of the CCQ total score, which were broader between test and retest than between raters.
 
Table 3 and Figure 2 show the correlations between the CCQ scores and SGRQ scores. All correlations were strong (r > 0.70), except for the correlation between the CCQ mental state domain and the SGRQ symptoms and activity domains, which were good (0.50 < r < 0.70).(19)
 
Only 4 patients had a minimum score (8%), and 1 had a maximum score (2%), indicating the absence of the floor and ceiling effects.(9)
 
DISCUSSION
 
In the present study, the measurement properties of the Brazilian version of the CCQ were tested in a sample of patients with COPD in Brazil. The results suggest that this PROM is reliable and valid when administered by interview. To our knowledge, this was the first study to analyze the measurement properties of the Brazilian version of the CCQ.
 
The internal consistency analysis of the Brazilian version of the CCQ revealed values between 0.77 and 0.94. In the original development study of the CCQ,(7) these values ranged from 0.78 to 0.91. Among the domains, the highest value was in the functional state domain,(7) which was similar in the present study. In other validation studies, values above 0.70(20-25) were also found. By definition, internal consistency determines the degree of interrelationship between items.(26) Values below 0.70 indicate a lack of correlation between the PROM items.(27) However, values above 0.95 may indicate that the PROM contains many items that are evaluating the same construct, suggesting redundancy. (28) Therefore, the internal consistency of the Brazilian version of the CCQ and its domains was positive (0.70 ≤ α ≤ 0.95)(9) and sufficient (α ≥ 0.70).(29)
 
This study presented the reliability analysis of the Brazilian version of the CCQ between two raters and over a time interval. The minimum ICC was 0.80. In the study that presented the measurement properties of the original CCQ version,(7) the ICC was 0.94 for the total score between test and retest. Our finding is similar to those reported in the validation studies for the Italian(20) (ICC = 0.99) and Persian(30) (ICC = 0.98) versions of the CCQ. Reliability is defined as the proportion of the total variance in the measurements that is due to true differences among patients. Statistical analysis should preferably be done by calculating the ICC, because it considers systematic errors between repeated measures.(26) In the present study, we chose to use the ICC3,1, in which each individual is evaluated by each rater, these being the only raters of interest, and reliability is calculated from a single measure. (17) ICCs range from 0 to 1; values close to 1 indicate small error variation when compared with patient variation. This means that such values also depend on the heterogeneity of the population, that is, when the population is more homogeneous, it is easier to find an ICC closer to 0.(6) Considering an ICC of at least 0.70 as a quality criterion, it can be said that the reliability of the Brazilian version of the CCQ between raters and between test and retest was positive(9) and sufficient(29) in our sample.
 
There were differences in the scores between the administrations of the CCQ. However, comparison tests are not recommended by the Consensus-based Standards for the Selection of Health Measurement Instruments(29) for reliability analysis. This is due to the fact that such tests show only the agreement between administrations for the central values but do not provide information about agreement between administrations for individual values.(31) In the present study, the CCQ scores in the three administrations were higher in the symptoms and the functional state domains, corroborating the results in the original validation of the CCQ.(7)
 
Measurement error was analyzed by means of SEMagreement, SDCindividual, SDCgroup, and LoA to determine interrater and test-retest reliability for the Brazilian version of the CCQ. The SEMagreement of the total CCQ score was the same between raters and between test and retest (0.65). Tsiligianni et al.(3), studying a sample of clinically stable COPD patients, reported a lower value of SEM for the CCQ regarding test-retest reliability. However, those authors calculated SEM using a different equation, which does not consider the variance due to systematic differences between raters.(6) This measurement property represents the systematic and random error of a patient’s score which is not attributed to real changes in the construct to be measured.(26) In the present study, we calculated SEM, which represents the standard deviation of repeated measures of an individual; and then SDC, which consists of the minimum change that must be overcome to guarantee a real change in the individual.(26) This means that an observed change must be greater than the limit of SDC to be considered true.(6) LoA were demonstrated in Bland-Altman plots to support the interpretation of measurement error size. It is possible to visualize the magnitude of the measurement error when we relate LoA with the score range. By definition, 95% of the differences between repeated measures must be within the LoA range. A value outside that range can indicate a real change.(6) In order to know whether the measurement error is acceptable or not, one must also analyze the minimal important change (MIC). SDC and MIC can be used to decide whether a real and clinically relevant change has occurred with a patient.(28) Other studies about CCQ described a MIC value close to 0.4.(32-35) In the present study, however, MIC was not calculated. Therefore, the interrater and test-retest measurement errors of the Brazilian version of the CCQ was classified as undetermined.(9,29)
 
According to the results, the hypothesis related to the construct analysis was confirmed. There was a positive correlation of r values of at least 0.5 between the total score of the Brazilian version of the CCQ and the SGRQ total and domain scores. For the elaboration of the hypothesis, the minimum r value (0.53) found between the CCQ total score and the SGRQ scores in the validity study of the original CCQ version was considered.(7) Other studies also reported similar r values between the CCQ and the SGRQ.(22,33) Similar results were also found between the CCQ and other PROM scores that assess overall and specific health-related quality of life in patients with COPD(20,21). The CCQ symptoms and functional state domain scores showed a strong correlation (r > 0.70)(19) with all SGRQ domains. The CCQ mental state domain score strongly correlated only with the total and the impact domain scores of SGRQ, probably because this is the only SGRQ domain that has questions about psychosocial changes.(15) The construct validity estimates the degree to which PROM scores are consistent with assumptions based on the hypothesis that the PROM validly measures the construct that is intended to be measured.(9,28) In the sample studied, the construct validity of the CCQ reached the quality criteria, being rated as positive(9) and sufficient,(29) because the hypothesis of the construct validity was met.
 
In the interpretability analysis, the floor and ceiling effects were not observed. In similar studies, the floor and ceiling effects were not detected in the CCQ total score either.(21,22) The presence of a floor or ceiling effect may indicate that extreme items are missing in the lower or upper end of the PROM and can limit its ability to discriminate patients and to measure changes.(9)
 
Due to the low level of education of part of the sample studied, the CCQ was administered by interview. As expected, the completion of the CCQ by interview in the present study was slightly longer than that of the original self-administered CCQ reported in the original study (approximately 2 min).(7) Agreement between the self-administered CCQ scores and clinician-administered CCQ scores obtained during a medical visit, as well as between self-administered CCQ scores and clinician-administered CCQ scores obtained through semi-structured, in-depth interviews, have been reported.(36) However, this was the first study to analyze measurement properties of the CCQ completed by interview, with no interference from the raters. A meta-analysis(37) reported that, in general, the self-completion and assisted completion of a PROM produce equivalent scores, supporting that the interview format is a valid mode of administration. Moreover, in the present study, assisted completion of the CCQ allowed the unprecedented analysis of interrater reliability and measurement error.
 
The time interval between the administrations of the CCQ by the raters might have been too short to avoid recall bias and, therefore, might have compromised interrater reliability. However, as far as we know, there is not a recommendation regarding an appropriate time interval for the application of a PROM by raters in the literature. In addition, although the 95% CI range between raters was shorter than it was between test and retest, interrater and test-retest ICC3,1 were similar and higher than 0.70.(9,29) Another possible limitation of the study was the cross-sectional design, which prevented the sufficiency of the measurement error from being tested. However, this was the first study that reported the values of SEMagreement, SDCindividual, SDCgroup, and LoA of the Brazilian version of the CCQ. These values reveal that the changes are real, and not due to measurement error, by showing how much the score needs to change before ensuring that a real change has occurred, providing conditions to interpret longitudinal measurements. In addition, the values of SEMagreement, SDCindividual, SDCgroup, and LoA are presented in the same measurement unit of the PROM being studied, which facilitates the interpretation of the scores by health professionals in clinical practice.(6)
 
In conclusion, the Brazilian version of CCQ has sufficient internal consistency and reliability, that is, the PROM items are interrelated, and their scores are stable and capable of reproducing consistent results in repeated measures between different raters and over time. In addition, the Portuguese version of the CCQ for use in Brazil demonstrates sufficient construct validity, and the correlations between the CCQ total scores and the SGRQ scores are consistent. In the present study, SEMagreement, SDCindividual, SDCgroup, and LoA parameters of the measurement error were shown. However, it is recommended that further studies be conducted to test the sufficiency of measurement error by calculating the MIC. No floor or ceiling effects in the total score of the Brazilian version of the CCQ were found. To our knowledge, this was the first study to evaluate the measurement properties of the CCQ in a sample of patients with COPD in Brazil, which contributes to disseminating this PROM to and promoting its use by health professionals and researchers in order to assess the health status of their patients.
 
AUTHOR CONTRIBUTIONS
 
AR, FRF, and RM: conception and planning of the study; interpretation of evidence; drafting and revision of preliminary and final versions; and approval of the final version. APQ: drafting and revision of preliminary and final versions; and approval of the final version. CMR: conception and planning of the study; interpretation of evidence; and approval of the final version. MMB: revision of preliminary and final versions; and approval of the final version. JK and TvdM: drafting and revision of preliminary and final versions; and approval of the final version.

1. Soler-Cataluña JJ, Alcázar-Navarrete B, Miravitlles M. The concept of control of COPD in clinical practice. Int J Chron Obstruct Pulmon Dis. 2014;9:1397-1405. https://doi.org/10.2147/COPD.S71370


2. Vestbo J, Hurd SS, Agustí AG, Jones PW, Vogelmeier C, Anzueto A, et al. Global strategy for the diagnosis, management, and prevention of chronic obstructive pulmonary disease: GOLD executive summary. Am J Respir Crit Care Med. 2013;187(4):347-365. https://doi.org/10.1164/rccm.201204-0596PP


3. Tsiligianni IG, van der Molen T, Moraitaki D, Lopez I, Kocks JW, Karagiannis K, et al. Assessing health status in COPD. A head-to-head comparison between the COPD assessment test (CAT) and the clinical COPD questionnaire (CCQ). BMC Pulm Med. 2012;12:20. https://doi.org/10.1186/1471-2466-12-20


4. Cave AJ, Atkinson L, Tsiligianni IG, Kaplan AG. Assessment of COPD wellness tools for use in primary care: an IPCRG initiative. Int J Chron Obstruct Pulmon Dis. 2012;7:447-456. https://doi.org/10.2147/COPD.S29868


5. Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010;19(4):539-549. https://doi.org/10.1007/s11136-010-9606-8


6. de Vet HCW, Terwee CB, Mokkink LB, Knol DL. Measurement in Medicine: A Practical Guide (Practical Guides to Biostatistics and Epidemiology). Cambridge: Cambridge University Press; 2011. https://doi.org/10.1017/CBO9780511996214


7. van der Molen T, Willemse BW, Schokker S, ten Hacken NH, Postma DS, Juniper EF. Development, validity and responsiveness of the Clinical COPD Questionnaire. Health Qual Life Outcomes. 2003;1:13. https://doi.org/10.1186/1477-7525-1-13


8. Lourenço RA, Veras RP. Mini-Mental State Examination: psychometric characteristics in elderly outpatients. Rev Saude Publica. 2006;40(4):712-719. https://doi.org/10.1590/S0034-89102006000500023


9. Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60(1):34-42. https://doi.org/10.1016/j.jclinepi.2006.03.012


10. Miller MR, Hankinson J, Brusasco V, Burgos F, Casaburi R, Coates A, et al. Standardisation of spirometry. Eur Respir J. 2005;26(2):319-338. https://doi.org/10.1183/09031936.05.00034805


11. Pereira CA, Sato T, Rodrigues SC. New reference values for forced spirometry in white adults in Brazil. J Bras Pneumol. 2007;33(4):397-406. https://doi.org/10.1590/S1806-37132007000400008


12. Vogelmeier CF, Criner GJ, Martinez FJ, Anzueto A, Barnes PJ, Bourbeau J, et al. Global Strategy for the Diagnosis, Management, and Prevention of Chronic Obstructive Lung Disease 2017 Report: GOLD Executive Summary [published correction appears in Eur Respir J. 2017 Jun 22;49(6):]. Eur Respir J. 2017;49(3):1700214. https://doi.org/10.1183/13993003.00214-2017


13. Silva GP, Morano MT, Viana CM, Magalhães CB, Pereira ED. Portuguese-language version of the COPD Assessment Test: validation for use in Brazil. J Bras Pneumol. 2013;39(4):402-408. https://doi.org/10.1590/S1806-37132013000400002


14. Kovelis D, Segretti NO, Probst VS, Lareau SC, Brunetto AF, Pitta F. Validation of the Modified Pulmonary Functional Status and Dyspnea Questionnaire and the Medical Research Council scale for use in Brazilian patients with chronic obstructive pulmonary disease. J Bras Pneumol. 2008;34(12):1008-1018. https://doi.org/10.1590/S1806-37132008001200005


15. Camelier A, Rosa FW, Salim C, Nascimento OA, Cardoso F, Jardim JR. Using the Saint George’s Respiratory Questionnaire to evaluate quality of life in patients with chronic obstructive pulmonary disease: validating a new version for use in Brazil. J Bras Pneumol. 2006;32(2):114-122. https://doi.org/10.1590/S1806-37132006000200006


16. Fan X, Thompson B. Confidence Intervals for Effect Sizes: Confidence Intervals about Score Reliability Coefficients, Please: An EPM Guidelines Editorial. Educ Psychol Meas. 2001;61(4):517-531. https://doi.org/10.1177/0013164401614001


17. McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychol Methods. 1996;1(1):30-46. https://doi.org/10.1037/1082-989X.1.1.30


18. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307-310. https://doi.org/10.1016/S0140-6736(86)90837-8


19. Hazra A, Gogtay N. Biostatistics Series Module 6: Correlation and Linear Regression. Indian J Dermatol. 2016;61(6):593-601. https://doi.org/10.4103/0019-5154.193662


20. Damato S, Bonatti C, Frigo V, Pappagallo S, Raccanelli R, Rampoldi C, et al. Validation of the Clinical COPD questionnaire in Italian language. Health Qual Life Outcomes. 2005;3:9. https://doi.org/10.1186/1477-7525-3-9


21. Reda AA, Kotz D, Kocks JW, Wesseling G, van Schayck CP. Reliability and validity of the clinical COPD questionniare and chronic respiratory questionnaire. Respir Med. 2010;104(11):1675-1682. https://doi.org/10.1016/j.rmed.2010.04.023


22. Ställberg B, Nokela M, Ehrs PO, Hjemdal P, Jonsson EW. Validation of the clinical COPD Questionnaire (CCQ) in primary care. Health Qual Life Outcomes. 2009;7:26. https://doi.org/10.1186/1477-7525-7-26


23. Papadopoulos G, Vardavas CI, Limperi M, Linardis A, Georgoudis G, Behrakis P. Smoking cessation can improve quality of life among COPD patients: validation of the clinical COPD questionnaire into Greek. BMC Pulm Med. 2011;11:13. https://doi.org/10.1186/1471-2466-11-13


24. Antoniu SA, Puiu A, Zaharia B, Azoicai D. Health status during hospitalisations for chronic obstructive pulmonary disease exacerbations: the validity of the Clinical COPD Questionnaire. Expert Rev Pharmacoecon Outcomes Res. 2014;14(2):283-287. https://doi.org/10.1586/14737167.2014.887446


25. Berkhof FF, Metzemaekers L, Uil SM, Kerstjens HA, van den Berg JW. Health status in patients with coexistent COPD and heart failure: a validation and comparison between the Clinical COPD Questionnaire and the Minnesota Living with Heart Failure Questionnaire. Int J Chron Obstruct Pulmon Dis. 2014;9:999-1008. https://doi.org/10.2147/COPD.S66028


26. Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737-745. https://doi.org/10.1016/j.jclinepi.2010.02.006


27. Cortina JM. What is coefficient alpha? An examination of theory and applications. J Appl Psychol. 1993;78(1):98-104. https://doi.org/10.1037/0021-9010.78.1.98


28. Scholtes VA, Terwee CB, Poolman RW. What makes a measurement instrument valid and reliable?. Injury. 2011;42(3):236-240. https://doi.org/10.1016/j.injury.2010.11.042


29. Prinsen CAC, Mokkink LB, Bouter LM, Alonso J, Patrick DL, de Vet HCW, et al. COSMIN guideline for systematic reviews of patient-reported outcome measures. Qual Life Res. 2018;27(5):1147-1157. https://doi.org/10.1007/s11136-018-1798-3


30. Hasanpour N, Attarbashi Moghadam B, Sami R, Tavakol K. Cross-Cultural Adaptation, Reliability and Validity Study of the Persian Version of the Clinical COPD Questionnaire. Acta Med Iran. 2016;54(8):518-524.


31. Bland JM, Altman DG. Comparing two methods of clinical measurement: a personal history. Int J Epidemiol. 1995;24 Suppl 1:S7-S14. https://doi.org/10.1093/ije/24.Supplement_1.S7


32. Kocks JW, Tuinenga MG, Uil SM, van den Berg JW, Ståhl E, van der Molen T. Health status measurement in COPD: the minimal clinically important difference of the clinical COPD questionnaire. Respir Res. 2006;7(1):62. https://doi.org/10.1186/1465-9921-7-62


33. Canavan JL, Dilaver D, Clark AL, Jones SE, Nolan CM, Kon SS, et al. Clinical COPD Questionnaire in patients with chronic respiratory disease. Respirology. 2014;19(7):1006-1012. https://doi.org/10.1111/resp.12350


34. Smid DE, Franssen FM, Houben-Wilke S, Vanfleteren LE, Janssen DJ, Wouters EF, et al. Responsiveness and MCID Estimates for CAT, CCQ, and HADS in Patients With COPD Undergoing Pulmonary Rehabilitation: A Prospective Analysis. J Am Med Dir Assoc. 2017;18(1):53-58. https://doi.org/10.1016/j.jamda.2016.08.002


35. Alma HJ, de Jong C, Jelusic D, Wittmann M, Schuler M, Kollen BJ, et al. Assessing health status over time: impact of recall period and anchor question on the minimal clinically important difference of copd health status tools. Health Qual Life Outcomes. 2018;16(1):130. https://doi.org/10.1186/s12955-018-0950-7


36. Kocks JW, Kerstjens HA, Snijders SL, de Vos B, Biermann JJ, van Hengel P, et al. Health status in routine clinical practice: validity of the clinical COPD questionnaire at the individual patient level. Health Qual Life Outcomes. 2010;8:135. https://doi.org/10.1186/1477-7525-8-135


37. Rutherford C, Costa D, Mercieca-Bebber R, Rice H, Gabb L, King M. Mode of administration does not cause bias in patient-reported outcome results: a meta-analysis. Qual Life Res. 2016;25(3):559-574. https://doi.org/10.1007/s11136-015-1110-8