Psychometric Validation of the Speed of Change in Feeling of Health (SCFH) Questionnaire: Findings from the NEWTON Sub-Study in Italian Asthma Patients

Introduction

Asthma is a chronic inflammatory disease of the airways affecting more than 300 million people worldwide.¹ Due to its individual and societal burden, it is considered a major public health problem.^1,2

The aim of asthma treatment is to control the disease and prevent the exacerbations.^1–3 These goals can be achieved with the available treatments for most adherent patients.⁴ However, real-world observation shows that long-term adherence is poor and influenced by several factors, including medication-related issues (complex regimens, side effects, etc.), a lack of perceived need for therapy when symptoms are absent, and an inadequate patient-physician communication.^5,6 Identifying patients’ perceptions of health improvement can enhance their engagement in the treatment plans.^7–9 This assessment is also valuable for healthcare professionals to define the best treatment schedule.^10,11

However, there is a lack of validated tools that assess what changes patients perceive in terms of body sensations, thoughts, and emotions.¹² Moreover, as patients might easily feel inundated with excessive surveying, Patient-Reported Outcomes (PROs) must remain concise and efficient in evaluating relevant, actionable outcomes.¹³ This study aimed to develop and validate a simple and psychometrically sound questionnaire, the “Speed of change in feeling of health” (SCFH), that allows for assessing the changes in feeling of health from the start of a new treatment. The questionnaire was developed and validated in the context of the NEWTON study.¹⁴ The questionnaire asked patients to express their feelings about their illness since beginning treatment with beclometasone dipropionate (BDP)/formoterol fumarate (FF) 100/6 μg fixed combination administered via a DPI (NEXThaler^®). More specifically, it captured the patient’s perception of health at each visit.

As all questions were intended to capture changes in patient perception over time, we assessed whether patients reported any change in their condition at various time points. Thus, by administering the questionnaire at 7 (±1), 14 (±2), and 30 (±3) days after treatment start, we were able to monitor the speed of any changes effectively.

To optimize its psychometric properties, improve clarity, and minimize participant burden, redundant items of SCFH questionnaire were removed and a new simplified 8-item version was used in the validation process.^13,15 This paper reports the method of item reduction and the psychometric validation of the 8-item SCFH.

Materials and Methods

Study Design

A substudy involving patients from the Italian sites of the NEWTON trial (NCT05168995) was conducted to psychometrically validate the SCFH questionnaire (primary objective of the substudy).¹⁴ Enrolled patients were asked to complete the SCFH and other validated PROs tools at 7 (±1), 14 (±2), and 30 (±3) days after initiating treatment with extrafine BDP/FF NEXThaler^® 100/6 μg. The substudy ended on day 30 of the observation period of the last subject enrolled. The subjects completed PROs using their electronic devices.

At enrollment, subjects were asked to fill out the Stanford Expectation of Treatment Scale (SETS), which is a six-item questionnaire used to detect patients’ positive and negative expectancy effects on treatment outcomes.¹⁶ The SETS is structured into two distinct subscales, which separately assess positive expectations (eg, anticipated benefits) and negative expectations (eg, concerns or doubts about treatment). The Asthma Control Questionnaire-5 items (ACQ-5), used to measure asthma control, was completed by subjects at the beginning of the study and during the 30 days of the observational period.^14,17 The Global Rating Scale (GRS), a standard general scale (ranging from −3 to +3) to assess the current improvement in perceived global health status compared to the day the new treatment started, was completed at 7 (±1), 14 (±2), and 30 (±3) days after enrollment.¹⁸

Subjects

The sample size was determined according to the feasibility criteria. Considering the number of patients managed by the Italian centers involved in the main study, the inclusion of 200 subjects was deemed reasonable for the planned enrolment period according to methodological standards. In the contest of the main European study, the competitive enrollment was allowed to ensure the recruitment of the overall population; therefore, 270 subjects affected by not-well-controlled or poorly controlled asthma were recruited at 19 Italian Respiratory Medicine Centers over 15 months.¹⁹ To assess the precision of convergent validity estimates for the questionnaire, simulations were conducted using assumed effective sample size of 100–150 and correlation coefficients (ρ = 0.75, 0.50, 0.30), reflecting strong, moderate, and weak validity, respectively (Supplementary Methods S1). Details about inclusion and exclusion criteria and study objectives were reported previously.¹⁹ Subjects who discontinued treatment with BDP/FF NEXThaler^® 100/6 μg before the 30th day of observation were also excluded.

The study was performed according to the Declaration of Helsinki, Good Pharmacoepidemiology Practices (GPP), and applicable regulatory requirements and was approved by all participating institutions’ ethics committees before enrolment and data collection.²⁰

Participants provided written informed consent for enrolment in the study.

Endpoints

The primary endpoint of the substudy was a psychometric validation of the SCFH questionnaire by assessing properties such as the convergent and construct (known-group) validity, the reliability (internal consistency), and the minimal clinically important difference (MCID). The secondary endpoint was the rate of patients perceiving a clinically significant change of the SCFH questionnaire at 7(±1), 14 (±2), and 30 (±3) days after the first administration of BDP/FF NEXThaler^® 100/6 μg.

“Speed of Change in Feeling of Health” (SCFH) Questionnaire

The SCFH questionnaire aims to rate how quickly the subjects perceive any changes in their feeling of health from the start of a new treatment. A preliminary list of items was generated based on key phrases from physicians, psychologists, and patients. The provisional SCFH consisted of 30 items (SCFH-30) and was developed in Italian (Supplementary Table S1). Each item is scored on a 5-point scale, representing different grades of agreement based on clinical considerations (0 = strongly disagree, 1 = quite disagree, 2 = neither agree nor disagree, 3 = quite agree, 4 = strongly agree). More in detail, 0 score was assigned to all negative responses (answers 0, 1, and 2), 1 to “quite agree” response (answer 3), and 2 to “strongly agree” positive response (answer 4). The final score is derived by summing the individual item scores. Therefore, subjects obtaining higher scores are those who perceive a greater change in their feeling of health. Missing answers to the SCFH questionnaire items were not imputed or replaced.

Validation Process and Application

The validation process involved three phases: (1) Internal consistency was assessed to ensure the reliability of the questionnaire; (2) Validity was evaluated by comparing the scores of the questionnaire with those of the SETS, ACQ-5, and GRS; (3) The MCID was calculated to determine the smallest change in questionnaire scores that participants consider clinically meaningful.

Reliability

To optimize its psychometric properties, improve clarity, and minimize participant burden, we first evaluated whether redundant items were present in the 30-item SCFH. An internal consistency analysis of the SCFH-30 questionnaire was performed by calculating the Cronbach’s α values overall and for each item to remove redundancy (values >0.70 were considered acceptable).²¹ Values >0.90 indicate that the items measure the same feature without adding new information.^22,23 If redundant items were found, they were removed and Cronbach’s α was recalculated on the remaining items. To evaluate construct validity, the principal component method with Varimax rotation was adopted on the SCFH-8 tool.

Validity

Construct (Known-Group) Validity

Known-group validity is the ability of a questionnaire to be sensitive to differences between two groups. Known-group validity was assessed by calculating the differences between the median (25th–75th percentile) SCFH questionnaire scores for the two patient groups: those classified as “improved” and those categorized as “not improved or worsened”. This classification was based on the ACQ-5 scores and consistently used in the main study, which classified “improved” patients as those with an ACQ-5 score ≤0.75 and those “not improved or worsened” with an ACQ-5 score higher than 0.75.¹⁴ The Kruskal–Wallis/Wilcoxon test for computed differences was used at a significance level of α=0.05. Specific hypotheses were not defined a priori.

Convergent Validity

Convergent validity examines whether a new instrument correlates with existing validated tools which evaluate related constructs.²⁴ The Spearman correlation coefficients (with respective p-values; significance level of α=0.05) were used to determine a linear correlation between the scores of the SCFH questionnaire and the other PROs (SETS, ACQ-5, and GRS) at 7 (±1), 14 (±2), and 30 (±3) days after enrollment. As the SETS assesses positive and negative expectations, correlations were calculated independently between the SCFH and the SETS positive and negative subscales.

Considering the meaning of each scale/subscale, correlations coefficients were expected to be positive for SETS positive expectation subscale and GRS, and negative for SETS negative expectation subscale and ACQ-5.

The rationale for including expectancy was that patients’ expectations are known to influence health-related outcomes and perceived benefit from treatment. Therefore, we aimed to explore whether both of these constructs converge with the dimensions captured by the SCFH-8 questionnaire.

Minimal Clinically Important Difference (MCID)

An anchor-based method was implemented to calculate the MCID value, which is defined as the smallest change in the questionnaire scores considered clinically meaningful by the subjects.²⁵ The anchor was based on the GRS answers: “responder patients” were defined as subjects who scored the GRS >0 points, while “non-responder patients” as those who scored ≤0 points.

The MCID value was calculated at the first time point of completion of the SCFH as the difference between the mean score of “responder” and “non-responder”. The number of patients who reached the MCID in SCFH at each time point (day 7(±1), 14 (±2), and 30 (±3)) was calculated.

Statistical Analyses

Analyses were performed using SAS Enterprise Guide v. 8.2 and SAS 9.4 (SAS Institute, Cary, NC, USA). Study design and conduct, data monitoring, eCRF set-up, and statistical analyses were performed by Medineos SRL (Modena, Italy), a company merged into IQVIA Solutions Italy srl, on behalf of Chiesi Italia S.p.A.

Results

Population

196 of the 270 subjects (72.6%) were eligible to be evaluated in the substudy. Seventy-four subjects were excluded from the study for the following reasons: 12 subjects were not eligible according to further inclusion/exclusion criteria described by Braido et al;¹⁹ 2 subjects did not complete the questionnaire within the specified time windows, and 60 subjects did not complete the questionnaire at any available time point for assessment. The median age of the evaluated population was 45.0 years (interquartile range [IQR]: 29.0–59.0), 39.8% were male (n = 78), and 60.2% were female (n = 118), with a median time from diagnosis of 9.4 (0.4–24.5) years. The median ACQ-5 score at enrollment was 1.6 (0.8–2.8) (Table 1).

Table 1 Demographic and Clinical Characteristics at Enrolment

Reliability

The internal consistency analysis resulted in a high standardized Cronbach’s alpha (0.98), which indicates redundancy, ie, several items were capturing the same information. A Cronbach’s alpha value between 0.70 and 0.90 is recommended to ensure the reliability of the test.^21–23 The selection of items based on high inter-item correlation and clinical relevance led to the definition of an 8-item questionnaire (questions: 1, 2, 3, 7, 8, 16, 29, and 30). A further internal consistency analysis was conducted on these 8 items and resulted in a standardized Cronbach’s alpha of 0.90. The principal component analysis showed a single-factor structure in the shortened version of the SCFH-8 questionnaire: one latent factor explained 58% of the overall variance (Figure 1, Table 2).

Table 2 Latent Factors Dimension of the “Speed of Change in Feeling of Health” (SCFH) Questionnaire Domains (8 Items) (SCFH-8)

Figure 1 Scree plots (Validation set).

Validity

Construct (Known-Group) Validity

In subjects with improved condition, focusing on 30 (±3) days after enrollment, the median value was 6.0 (2.0–8.0); at the same time point in subjects with deteriorated or unchanged condition, it was 3.0 (1.0–6.0) (Table 3). The difference between the two groups was statistically significant (p = 0.0003).

Table 3 “Speed of Change in Feeling of Health” (SCFH-8) Questionnaire Known-Group Validity at 30 (±3) Days After Enrollment

The same evaluations on the known-group validity were valid for the other study time points (data not shown).

Convergent Validity

The convergent validity analysis, focusing on 30 (±3) days after enrollment, found a weak negative correlation between the total score in the SCFH-8 questionnaire and the SETS negative total score (r=−0.17, p = 0.0596). In contrast, there was a weak positive correlation between the total score of the SCFH-8 questionnaire and the SETS positive total score at enrolment (r = 0.22, p = 0.0151) (Table 4), while SCFH-8 total score was negatively correlated with the ACQ-5 total score (r=−0.38, p<0.0001) (Table 4). Finally, the SCFH-8 total score and GRS total score were positively correlated (r = 0.70, p<0.0001) (Table 4).

Table 4 Convergent Validity Analysis at 30 (±3) Days After Enrolment

Descriptives of SCFH-8 Questionnaire

The median interquartile range (IQR) of the SCFH-8 is 3.0 (1.0–6.0) on day 7(±1), 3.0 (1.0–7.0) on day 14 (±2), and 4.0 (1.0–8.0) on day 30 (±3) (Table 5).

Table 5 Speed of Change in Feeling of Health Questionnaire Score at Each Time Point

Minimal Clinically Important Difference of the SCFH-8 Questionnaire

The MCID value calculated at 7 (±1) days after enrollment was 2.9. A change of 3 points (ie, the minimum value obtained at 7 days) was considered the minimum change that patients could perceive as clinically relevant. In the validation set (n = 196), 43.4% of patients (n = 85) reported a clinically significant change in SCFH-8 at day 7 (±1), 18.4% (n = 36) at day 14 (±2), and 8.7% (n = 17) at day 30 (±3) after enrollment (Table 6). Consequently, the cumulative frequency of patients reporting a clinically relevant change in their feeling of health was 43.4% (n = 85) at day 7 (±1), 61.7% (n = 121) at day 14 (±2), and 70.4% (n = 138) at day 30 (±3). For 58 (29.6%) patients, a clinically relevant change was not reported at any of the time points (Table 6).

Table 6 Time to Achieve a Perceived Clinically Relevant Change for the First Time

Discussion

The SCFH-30 questionnaire was developed to evaluate the patient’s perceived changes in their health in terms of bodily sensations, thoughts, and emotions, as well as the speed at which these changes occurred, factors that existing PROs do not address. The NEWTON study provided the framework for validating this new tool.¹⁴ A two-step internal consistency analysis allowed for condensing the provisional SCFH-30 questionnaire into 8-item tool (SCFH-8), demonstrating its ability to reliably measure the construct of interest. The SCFH-8 does not primarily focus on symptoms reported by patients but rather on their perception of general health and confidence in the prescribed treatment. Moreover, it is sensitive to changes in a short time period (namely 1 week), therefore by administering the questionnaire at short, successive intervals, it is possible to assess the speed with which the change is perceived after treatment initiation. This approach may allow a better understanding of how patients experience changes in their health, thereby supporting engagement in the treatment plan, facilitating physicians’ decision-making, and enabling evaluation of treatment impact.^26–28

Compared with other PROs, the SCFH-8 offers several advantages. A single-item global assessment such as the GRS would not allow subjects to easily recognise and record changes in their perception of physical and psychological health, as the SCFH-8 does. Additionally, the brevity of the 8-item version makes it suitable for repeated use in real-world settings. The global rating of change (GRC) scales asks patients to assess not only their current health status but also recalls their status at a previous time point. Several studies have pointed out the difficulties for patients in accurately recalling prior health states, thus reducing the reliability of GRC tools.^29,30 Furthermore, GRC scales allow patients to rate improvement based on whichever aspect of health they find most relevant. As patients may base their ratings on different factors (eg, pain, function, quality of life, and side effects), this variability complicates construct validity.^18,31

The SCFH-8 assessed the patient perception shortly after the start of treatment, limiting the risk of recall bias.

Moreover, the SCFH-8 tool has a strong construct (known-group) validity as demonstrated by a statistically significant difference between the two groups “improved” and “not improved or worsened”, according to ACQ-5 score changes. This indicates that the SCFH-8 is sensitive enough to discriminate between improvement and worsening.

The negative correlation between the SCFH-8 questionnaire score and both the SETS negative total score and the ACQ-5 score suggested that subjects reporting faster improvements in their health (ie, higher SCFH scores) also have better asthma control (ie, lower ACQ-5 scores) and fewer negative expectations about their treatment outcomes (ie, lower scores on the SETS negative scale), which predict negative responses to treatment. The weak positive correlation with the SETS positive total score suggests that subjects who perceive health status improvements also have more positive expectations about their treatment outcomes (ie, higher SETS positive scores).

The MCID value at 7 (±1) days after starting BDP/FF NEXThaler^® 100/6 μg was 3 points, corresponding to the smallest change perceived as beneficial. This threshold was reached by 43.4% of subjects at 7 days from the initiation of treatment, with a further 18.4% and 8.7% of subjects reaching the MCID at day 14 (±2) and day 30 (±3). Only 29.6% of patients did not perceive any improvement in their feeling of health up to 30 days after enrolment.

The patient’s perceptions at the start of a new treatment may influence their adherence and persistence. Previous studies reported a positive correlation between patient satisfaction, improved clinical outcomes, and treatment adherence. Patients who report positive scores on questionnaires measuring treatment satisfaction and asthma control tend to adhere more to therapies.^32–34 Therefore, the validation of the SCFH-8 questionnaire can significantly improve clinical practice for asthma in patients starting a new treatment.

Some limitations need to be considered: (1) The participants enrolled do not represent the entire population of subjects affected by asthma; (2) Participation in the study alone could influence the patient’s perception of their medical care or clinical condition; (3) The subjects filled in the full 30-item questionnaire, but only the selected 8 items were analyzed for the validation. Consequently, to minimize the data collection effort on both subjects and centers, the validated SCFH-8 questionnaire was not resubmitted, given the observational nature of the study. However, since the 22 excluded questions were redundant, any potential interpretation bias can be considered negligible; 4) As mentioned in the methods, the SCFH-8 tool assesses patients’ health perception at 7, 14, and 30 days after treatment start. Therefore, the retrospective recall of prior states can be considered limited, and the risk of recall bias is constrained. Although the selected time points are very close to the start of treatment, these time intervals are considered sufficiently long to observe a change in the patient’s perception of their condition. In this study, the SCFH questionnaire was validated in subjects with asthma. However, since the questions formulated for the questionnaire are generic and not precisely related to asthma, this tool could potentially be used in other healthcare contexts and treatments.

Conclusion

This substudy allowed for the successful validation of a questionnaire to assess subjects’ improvement in feeling of health after the initiation of a new treatment.

The evaluation of the psychometric properties of the final 8-item version of the questionnaire demonstrated that the SCFH-8 questionnaire is a valid and reliable tool for capturing changes in feelings of health. In everyday practice, the SCFH-8 can complement the established tools, particularly during early follow-up visits, to assess how quickly patients perceive an improvement after starting treatment and help physicians interpret the treatments.