INSIGHTS Asthma Pragmatic Registry – A Pragmatic Approach to High-Validity Real-World Evidence for Asthma

Introduction

Asthma is a complex respiratory disease affecting approximately 7.7% of people living in the US.¹ The heterogeneity of the disease results in varying clinical presentations, pathobiology, biomarkers, treatment management and response, and outcomes.^2–6 Patients with severe asthma, defined as asthma requiring maximal optimized high-dose inhaled corticosteroids (ICS) in combination with longer-acting medications or uncontrolled asthma despite use of these medications, make up approximately 10% of all asthma patients.^7,8 Current therapy for severe disease includes inhaled corticosteroids (ICS) plus long-acting bronchodilators with or without oral glucocorticoids. When disease remains uncontrolled, six biologics are approved for add-on treatment for severe asthma in the United States (US), including those indicated for eosinophilic asthma (benralizumab,⁹ mepolizumab,¹⁰ and reslizumab),¹¹ eosinophilic asthma or corticosteroid-dependent asthma (dupilumab),¹² allergic asthma (omalizumab),¹³ and asthma regardless of biomarker or phenotype (tezepelumab).¹⁴

Industry-sponsored randomized controlled trials (RCTs), while essential to establishing efficacy and safety of treatments, have restrictive eligibility criteria, and thus limited generalizability in the real-world setting. As diseases and healthcare decisions become increasingly nuanced, real-world evidence (RWE) becomes an essential means of augmenting evidence generated via RCTs. However, not every type of real-world data (RWD) source may be fit-for-purpose to answer clinical questions. For instance, relying solely on structured electronic health records (EHRs) or administrative claims to identify severe asthma patients can be problematic due to inconsistent capture of disease severity codes.¹⁵ Obtaining a comprehensive understanding of the patient journey can also be problematic when there is limited capture of data on symptoms, pulmonary function tests, and other clinical outcomes relevant to asthma.

The inventory of severe asthma registries is growing globally,¹⁶ with approximately 70% of the 37 identified registries being affiliates of the International Severe Asthma Registry (ISAR),¹⁷ currently the largest with over 21,000 adult patients with severe asthma. In the US, CHRONICLE¹⁸ and Severe Asthma Research Program (SARP)¹⁹ are non-ISAR affiliated severe asthma registries enrolling patients across health systems. These traditional registries involve enrollment of consented patients, thus enabling primary collection of data tailored to specific research needs, such as patient-reported outcomes. However, they can have limited scalability (doubling patient count often doubles cost), flexibility (added variables can take years to populate), and efficiency (reaching enrollment targets can require several years).^20–22 There can also be sampling bias resulting from consent requirements.^23,24

An alternative option is a real-world pragmatic registry that leverages documentation of routine care while organizing retrospective data in a standardized format for a population. A pragmatic registry offers certain advantages over a consented registry, such as enabling better representativeness, larger sample sizes, flexibility to reuse infrastructure, and more rapid adaptation to evolving research needs. Whereas data are primarily collected with strict quality controls in a consented registry, routine data are used in a pragmatic registry. Thus, ensuring data quality is paramount.

Fortunately, data quality standards have been clarified in recent years. The Food and Drug Administration (FDA) published guidance titled “Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision Making for Drug and Biological Products”, calling out accuracy, completeness, and traceability as critical aspects of data reliability.²⁵ The life sciences industry, payers, and providers are increasingly adopting these standards. Thus, for a pragmatic registry to be considered scientifically comparable and a credible alternative to traditional registries, it should have high accuracy, completeness, and traceability.

The INSIGHTS Asthma Pragmatic Registry is implemented as a uniquely rich and high-reliability data set using routinely collected data for epidemiology and clinical research applications in asthma. Leveraging data from health systems in the US, the registry combines artificial intelligence (AI)-based extraction from EHR unstructured data and linkage of the EHR records to pharmacy claims, medical claims, and death registry data on every patient to achieve high levels of accuracy, completeness, and traceability. The purpose of this manuscript is to describe the pragmatic registry design including specific efforts to meet data quality standards and to characterize currently selected patients diagnosed with moderate-to-severe asthma.

Materials and Methods

Design and Setting

The INSIGHTS Asthma Pragmatic Registry is a real-world, retrospective, noninterventional cohort of patients with moderate-to-severe asthma. Patients are identified from a distributed collection of academic and community health systems, spanning over 65 hospitals and approximately 1,600 clinics, covering over 38 million patient visits across different geographies in the US. The registry will continue to expand to additional health systems.

Data Sources

The pragmatic registry includes data from multiple data sources, including EHR data from integrated health systems, medical and pharmacy claims data from claims aggregators, and mortality data from death registry vendors. The EHR data from unstructured notes are extracted using robust AI technology, which includes natural language processing and machine learned inference as detailed in prior work.^15,26 EHR data for all selected asthma patients are linked to adjudicated claims and death registry data via tokenization. Details on each data source are summarized below:

• EHR structured: Lists such as problem lists, procedure lists, medication lists, unadjudicated claims, and other captured data such as labs.
• EHR unstructured: Primary care notes, specialty notes, radiology and pathology notes, and other narrative content.
• Claims: All adjudicated pharmacy (medication), provider, and facility billing claims.
• Death registry: Data on deaths from any cause sourced from government, private, and historical sources including death certificates, person-reporting, funeral homes, newspapers, and online obituaries. Death data are considered to be sufficiently tracked if at least one year has passed since the last observation date to account for the expected lag in updating the death registry.

Target Population

Patients meeting eligibility criteria for moderate-to-severe asthma (see Table 1) since January 1, 2014, to the most current data available from data sources are selected. Patients are either required to have a receipt of a medium-to-high dose ICS²⁷ plus an additional controller medication or required to have documentation of an explicit diagnosis of severe asthma by their physician. The index date is the earlier date of meeting either criterion. While the focus of the pragmatic registry is severe asthma, the specified criteria are broad enough to include patients who may have moderate disease at the time of pragmatic registry inclusion, thus enabling the study of disease progression. Additionally, the broad inclusion criteria provide users greater flexibility in defining specific study populations of interest. The data set is refreshed every quarter, with each refresh including additional longitudinal data for existing participants as well as new patients that meet criteria.

Table 1 Eligibility Criteria for the INSIGHTS Asthma Pragmatic Registry

Data Reliability

The INSIGHTS Asthma Pragmatic Registry is the first pragmatic registry to not only measure but also require high accuracy, completeness, and traceability.

Accuracy

Accuracy is a quantitative assessment of whether extracted data properly reflects patient characteristics. The quality of data extracted from unstructured EHR using advanced technology is assessed against manual chart abstraction (reference standard) for asthma-related concepts in a subset of data.¹⁵ Manual chart abstraction was conducted by two independent clinical annotators who were blinded to each other’s annotations. A minimum inter-rater reliability of 0.70 Cohen’s kappa score was required within the manual reference standard. Discrepancies between annotators or ambiguous language were escalated to a clinical informaticist for review and adjudication. The target accuracy level for high validity is an average F1-score (summary measure of recall and precision) of 0.80 or above. Recall, precision, and F1-score are considered to be important metrics to report in healthcare studies evaluating EHR data quality.²⁸ Recall measures to what extent a concept (such as severe asthma) noted by the physician is also captured by the AI model. Precision measures to what extent a concept identified by the AI model are also noted by the physician. An F1-score of 1 indicates maximum recall and precision, while 0 indicates zero recall and/or precision.²⁹ Hence, values closer to 1 are desirable. Achieving this high standard requires addition of unstructured data and optimization of AI technology specific to asthma and to each provider organization. To ensure a high level of accuracy for the INSIGHTS Asthma Pragmatic Registry, key concepts were required to have an F1-score of 0.80 or greater. Prior work on accuracy that enabled development of the INSIGHTS Asthma Pragmatic Registry as well as ongoing work has been described previously.^15,30 This methodology is also detailed in prior work related to migraine and cardiovascular disease.^26,31,32

Completeness

Completeness is a quantitative assessment of inclusion of multiple views into the patient journey. This requires bringing together multiple data sources for each patient, typically via data linkage. The high standard is achieved by requiring linkage of data from structured EHR, unstructured EHR, medical claim, pharmacy claim, and death registry for all patients within the pragmatic registry. To ensure a high level of completeness for the INSIGHTS Asthma Pragmatic Registry, we required a minimum of 12 months of overlap of EHR structured, EHR unstructured, medical claim, and pharmacy claim data.

Traceability

Traceability is a measure of how well individual data elements are proven correct against a patient-specific source of truth. Traceability is achieved by connecting each data element within the dataset to a source of truth. For example, a reference to a medication must be traced to a pharmacy claim. Achieving a high standard of traceability requires maintenance of all data for a patient within a common system, normalization of coding, and identification of granularity changes such as asthma to severe asthma. Since the majority of clinical information about the patient resides in unstructured notes,^33,34 to ensure a high level of traceability for the INSIGHTS Asthma Pragmatic Registry, we required that unstructured data from EHR visits were available for at least half of all structured EHR visits.

Data Standardization

All data from each data source are transformed into a common data model (CDM) based on the Observational Medical Outcomes Partnership (OMOP) CDM³⁵ to enable standardization of the data structure. Additionally, encoding is normalized to standardized vocabularies.³⁶ A data quality review framework is used for verification of data received from each data source to ensure consistency in data standards.

Data Collection

The pragmatic registry includes routinely collected data for included patients. Data elements include demographics, conditions, visits, procedures, devices, measurements, observations, drug exposures, insurance coverage, site where care was provided, costs of care, and death from any cause. These concepts are defined in OMOP, with the current version being v20240830.³⁵ The OMOP model is extended to capture variables extracted from unstructured text via AI models. Each row of data contains several variables related to extracted concepts, including available information on attributes such as severity of a symptom and whether the symptom is experienced currently.

Privacy and Security

The Pragmatic Registry follows the most rigorous protocols available for secondary use of routinely collected healthcare data. Data are de-identified based on expert determination according to 45 C.F.R.§ 164.514: Expert Determination and Safe Harbor prior to merging data sets or adding data into the pragmatic registry.³⁷ For example, sensitive dates such as date of death are masked to month and year only, and variables such as age are truncated. Tokenization is used for linkage, thus ensuring a one-way matching without the possibility of viewing patient identifiers. Data storage and transmission are compliant with Health Information Trust Alliance (HITRUST) certification. Consistent with HIPAA, informed consent is not required. Additionally, the approach was reviewed by the Salus Institutional Review Board and was granted exemption from the requirements for institutional review board review under 45 CFR 46.104(d)(4(ii)).

Data Analysis

Data elements summarized include patient demographic and clinical characteristics, provider type, labs, and follow-up time. Age and data on biomarkers, including blood eosinophil and total immunoglobulin E (IgE) are summarized based on value at or date closest prior to index, as well as observations throughout study period. The baseline period includes the 12 months prior to and including index date, and follow-up period starts one day after index and ends on the earliest of dataset end (August 1, 2024), last observation, or date of death.

Continuous variables are summarized using mean and median. Categorical variables are summarized using frequency distributions. To contextualize with a US-based and a global registry, a side-by-side description of characteristics of the INSIGHTS Asthma Pragmatic Registry, CHRONICLE,^18,38 and ISAR^17,39,40 are also provided.

Results

Initial Observations

From January 1, 2014, through August 1, 2024, of 29,750 patients with an asthma diagnosis and high-reliability data, defined as data with high accuracy, completeness, and traceability,²⁵ 9,185 (30.9%) patients met pragmatic registry eligibility criteria. On index date, most patients (n = 8,585; 93.5%) had met eligibility criteria through receipt of asthma medication, 523 (5.7%) patients met criteria by having evidence of physician-documented severe asthma, and 77 patients (0.8%) met both criteria on the same date. Baseline characteristics and summary statistics on follow-up are shown in Table 2. Patients had an average 5.4 years and median 6 years of follow-up.

Table 2 Characterization of Patients Included in the INSIGHTS Asthma Pragmatic Registry

The average and median ages of patients at the time of meeting eligibility criteria are 51 and 53 years, respectively, with the majority (61.2%) of patients in the age group 40–69 years. Patients are mostly female (65.7%) of white race (65.5%), and non-Hispanic ethnicity (73.5%). Approximately one-fifth of the patients are ever smokers.

Data Reliability (Accuracy, Completeness, and Traceability)

Data reliability metrics were high across all health systems and are summarized in Table 3. These are stratified by health system, as a proxy for geographic region. Accuracy, as measured by the F1-score, for the concept of severe asthma ranged from 0.80 to 0.90. The median values for traceability, measured as the percentage of structured EHR with corresponding unstructured EHR, ranged from 90% to 100%. The median values for completeness, measured as the number of months of overlap between claims, structured EHR, and unstructured EHR ranged from 37.0 months to 41.7 months. Data reliability metrics for a more comprehensive list of concepts are published in Kilpatrick et al.¹⁵

Table 3 Summary of Data Reliability Metrics for the INSIGHTS Asthma Pragmatic Registry

Summary of Asthma Registries

Design and characteristics of the INSIGHTS Asthma Pragmatic Registry, CHRONICLE, and ISAR are summarized in Table 4. Differences across the registries include variations in design, frequency of data collection, measurement of data reliability, and eligibility criteria. Registries appear to be similar with respect to select characteristics such as data elements and follow-up duration.

Table 4 Characteristics of the INSIGHTS Asthma Pragmatic Registry, CHRONICLE,17,30 and ISAR16,31,32

Discussion

This manuscript describes the ongoing US-based INSIGHTS Asthma Pragmatic Registry as a resource-efficient, yet comparably robust alternative to a traditional consented disease registry of moderate-to-severe asthma patients. Approximately 9,200 patients are currently included with patient numbers continuing to grow each quarter. The pragmatic registry is differentiated from other uses of RWD through substantial efforts to achieve high data reliability²⁵ (including accuracy, completeness, and traceability) and richness, while relying solely on data collected in routine clinical practice. Pragmatic registries may be used independently or combined with primary data collection as required to ensure relevant and reliable data are applied to research questions.

Data Reliability (Accuracy, Completeness, and Traceability)

Accuracy of key clinical variables is increased by measuring and optimizing technology-based extraction from unstructured data. A robust reference standard with minimum inter-rater reliability of 0.70 and high expectations of curated variable accuracy of minimum 0.80 F1-score are among the most rigorous requirements in the industry. Measurement and optimization for each key concept at each local health system represents a critical step to ensure proper application of data reliability.

Completeness is increased by requiring all patients included in the pragmatic registry to have linked EHR (structured and unstructured), medical and pharmacy claims, and death data (as appropriate). The linkage requirement is enhanced to avoid issues with linkage scenarios where a patient’s EHR and claims data are years apart. In the pragmatic registry, while every individual had at least 12 months of overlap in EHR structured, EHR unstructured, medical claim, and pharmacy claim data, 82% of patient-years had two or more data sources and 49% of patient-years had three or more data sources.

Traceability is tested for all variables within the pragmatic registry. For example, a medical claim for asthma is traced back to a clinician’s note documenting asthma or a subtype. Through tracing, errors in linkage, limitations in care coverage for a given patient, and reimbursement-related upcoding for a given condition become immediately obvious. Hence, each data element included in the pragmatic registry is indicated with the source data from which that record originated. This enables calculation of percent of data elements of interest that can be traced back to the source document, thus demonstrating the suitability of the dataset to meet Health and Human Services audit requirements.

Data Richness

Curation, linkage, and enrichment support rich and reliable data, enabling the extraction of variables that are incompletely captured in claims data such as pulmonary function tests (eg, forced expiratory volume in 1 second [FEV1], forced vital capacity [FVC]), as well as variables that are rarely populated such as disease severity and symptoms. Symptoms such as chest tightness are poorly captured in structured data due to lack of reimbursement incentive to track symptoms.¹⁵ Disease subtypes such as severe asthma are poorly captured in structured data due to lack of granularity required in the reimbursement pathway but may be captured either from curation of stated concepts in unstructured data or enrichment based on multiple linked data sources (eg, asthma diagnosis code, high-dose ICS, and lack of control). The pragmatic registry includes findings, symptoms, granular disease subtypes, and other rich clinical content that is typically absent or rarely captured.¹⁵ These can be critical for research questions which may influence clinical care.

Contextualizing with Severe Asthma Registries

Consistent with demographic distributions observed in severe asthma registries,^18,39 patients are predominantly female, white, and of non-Hispanic ethnicity. A differentiating aspect between the INSIGHTS Asthma Pragmatic Registry and traditional severe asthma registries (such as CHRONICLE and ISAR) is the broader eligibility criteria with respect to diagnosing asthma severity. To generate future RWE for heterogeneous asthma clinical subtypes while ensuring sufficient sample sizes for subgroups, eligibility criteria are broad to enrich with patients meeting the definition of severe asthma according to current guidelines,^41,42 as well as patients with less severe disease. These criteria allow researchers the flexibility to define cohorts based on level of clinical confirmation for severe asthma and research question at-hand. Consequently, some patient characteristics may not be directly comparable with other severe asthma registries with stricter criteria (such as requiring diagnosis by a sub-specialist).

Another differentiating factor is the inclusion of adolescent patients aged 12 years and older. In the US, five of the six approved biologics include adolescents in their label, underscoring the need for high-reliability RWE and long-term safety data in both adult and non-adult populations.⁴³ Inclusion of adolescent patients and not requiring that patients are being seen by a subspecialist enable a more comprehensive understanding of prescription patterns and disease management across care settings, which could potentially facilitate systems-wide improvements. Finally, unlike other severe asthma registries, the pragmatic registry is purely observational in nature with no intervention to collect non-routine data.

Strengths and Limitations

The main strength of the INSIGHTS Asthma Pragmatic Registry lies in the combination of high-reliability data with the flexible, scalable, and efficient use of data collected in routine clinical practice. Additionally, the high standards for data reliability are maintained as the registry is expanded to include increased data capture. Patients are selected from a network of academic and community inpatient and outpatient facilities across various geographies in the US and spanning specialist and non-specialist clinics, increasing the generalizability of the asthma patient population in the dataset. The observed longitudinality of the pragmatic registry is comparable to other registries¹⁸ and sufficient to enable long-term treatment safety and effectiveness assessments. The dataset reflects real world care without the biases inherent in selective trial enrollment or registry consent.

Limitations of the pragmatic registry largely reflect the reliance on data collected through routine clinical care. Patients may not visit a physician if they are feeling well and information during that period may not be recorded. Similarly, a physician will not perform every test or document every finding on every encounter. Patients may move into and out of a given health system, resulting in both left and right censoring of the patient journey. Certain clinical outcomes of interest, such as asthma exacerbations, are not explicitly captured or extracted. However, in the absence of such data, algorithms can be applied to claims and/or structured EHR to analytically define exacerbations. Finally, the greatest limitation of the pragmatic registry is a lack of data elements that are not part of routine care and require patient consent. This includes patient-reported outcomes and structured assessments that are not captured at a frequent cadence in practice, such as quality of life measures and Asthma Control Test scores. This limitation, however, makes the pragmatic registry a truly real-world data set.

Conclusion

The INSIGHTS Asthma Pragmatic Registry can meet the demand for clinically rich, highly valid, and timely RWE as the data set is specifically developed to meet data reliability standards while continuing to offer the advantages of leveraging data collected in routine clinical practice. By incorporating the strengths of traditional registries in obtaining clinically nuanced data but doing so in a more scalable and efficient manner, the pragmatic registry presents a new paradigm in generating high validity RWE for clinical and epidemiological research.