Pharma Market Access Insights - from Mtech Access

Best practice RWE approaches to support economic modelling for HTA

• Mtech Access - Powered by Petauri • Season 7 • Episode 22

Share episode

How can real-world evidence (RWE) support health technology assessment (HTA)? Can real-world data (RWD) supplement clinical data? How can RWE be used to solve common challenges with treatment comparison?

Here, Mtech Access are joined by experts from Arcturis and Delta Hat. Dan Howard (Associate Director – Health Economics, Mtech Access) shares some of the challenges that our clients face when developing HTA-ready health economic models with limited clinical trial data. Joseph O’Reilly (Principal Medical Statistician, Arcturis) introduces solutions to these challenges using RWD and RWE approaches. Nick Latimer (Analyst, Delta Hat; Professor of Health Economics, University of Sheffield; former NICE Appraisal Committee member) discusses how RWE is assessed by HTA committees. Samantha Gillard (Director – HTA, Mtech Access) facilitates the discussion and puts your questions to our experts

This episode was first broadcast as a live webinar in October 2024. To request a copy of the slides used or to learn more visit: https://mtechaccess.co.uk/rwe-approaches-economic-modelling-hta/

For support with real-world evidence analysis, health economic modelling of health technology assessment, email info@mtechaccess.co.uk or visit https://mtechaccess.co.uk/

Subscribe to our newsletter to hear more news, insights and events from Mtech Access.

- Well good afternoon everyone. Thank you for joining us this afternoon to discuss the use of real world evidence in HTA submissions. My name's Sam Gillard, I'm the director of HTA here at Mtech Access, and I'm joined this afternoon by colleagues from both Mtech Access, Arcturis and Delta Hat. And we are delighted to sort of talk you through our webinar this afternoon. I'm going to briefly hand over to my colleagues to introduce themselves before we get started with the webinar. So I'm going to hand over to Dan first.- Thanks, Sam. Hi, I'm Dan Howard, associate Director in Health Economics here at Mtech Access. I've got around 10 years of experience of technical modelling and consulting and industry primarily for HTA submissions including submissions to NICE, the SMC and the NCP in Ireland. Most of the submissions I've worked on have been in oncology and as a result I have firsthand experience of everything that we'll be discussing today. I'll hand over to Joe.- Thanks, Dan. Good afternoon everyone. My name's Joe and I'm a principal medical statistician at Acturis and in this role I design and implement real world evidence studies using the secondary electronic healthcare record data. We have access to Acturis. I'm looking forward to talking about both real world evidence and real world data today. I'll hand up to Nick how to introduce himself.- Thanks, Joe. Hi everyone, I'm Nick Latimer. I'm a professor of health economics at the University of Sheffield and also an analyst at Delta Hat. So I'll be talking about decision maker perspectives because I've sat on a NICE appraisal committee for five years. Back to you, Sam.- Thank you very much. I just wanted to say at this point thank you to everyone who's already submitted questions ahead of the webinar. Please feel free as we're going through the the webinar to add any additional questions to the chat. And as we get to the end of the webinar, hopefully we'll have some time to to go through your questions. So what we're going to talk about today is we're going to, I'm going to hand over to Dan to talk about partition survival modelling for HTA and some of the common challenges that are faced. Then we're going to talk about how real world data can be used to generate external control arms and as Nick said, he's going to be talking about sort of some real world examples of how real world evidence can sort of support economic modelling and some insight on how committees might see that. We loop back around for some reflections and next steps and I said hopefully we'll have about 10 minutes at the end for some Q&A. So Dan, over to you.- Thanks, Sam. I should have control now. Great. So in my section I'll be doing a refresher on partition survival modelling and more importantly I'll be doing an overview of the types of challenges that we face and the evidence that we use to inform partition survival models and in particular how real world evidence can contribute to these analyses. So partition survival models, they're an ever more popular method of modelling cancer in HTAs. Most oncology appraisals for NICE and HAS use this approach. So the figure for NICE is 54% of a hundred oncology STAs conducted between 2013, 2018. STA is a single technology appraisal utilised partition survival approach. And this figure was 82% for in France for submissions to the HAS and that's between 2016 and 2022. They're used so often because they're conceptually really attractive. You only need two outcomes in principle, which are captured very frequently in cancer trials. And these are overall survival and progression-free survival, which is a composite endpoint of time to progression or death. With these two outcomes you can model three health states. So you can see on the graph on the right we've got progression free, progressed disease, and death. However, you're rarely going to come across complete survival data where the Kaplan-Meier plots for OS and PFS reach zero with no censoring. So in order to estimate what the full curves would look like or to rather to estimate them, we often fit survival models. There are lots of different methods for extrapolating these survival curves that can be with independently fitted parametric models, jointly fitted models, some more complicated ones like flexible splines and mixed secure models. But they basically all attempt to do the same thing, which is estimate long-term outcomes from incomplete clinical data. These methods are all described in detail in NICE technical support documents. If you want to go and have a look at them, which I recommend, that's 14, 21 and 24, which were all essential reading if you're going to submit to NICE and use a partition survival model structure. Once the curves have been extrapolated to the end of the time horizon that you're going to need to model, which is typically the entire lifetime of patients, you can categorise patients into health states. So here I've got a different graphical representation of a partition survival model, which illustrates the health states that we're going to model. These are the exact same ones that are in the graph on the right and we've got the proportion of progression-free patients, which is just simply calculated as the area under the PFS curve. Then to work out the proportion of patients who've died, that's calculated as one minus the OS curve and that leaves us with the final health state, which is the number of patients who progressed but haven't died. And that's calculated as OS minus the PFS curve. So that very briefly, has covered a simple partition survival model. You can have more complicated structure than this, but this is the most common structure that's typically used the three health states. Right now I'm going to cover some universal issues that you'll face in oncology modelling. So as I mentioned, like while the structure of these models can be quite conceptually simple, the main areas of complexity and in my opinion the really interesting part of this is to do with the data that informs these models. So even a well-designed phase three trial can still have issues like the wrong comparators being included, desirable comparators being excluded. For example, if you submit to NICE using an American trial, you might have comparators which aren't yet reimbursed in the UK. You can have heterogeneity in your trial population versus the target reimbursed population. And you'll definitely almost always have uncertainty around your long-term outcomes for OS and PFS. And this is an area where real world evidence can really help fill these gaps. So what to do if you have irrelevant comparators or a comparator that you need isn't included. So as I mentioned previously, treatment pathways differ by market and some treatments will be reimbursed in one market and not in another. So in these cases you're going to lack head-to-head evidence. So you'll need to syn synthesise indirect evidence. Now when it comes to indirect evidence, HTA bodies typically accept it, but historically this is focused on what I'm going to call traditional evidence synthesis methods, things like Bucher, indirect treatment comparisons and network meta analyses, which typically require at least some direct evidence in order to build your evidence network with your comparators. However, this isn't always possible and recently some other methods which do not require direct evidence have gained popularity. And these include things like matching adjusted indirect treatment comparisons, simulated treatment comparisons and external control arms in particular to name a few. These are all approaches where real world evidence can be utilised and is really useful. You're also going to have uncertainty in your long term survival curve extrapolations, which I mentioned earlier, this can be due to having immature trial data. You might also have strange behaviour if your PFS and OS curves when they're estimated independently. For example, these curves can cross if they're estimated independently, so your PFS can exceed OS, which isn't possible. The subsequent treatments, the impacts of subsequent treatments, sorry, on OS, can also have a factor in uncertainty in your long term extrapolations and uncertainty in the specific method that you're choosing. So the the specific model that you're fitting to it is always a sticking point overall that usually a is sticking point in oncology appraisals, just an example, like a Weibull model, gamma, lognormal, et cetera. However, real world evidence data sets can address some of these concerns often by a longer duration of follow up and more complete survival data as a result. So another issue is treatment effect waning that we need to account for. This is particularly relevant in immuno-oncology where trials often don't have a long enough follow-up to observe treatment waning so we need to use assumptions. But we can inform and validate these assumptions by using real world evidence and shed light on what treatment effect waning might look like in our population. And another area that I'm going to have a look at as well is, while I've been focusing on clinical outcomes, the other major components to a cost effectiveness model are obviously the costs and historically micro costing studies and key opinion leader views in ad boards, elicitation sessions and interviews have been used to inform all costs, but real world evidence can potentially offer a solution here as well. So now I'm going to quickly go over some different types of evidence which are used in oncology modelling for HTA. These are going to be categorised broadly into direct and indirect evidence, which will lead onto the next section that Joe will present. So first of all, the evidence which companies are submitting for HTA is changing. Direct evidence is typically held as the gold standard with good reason provided it contains the population and comparators that you're interested in and in great enough quantities and quality. It's often straightforward to analyse head to head data as well. But this isn't always the case and especially in recent years with legislative, legislative, sorry, changes which have impacted the trial data, which we end up submitting to HTA. So some of these changes include the the innovative licencing and access pathway, which was introduced by the last government with the goal of speeding up marketing authorisation. The current government have also cited early approval of drugs as a goal in their life sciences strategy in the manifesto. In addition, we've got Project Orbis, which means that the FDA in the US and the MHRA can concurrently review new cancer drugs, meaning that rather than UK marketing authorisation happening after US marketing authorisation, this can potentially, if you go down the Orbis type A route happen at the same time. And, but why am I talking about regulatory pathways here and why does this matter for HTA? Well in the normal NICE STA pathway, the first committee meeting is normally held within a month of marketing authorisation being granted. So there's a knock on effect of having earlier regulatory approval with having earlier reimbursement and with earlier data, maybe quite a bit earlier than we've been used to using. Moving on. So as I just said, HTA bodies generally prefer direct evidence for obvious reasons, but when you use earlier data, you're often using phase two data, for example, which might not have a comparator arm for technical or ethical reasons and will often have lower patient numbers than phase three trials. So in these circumstances you'll need to rely on indirect evidence, which comprises of a variety of techniques which I've listed on this slide. It's not exhaustive, these are just a few that I've picked out. Even though direct evidence is often seen as more reliable, we can use statistical techniques. For example, matching and waiting to decrease the impact of heterogeneity on our study results. So real world evidence data sets have the potential to be a lot larger than anything you would see in a normal trial and more complete because we potentially have a longer duration of follow up as well. And may be subject to less uncertainty when you're extrapolating long-term outcomes. NICE themself have acknowledged the importance of real world evidence and have published a framework to guide the use of it in appraisals. So where does real world evidence actually fit in, in the context of what I've just described in terms of the techniques we can use and the analyses that we can do? So here I've just got a very simple diagram of some indirect evidence synthesis methods. I've tried to categorise them into ability to adjust for heterogeneity across the top and the data requirements for each of these methods in terms of cohort level versus patient level data down the side. So the top row here covers network meta analyses and meta regression and naive comparisons and Bucher ITCs. Now these are all extremely useful and well understood methods which make use of cohort level data, especially the the network meta analysis and meta regression methods. But we can't always use these techniques due to a lack of head-to-head data, meaning that you can't form a network or they might just be less desirable for whatever reason. For example, I, I would argue that an naive comparison is, is going to be less desirable than any of the other methods because you can't adjust for patient characteristics. These methods all have their place, however they're just not the focus of today's webinar. Instead we'll be focusing on these bottom ones which we've highlighted here. So matching adjusted indirect comparisons, simulated treatment comparisons and in particular and arguably the strongest method here, external control alarms. These techniques are all data hungry but they have huge potential for generating high quality evidence which can be used for HTA submissions. So now I'll hand over to Joe from Arcturis Data, who's going to talk about real world evidence synthesis techniques in a bit more detail.- Yeah, thanks for that, Dan. That was a really nice overview of the current state of play. So as we've seen real world evidence may provide solutions to many issues that arise in the health technology appraisal process. And in this section of a webinar, I'd like to take a detailed look at how real world data could be applied to achieve this. But what is real world data and what does it actually look like in practise? Well there are multiple ways in which real world data is defined in literature, but most definitions do agree on the following aspects. So first, while real world data can be obtained from several different sources, so things such as the electronic healthcare records, or insurance claims data, a unifying feature is of the data has not been explicitly collected for a clinical study, but it's instead been collected for purposes of things such as audits or reimbursement. Second, real world data can take many forms. It may be structured data such as quantified lab results, or demographic information, or maybe unstructured free texts, so things such as pathology reports or clinical notes. It's also the case for real world data be obtained from a single electronic healthcare record system, but may also be constructed through linkage across multiple different systems. Third, real world data will likely consist of individual patient level data, but it may occasionally take a form of aggregate or summary data or data in this aggregate formats usually going to have less utility than individual level data. Fourth, collection of real world data can be either retrospective or prospective, but it should always capture the realities of routine clinical care. And finally, just the way that real world data is collected, any analysis that makes use of this type of data will have to be observational in nature. But I think it's fair to say the most common and I could believe the most useful real world data sources will be derived from retrospectively collected electronic healthcare records and will consist of individual patient level data. So as the quantity and quality of information stored within electronic healthcare records systems grows, the breadth of insight that could be obtained from this data source will grow too. The information content of this data is now so great that it often allows us to perform indirect treatment comparisons with clinical study data. And this is particularly useful as you've heard in situations where a randomised clinical study may not be possible for ethical or logistical reasons. And an increasingly popular approach to performing this type of indirect treatment comparison is for use of something called an external control arm or ECA. So what is an ECA? Well, as with the concept of real world data, there are a few ECA definitions that are present in the literature, but broadly speaking, an ECA is a control group consisting of patients who have been sourced from outside of some clinical study of interest who have individual level data available. Now this individual level data will have been used to assess each patient for the eligibility criteria of that clinical study of interest. The patients who fulfil these criteria will then constitute the external control arm. They can act as a control for the clinical study of interest. And this is achieved through comparison of outcomes between the external control and the patients who have received an investigational therapy in the clinical study of interest. I think to understand the role played by an ECA, it's important to first consider the gold standard of clinical evidence generation, which is a randomised trial. In a randomised trial, a control and intervention arm constructed from patients who fulfil the study eligibility criteria with assignment to each study arm being performed at random. This ensures that the received treatment depends purely on that random assignment, allowing us to assign a causal interpretation to any differences in outcomes that we observe between these two study arms. Now as you've seen, unfortunately a randomised study is not always possible. It's due to ethical or logistical constraints and in such cases a single arm trial may instead be used to demonstrate safety and response to novel therapy. In a single arm trial, again, a cohort of patients are selected based on their fulfilment of some eligibility criteria, but all patients in the study will receive the investigational therapy. There's been recent interest in safely expediting the health technology appraisal process using these results from single arm trials. This is achieved by taking the single arm trial data and performing by indirect treatment comparison with some external control of the ECA, a clear option for this process. Now in the real world, patients will receive therapy based on clinical decision making and reimbursement guidelines. But if the eligibility criteria of the single arm trial trial are applied to these real world patients, we can effectively filter them down to create an external control lab consisting exclusively of patients who could have hypothetically participated in that same single arm study but who've actually received usual care instead of a novel treatments. If data from this ECA has been combined with a single arm trial, use of statistical causal inference methods can allow for a comparison of outcomes in which the bias introduced by baseline differences between the cohorts can be mitigated, allowing us to generate insightful indirect treatment comparison evidence. So ECA analysis are a particularly impactful application of real-world data in the HTA process. And this is because for indirect treatment comparison that's being conducted with an external control arm can really integrate large quantities of representative data from real world sources providing estimates of the effects of a novel treatment relative to one or more therapies currently being used in routine care. But how can we actually construct an ECA analysis in practise and what should guide our decision making when applying concepts we see in randomised studies such as eligibility assessment to real world data sources? Well a document that can help guide us in this process is NICE as real world evidence framework. This describes best practices for generating high quality evidence from real world data. Obviously it covers a range of topics, but fundamental aspects of a real world evidence framework is the use of something called the target trial emulation framework. And this is particularly relevant when conducting an indirect treatment comparison. So target trial emulation framework explains that analysis utilising an external control should aim to actually emulate the randomised study that ideally would've been performed instead of a non-randomised study if it was actually performed. So to achieve this and must consider all aspects of a construction of an external control arm and do our best to align them with that hypothetical randomised study we would like to have performed. Of particular interest here, are the ways in which we should perform eligibility assessment, how we should define the study index date, how study outcomes can be harmonised across real world and trial data sources and what our choice of statistical method will be to perform about comparison of clinical outcomes. So to demonstrate best practise for ECA construction, I'd like to focus on a fairly recently published study that incorporated many aspects of a target trial emulation framework into its design. So in this study the authors compared participants who participated in ZUMA-5, which was a single arm study of axi-cel for follicular lymphoma with an external control arm consisting of patients source electronic health care records who had received routinely used therapy for follicular lymphoma. The authors called the ECA cohort for SCHOLAR-5 cohort they compared multiple categorical end times within outcomes between the ZUMA-5 and SCHOLAR-5 cohorts. To handle differences in these cohorts at baseline authors use propensity score based methods when comparing outcomes and this allowed them to adjust the differences in the demographic and clinical profile of patients across these two study arms. Now as mentioned previously, the design of an external control arm could be deconstructed into several concepts, each of which align with the different aspects of a target trial emulation framework. And what I'll do on the following slides is detail how this study was conducted and how the study designed aligned with those key concepts from a target trial emulation framework. So for this ECA analysis, the first concept to consider is the assessment of each available real-world patient for the eligibility criteria of ZUMA-5. Now due for prevalence of missing this in the electronic health healthcare records, it's not always possible to assess all eligibility criteria when using real world data. So it's important to identify which criteria can be assessed directly, which cannot be assessed, and which criteria may be assessed using proxy criteria that are constructed specifically to work with the content of real world data sources. When generating a SCHOLAR-5 cohort authors were able to directly assess many of the ZUMA-5 eligibility criteria directly, including things such as the number of prior therapy lines each patient had received. They couldn't assess criteria such as whether patients were pregnant or breastfeeding. This is because this information may not always be easily determined from electronic healthcare records. The authors also built a proxy assessment criteria for ECOG performance status being zero or one and they did this converting Karnofsky scores, which is a similar measure to ECOG into ECOG scores where ECOG scores were not available but Karnofsky scores were. And then assessing these new observations for eligibility criteria. This approach of using proxy criteria where necessary makes best use of all available data and allows for more assessment of eligibility. Following the construction of the ECA eligibility criteria, it's necessary to identify when the follow-up period for each rule a patient should start and usually this index date will coincide with the initiation of some comparative therapy of interest. In SCHOLAR-5 the comparator was any therapy that was routinely used for the treatment of follicular lymphoma if it was initiated after eligibility was first achieved by a patient. This analysis that meant that multiple index dates were actually possible for any patient who had received more than three prior lines of therapy. As any subsequent line could actually be used as a comparator. So the presence of multiple potential index dates as a common issue encountered when constructing an external control arm and can lead to the introduction of the multiple time bias if not properly accounted for. In this study, the authors randomly selected the start of one eligible therapy line as the index date for each patient. And this represents one of a number of approaches that are applicable in the target trial emulation framework that can be used to circumvent the issue from multiple index dates. Once the index date can be identified for each real world patient, the next step is to capture key eligibility and measured confounded data around that index date. So to do this, a capture window is defined around index date and data points that reflect the status of the patient at baseline and retained. This step the diagram we can see how ECOG and Karnofsky score data may be captured around index date enabling assessment of whether each real world patient had an ECOG score of less than two index date and consequently whether that patient fulfilled with eligibility criteria for entry into the external control arm. Now importantly, multiple relevant data observations may be made during the capture window and rules must be defined during the study design stage to ensure that eligibility is assessed using only the data that's most reflective of the status of each patient at index date. In the example presented on this slide, we can see multiple observations captured around index date, but only those are closest to index date and preferentially goes before index date are used to assess that eligibility. Once eligibility and confounding data have been captured, it's possible to apply the eligibility criteria themselves and actually identify those patients who will fall by ECA. Once the ECA cohort is identified the next step is to construct outcome measures for these patients. In the ZUMA SCHOLAR-5 study the authors considered standard clinical trial outcome measures including progression-free survival and overall survival alongside therapy-based proxies of clinical trial outcomes. And in this case the authors considered time to next treatment, which often acts as a proxy for progression-free survival as progression is usually closely followed by a change in therapy. So if that change in therapy has been taken to be indicative of a progression event having occurred. On this diagram though we can see that in certain scenarios there may be a considerable time difference between observed disease progression and any change in therapy that may be used to construct a proxy outcome such as time to next treatment. And because of where possible the outcome measures that are compared between trial and real world sources should be constructed using the same criteria. This approach is preferable as it allows for a like for like comparison between trial participants and ECA members. This reduces the possibility of systematically inflating event times of a control arm, which could bias any estimates treatment effect. And at this stage it's also important to determine how censoring will be handled for the external control arm cohort. For most the electronic healthcare record based studies, the censoring date will reflect the data which data has been cut by the data provider and received for analysis. But it's also important to consider whether patients have become unobservable for any reason before that data cuts. And then the final step in ECA analysis is actually about comparison of outcomes between the external control arm and the trial cohorts. Importantly in this case study authors did identify a series of confounding covariates that included prior stem cell transplantation, relapse, or refractory status at baseline and the number of prior therapy lines amongst others. These covariates have an association with both the treatment assignment process and the measured outcomes, meaning that they were both, they were confounders and that they needed to be controlled for to minimise bias in any estimated treatment effect. By capturing these covariates around index state, applying them in a model which produced estimates of propensity scores and then using these scores to re-weigh the contribution of each patient's the analysis, the influence of measured confounders in comparisons of outcomes was mitigated in this case study. Now in any ECA study is imperative of a choice of confounders included in the analysis are based on clinical expertise and careful consideration of how all measured variables are associated with treatment, outcome measure but also with one another. And in this case, after following key aspects of the target trial emulation framework, the authors did identify a clear treatment benefit for axi-cel versus usual care for follicular lymphoma. We have a progression-free survival hazard ratio estimated between 0.18 and 0.49. And as you can see on this Kaplan-Meier plot there's clear separation between these curves. So we've seen how an ECA can be constructed from real world data and an overview of what constitutes best practice for the design of an ECA study for an indirect treatment comparison. But it's important to recognise that real world data can provide important evidence across the wider HTA process too. So for example, construction of a cohort of patients receiving a comparative therapy for a given technology appraisal enables clear characterisation of real world comparative use and this includes assessment of use of support medications, healthcare resource utilisation and capture of a wider cost associated with that comparator. Similarly, real world data can be used to generate evidence to support specific economic modelling assumptions regarding comparative use that may have a meaningful influence on the ICER. Now so far we've seen some potential uses of real world data in the HTA process and how this data should be applied. But fundamental to the use of this evidence is establishment of the suitability of the data and statistical methods that have been used to generate it. Ultimately, regulatory and reimbursement decision makers must make this assessment and there are many considerations they will make when determining whether the evidence they've been with is of suitable quality. So I'll now hand over to Nick from Delta Hat who will talk us through what's involved in that assessment process.- Great, thanks very much, Joe. I think I've got control now. Yeah, okay great. So as I said before, I'm a professor of health economics, oh sorry, I'll turn my camera on. I'm a professor of health economics at the University of Sheffield and also an Analyst at Delta Hat. And I'm going to be giving a reviewer or decision makers perspective on the use of real world data, real world evidence to support economic modelling for HTA. So just before I start, some quick disclosures, I was a member of NICE's, one of NICE's appraisal committees for five years. I'm a member of NICE's decision support unit. Sometimes I work with Sheffield's Technology Assessment group and in fact Delta Hat acts as a NICE external assessment group in collaboration with PenTAG at the University of Exeter. Also, I'm focusing my slides on the NICE real world evidence framework, which has already been mentioned today and my experience on NICE committees. But what I present, what I say just reflects my own opinions, not those of NICE or anyone else. Just want to make that clear. Okay, so when we are considering the acceptance of real world data analysis by health technology assessment agencies, I think the NICE real world evidence framework is a crucial document. Obviously it is specific to NICE, but for many of us, NICE is a key agency and I think a lot of what the document says is also representative of the views of several other HTA agencies around the world. Although of course not all agencies will agree on everything. In my view, the the real world evidence framework document is a bit like the technical support documents produced by NICE's Decision Support Unit. And I think it's a document that review groups and committee members will refer to for guidance and so will inform decision making. So the real world evidence framework has a key messages section right at the start. And that really highlights NICE's commitment to using real world data within technology appraisals. So it says, as described in the NICE strategy of 2021 to '26, we want to use real world data to resolve gaps in knowledge and drive forward access to innovations for patients. We developed the real world evidence framework to help deliver on this ambition. The framework document then goes on to explain how it will deliver upon that ambition by identifying when real world data can be used to reduce uncertainties and improve guidance and by clearly describing best practices for planning, conducting and reporting real world evidence studies to improve the quality and transparency of evidence. And it then says that the key aim of the document is to improve the quality of real world evidence in informing our guidance. So the framework document then goes on to describe three core principles around the use of real world data to inform appraisals. And I think these are really important. So firstly, we must ensure data is of good provenance, relevant and of sufficient quality to answer the research question. Secondly, we must generate evidence in a transparent way and with integrity right from study planning through to study, conduct and reporting. And thirdly, we must use analytical methods that minimise the risk of bias and characterise uncertainty. So I think it's obvious that if a company wishes to use real world data in an appraisal, NICE's review groups and appraisal committees are going to want to see evidence that each of these core principles has been adequately covered. So to think about what that means in a bit more detail, I'm going to go through each of these principles a little bit more. Okay, so firstly, data provenance, relevance and quality. To address this core principle, the framework document helpfully includes DataSAT, which is a data suitability assessment tool, which covers data provenance, quality and relevance. So for data provenance information must be provided on the data source and on things like data management and governance. For data quality information has to be provided on the completeness and accuracy of each key study variable, which are those variables that are particularly important for the analysis. So such as variables on interventions received, outcomes and prognostic characteristics. For data relevance, the population and setting of the data needs to be described so that a reviewer can see if the data are really relevant for the decision problem being addressed. So for example, this could be about whether the real world data source is from the NHS or from a population that is similar to the NHS. Now I think it's important to highlight that completing DataSAT isn't a requirement when using real world data to inform a NICE submission, but it certainly makes sense to complete it if you want to increase the chance that NICE will deem the evidence reliable enough to inform decision making. If NICE provide the tool, it makes sense that you should probably use it. Okay, so a bit about my experience with key issues for reviews and decision makers on on this particular topic. Firstly, I think committees almost always think about where the data are from and whether the data are generalisable to the population of interest, which is really about the relevance of data. So for example, NICE is making decisions for the NHS population, so wants to see if the real world data set is generalisable to the NHS population. If the real world data source is from the UK, then people in it are likely to be selected from that NHS population. But then the question is whether all types of patients in the NHS population are represented or does the real world data have specific eligibility criteria, which could mean that they're important prognostic differences between the patients in the NHS population and the patients in the real world data source. So NICE always going to think about this. Or if the real world data is from another country, committees will think about whether the data are transportable to the NHS population. And transportability is another thing that has been added quite recently to the real world evidence framework document. So again, they'll think about where is that data source from and are the populations similar to the target NHS population, or do they have imported differences. Beyond where the data are from, I think it's very common for NICE committees to consider the variables included in a real world data set. So are important variables measured and amongst the variables that are measured, is there any important missing data, which is about the quality of the data? And fundamentally, the reason committees care about missing data and the variables that are measured is because if they think that there could be important differences between the population in the real world data set and the NHS population or in the case of external control arm studies, the population in the uncontrolled study of the experimental drug and the population in the external controller, then the committee needs to know whether the data are sufficient to allow adjustments to be made to account for the differences in study populations. So what this fundamentally boils down to is that the committee, and this is my Chair of committee here, are going to want to know whether they can really rely on the real world data sources and whether the data are relevant enough to address the decision problem being faced in the appraisal. And it's likely that the committee will probably think that if the data aren't relevant, or of poor quality that they can't rely on it for decision making. So I've dwelt on this data provenance relevance and quality aspect here quite a lot because I think it's really important. Sometimes it is tempting to spend a lot of time thinking about exciting intricate analytical methods which are important, but if the data aren't right, analysis won't be useful no matter what we do. And I think that is important to bear in mind. Okay, so the next core principle covered by NICE's real world evidence framework is study planning, conduct and reporting. So obviously the framework document says a lot more than what I have on the slide here, but a really key part is that as much as possible should be pre-specified and published. And that includes the study design analysis plans and reporting. The document recommends the HARPER tool for developing protocols for real world data studies and refers to several reporting checklists as well to ensure that all important aspects are reported clearly and transparently. Well, I think most of this will be naturally covered if a target trial approach is used as explained by Joe. But still it's important that all of this is planned upfront and that it's documented transparently. Now I think the key questions that committees often ask around this is whether an analysis were pre-specified based on the concern that if they weren't, perhaps they've been cherry picked to provide the desired results. And related to this, the committee will want to know that the study has been designed and conducted in a way that's going to minimise bias. And for the committee to be confident about that the study needs to be planned, conducted and reported really clearly. Okay, so the final core principle covered by the NICE RWE framework about analytical methods. So Joe has already talked about the methods, so I'm not going to go into detail here, but some key points are that the framework recommends taking a target trial approach using analytical methods that avoid or remove bias and including sensitivity and scenario analysis to give an idea of the stability of the results and the importance of any potential biases that might remain due to things like missing data, or unmeasured confounding. I think that depending on the expertise of members of the review groups and appraisal committees, there are a whole range of questions that could be asked about the analytical methods used around things like choices of time, zero time points, variable selection, missing data, treatment pathways, censoring potential residual biases. So lots could come up there. Okay, so before I finish in Joe's slides, he talked a little bit about a published external control arm study for axicabtagene ciloleucel for follicular lymphoma. I thought it might be interesting to look at the NICE appraisal that used that external control study to see what kinds of things the NICE appraisal committee discussed in that case. So, so this was technology appraisal 894, you can look it up and it's important to say that my summary here isn't exhaustive. These are just some bits that came directly out the final guidance document on the website that I thought were interesting. So first of all about relevance of the real world data. If you remember, axi-cel had been investigated in an uncontrolled study called ZUMA-5 and an external control arm was created using real world data in a study called SCHOLAR-5. The NICE committee concluded that using data from SCHOLAR-5 was acceptable, although it did notice a mismatch between the standard treatments received in SCHOLAR-5 and in the NHS. So there were some differences. With reference to the conduct of the ECA study, the committee discussed the eligibility criteria used for people included in the comparative effectiveness analysis, which was important because the number of prior treatments did differ in the two studies. The guidance document also noted, and I thought this was interesting, the methods used to adjust for baseline differences between the ZUMA-5 and SCHOLAR-5 populations and that these methods had improved the comparability of the data sources. But there also, there was also quite a detailed and pragmatic, I think, discussion about covariate selection where it was noted that adjusting for lots of variables is difficult when sample sizes are small as they were in some cases here because this can lead to problems with convergence of statistical models. So the committee was clearly thinking quite hard about this and in quite a lot of detail. With respect to the reporting and I suppose conduct as well. The committee noted that the company had submitted sensitivity analysis using alternative methods and quantitative bias analysis, both of which were seen as good things. Overall the committee's conclusion on the analysis, at least the conclusion recorded in the guidance document is a, I suppose a little bit vague. It stated that the adjustment method used by the company is highly complex. It's not really directly saying whether the analysis were acceptable or not, but it is clear from the document that the analysis was used for the decision making, although the lack of direct comparative efficacy data was highlighted as contributing to a high level of uncertainty in the appraisal, which I guess is what you would expect. Okay, so finally some conclusions from me. So although this session is about using real world data in HTA, I think it is important to remember that RCTs remain the preferred source of evidence on the effects of new treatments. And that is stated in NICE's real world evidence framework document. However, it is also stated that non-randomised evidence can be used when RCTs don't exist or aren't relevant, or are of poor quality. So certainly we can use these data sources and analysis. HTA decision makers are generally open to this, but we can't just use any old RWE study. The data needs to be high quality and relevant and the analysis need to be well planned pre-specified and well reported. And finally, even though I am a fan of using complex methods is what I've spent a lot of my career trying to do, a lot of my research is on these methods and these data sources. We do need to understand that when we are not using randomised data, even high quality analysis using high quality data are likely to be associated with high levels of uncertainty. And I think it's fair to say that decision makers may be more aware of uncertainty and perhaps will be more tentative when making decisions based on real world evidence rather than RCT evidence. And with that, thanks very much for listening and I think we can move on to questions.- Thank you very much, Nick, and all the speakers, that's been absolutely fabulous. So yeah, I'm going to open the floor to some questions and I'm going to start with some of the, well actually I'm going to have a little bit of a reflection before we add on questions. So I think today thank you again to all our speakers. We've learned a bit about using real world evidence in oncology, we've learned about how external control arms can be devised, sort of the best methods for that and then some real life examples of that. So thank you very much. And then some real critique of how an EAG and a NICE committee have have critiqued some of that real world evidence. So that's, that's perfect. Thank you very much. So I'm going to start with some questions now and I think first question to you, Dan, can you explain how some of the concepts that we've covered can be applied in non-oncology settings? So for example, rare disease or nephrology?- Yeah, sure, I think that it's important to state that we picked oncology here because it's a really good example of a disease area where you're going to have a lot of data sources available and that the outcomes that you need are very well characterised and often easily attainable. I say that sometimes PFS isn't the easiest to get and sometimes you have to use proxies for it like time to discontinuation, but there's no reason why any of these principles that we've been describing today can't extend to other disease areas. I think the main issues that you're going to run into are the evidence sources themselves. So in rare disease for example, you're probably by the virtue of it being a rare disease, you're probably going to find less real world evidence than you would in a more common disease. And it really depends on the registries that are available and the outcomes as well that you're measuring. So that's why we picked oncology because OS is an objective outcome. It's easy to assess, you know, a patient's alive or they're not. And a lot of the other outcomes that you find in oncology, real world evidence sources are very well characterised, understood, and have a large amount of data available.- Thanks, Dan. And then question for you, Joe. So sort of building on this, we also asked about the use of real world evidence in the case of missing comparative data in hard to research areas. So any sort of thoughts on that?- Yeah, I guess my answer's going to be quite similar to Dan's one because the questions kind of overlap a bit. Maybe I can give a perspective from somebody who like has their hands dirty with the data. So yeah, obviously the example that I used wasn't in a rare disease or anything like that, but it was in an oncology setting and I think as Dan said, it would be fair to say the majority of examples of real world evidence use are in the oncology setting. And this is a consequence of the fact that a lot of the data that is generated in the oncology setting is required to be collected for audit purposes. So it's very, it's available, but also a lot of those single arm trials are also in the oncology setting as well. But having said that, obviously there's no reason to be approaches that have been discussed today, couldn't be applied to rare diseases. And we do see it happening in practice, but I think, you know, we have to accept you're going to see small patient numbers and therefore you're going to have high variance in the evidence that's been generated. But insightful evidence regarding comparative use, best supportive care, et cetera, is definitely achievable, particularly with larger data sets. We can capture many patients, but they have to have a sufficient granularity that you can assess some of those more complex clinical outcomes are often associated with rare diseases. Out here we spend a lot of our time actually collecting and clean that type of data and which required to construct those types of outcomes. You know, that data is out there, it just requires a bit of work to get it. And then from like the statistical perspective as well, rare diseases offer some slightly different approaches. So we might be inclined to reach for some slightly different methods. So things such as Bayesian approaches, they're particularly appealing use for ability to incorporate prior knowledge from other studies in a principled way. This should allow us to minimise some of the uncertainty associated with small numbers and really squeeze as much information as possible out of the data that's there. But yeah, in general, rare diseases, harder to research areas are complicated by small patient numbers requirement of often quite complex data and occasionally complex statistical methods. And in my experience earlier, a process for real evidence generation so integrated into the development of the HTA submission in a rarer disease and the higher quality of evidence that it's going to be produced'cause more time can be taken to actually identify the right sources and right statistical methods to generate that evidence.- Thanks, Joe. So Nick, we've had a lot of questions about how HTA agencies might view real world evidence. What do you feel are the sort of unmet opportunities for HTA agencies when it comes to real world evidence?- Yeah, okay, so I mean I think the real world evidence framework document demonstrates that these agencies are thinking a lot about these data and it's not uncommon for these data to be used. So use is being made, my personal opinion, I think there's two areas where they could use it a lot more. So the first is in terms of reviews of previous decisions. At the moment, you know, a lot of NICE decision making, a lot NICE recommendations say they'll be reviewed in X number of years, two or three years, or something, but quite often that's not really done in a particularly meaningful way. And I think it could be using real world data once, if a treatment had a positive recommendation, then it's been used for that number of years. You can look, look to see what's actually happening and review the decision. I think that would be useful. And then the more kind of obvious one is the Cancer Drugs Fund appraisals where the drugs in the Cancer Drugs Fund for a while, normally two years, sometimes longer. And then when the review is done, one of the key data sources is from SACT. But what you find is that the data from SACT is always just for the experimental drug, which leaves you with problems in terms of if you want to use that data to see what's the treatment effect in the real world. You can't really just with that data. So obviously the data on the comparator is there from prior years before that new drug had been kind of putting the Cancer Drugs Fund, I would like to see much more use of comparative effectiveness analysis using data from both the comparator and the experimental drug.- Yeah, and as you say, that data should be, should be there, so why not use it? Yeah. Okay, question for you, Dan. Have you had a sort of direct experience with NICE accepting real world evidence that changed? Sorry, the committee's decision making?- Yeah, well, I think we've seen an example that Nick just went through of an appraisal where real evidence was used as part of the main evidence submission. And speaking from my own experience, I think I've taken part in NICE submissions that I don't think would've been possible at all without the use of real world evidence. I simply don't think that it would've gone through. You could've made an evidence submission because there would've been no comparative evidence available. So I think more than just influencing a decision, I think it also makes it possible to make a decision as well. Or, you know, rather than the appraisal not going ahead at all. I think it may have happened just much later in the future when you have a phase three trial and all that time you are, you know, there are patients who are missing out potentially on access to drugs when potentially access could have been achieved earlier with the use of real-world evidence. So yeah, it's it definitely does influence the decision making and it even makes it possible to make a decision.- Thank you. Joe, question for you sort of looking beyond HTA, how can real world evidence contribute to sort of long term monitoring of health technologies post-approval? And what role can it play in reassessing technologies as new health data emerges?- Yeah, so I mean naturally real world evidence does play a key role in post-authorisation studies. I think it'd be fair to say this is probably the most established use of real world data in a regulatory setting or a reimbursement setting. And you know, as information content in the electronic healthcare records is only growing, sort of the effectiveness of these types of analysis will also only grow. But I think I in general, I would actually just agree with a lot of what Nick was saying, his answer previously, you know, the Cancer Drugs Fund is a really clear example of post authorisation use, really where the drug has entered through a patient access scheme and data's being collected in a real world setting and then a number of years later that data's being reassessed. And I would completely agree with what Nick said about the comparator, you know, that data is sat there, the same data was being collected for, the novel therapy is being collected just through routine care for the comparator. And this same principle applies essentially for any situation where there's been a recommendation from NICE and the drug is entered routine commissioning, all the drugs that are involved in that process are being used in the NHS will be generating data in the electronic healthcare records and that data is available for reassessment future date if required. So yeah, I was completely, I'd be completely on board with Nick's answer to his question and the similarities to the question there about post authorisation. Yeah.- Okay, great, thank you. I think we've got time for a couple more questions. I guess coming back to you again, Joe, sorry, to pick on you again, but what role can machine learning or AI play when we're generating real world evidence to support HTA process?- Yeah, I mean it's obviously like AI and ML are like hot topics at the moment and it seems like these concepts are being applied everywhere. But I think when it comes to the assessment of, you know, novel therapies, we do need to be careful about where these methods are applied. Many of these methods are essentially like black box methods, meaning you don't really have any understanding of how they've arrived at result that they've generated. And you know, when it comes to assessing whether a technology is, you know, effective or safe, it's not particularly reassuring. So yeah, for more traditional statistical methods, you know, parametric approaches, survival curves and so on, you know, they're here to stay because they have clear assumptions about how the data is being modelled and they can be, they can be tested directly. So you can assess whether these methods are appropriate for their use. But having said that, you know, I think there are some areas where AI or ML methods do have some utility. And I think a really clear example is data enrichment and content extraction. So at Arcturis we have an ML team who work with the free text documents that we access. You know, we've got sometimes tens of thousands of these and we need to pass them quickly and they can develop these ML models which can extract the key data from these unstructured sources, turning them into structured sources that can then be leveraged for analysis. And obviously there needs to be some assessment of the precision and accuracy of these methods, but you know, that's completely something that can be done. So I think our ability to take an electronic healthcare record system and turn it into an analysable data source using AI methods is a potentially really, really useful application.- And then following on from that a little bit, I know NICE have just sort of released a bit of a statement on the use of AI, but it's a little bit unclear and a bit woolly, but do you think that will be something that in the future NICE will be more happy to accept? And I guess I'm opening the floor to anyone, any of our experts on that one? Dan or Nick, you got a thought on that?- Yeah, I mean it's, it's not specific to AI, but I think it's also relevant to AI in any methods that use things like that, that are quite novel. Like a lot is down to reporting I think. Quite often when complex methods, or new methods are used in submissions, they're not actually that well reported. And so for a reviewer or a committee member, it's very difficult to know to what extent you can trust these analyses if they haven't been reported well. So I think that's a really key thing. The other thing is proof of concept that these methods work is an important thing. So more broadly than just the AI stuff, but benchmarking studies are things that people have started doing more. So in the US there was something called the RCT-DUPLICATE Study where they used US health data to try to emulate existing RCTs and see if you could get similar results using the real world data. We are doing similar work to that using English cancer data at Sheffield actually. But I think if you want to use a data set proving that it can give reliable results by benchmarking against an existing RCT and then doing the analysis you actually need to contribute to the submission you need it for is also a good process, which I don't think has been done much.- Okay, well, we're just up on the hour so I just want to wrap up by saying thank you everybody for attending the webinar. The webinar is available I think on the, one of the links at the top and the materials, but we will be following up with everybody and for everyone who has put in questions that we haven't got round to answering, we will endeavour to follow up with as many questions as possible within the next two weeks. So thank you very much for your time this afternoon.