Using nowcasting and projections for statistical understandings in health

Introduction

A current picture of the health of Australians is important when developing health policy and when planning health services. Investment decisions for specific treatment facilities, workforce planning, and evaluation of health policy rely not only on an accurate measure of current disease burden (including new cases, prevalence and mortality), but also on a reliable indication of the burden that might be expected in the future.

The more recent and complete data are, the more likely they will provide a reliable picture of today. Recent data combined with time series data provide information about today, while enriching an understanding of the journey to this point. 

It is not uncommon, though, for the most recent data to be several years old. The reasons for this are many and varied:

• Some surveys that provide the main source of data for certain health conditions are conducted only every few years. 
• For some annual data collections, such as hospital admissions and cancer incidence data, the time it takes to collect, code, collate and validate the data so they are suitable for analysis can contribute to reporting delays. 

With such data, the journey can be understood – but what about the understanding of today? 

A number of techniques are used to provide more timely data, including:

• using ‘preliminary’ or ‘fast-tracked’ data that is subject to change
• imputing data based on similar population groups to fill data gaps
• undertaking large-scale data linkages across a wide range of existing datasets which provide valuable information but can require considerable time and expense to obtain necessary approvals. 

Nowcasting and projections are feasible and oft-used techniques that use understandings from the journey to provide a possible picture of not only today but also future health needs.

This article briefly explores why the most recent data in some collections can be several years old. It uses examples from the cancer reporting program and the Australian Burden of Disease Study to show how projecting both to the current period (nowcasting) and the future can provide statistical health understandings about today and beyond. It also discusses the limitations of this approach when unforeseen events, such as the COVID-19 pandemic, occur. (See Box NP.1 for definitions of key terms used in this article.)

What are nowcasting and projections?

Nowcasting may be described as a forecast for today. 

While similar in concept to nowcasting, projections use historical trends and assumptions about the future to indicate how things may look in the years to come. 

There are 2 key questions to consider in deciding whether to nowcast data:

• Are the most recent statistics available sufficiently relevant to today?
• If not, can a reliable estimate for today be produced?

Suitability of data

Not all data are suitable to nowcast or project. Producing a reliable estimate of current health burden depends on stable trends in disease data, coupled with knowledge or reliable assumptions of current policies and practices that may affect these trends. 

Diseases that have volatile trends with the potential to change quickly (such as infectious diseases) are difficult to accurately nowcast, as are diseases or conditions for which there are known changes in detection or treatment patterns that will influence the trend.

Diseases with slow moving and stable trends – such as cancer, diabetes, chronic kidney disease and cardiovascular disease – are much more suited to nowcasting. 

In addition, estimates of future health burden depend heavily on stable trends in demographic data (such as for population growth and ageing), as well as assumptions of future trends.

Limitations of nowcast or projected data

Of course, nowcast or projected data do have limitations. Determining current or future health burden involves some uncertainty, as it is virtually impossible to derive a perfect prediction. The further the actual data are from the point of the projection, the greater is the opportunity for inaccuracies to compound, and the more prone the projection is to changes in the environment, which may affect its reliability. Regular updates of nowcast and projected data help to minimise the impact of these changes.

It is important to indictate the reliability of nowcast and projected estimates. Rounding and prediction intervals help the user to understand the level of certainty associated with the estimate. 

AIHW practice

The AIHW uses nowcasting and projections in several products, including for its cancer reporting program and burden of disease estimates. This article provides a brief description of the rationale, methods and limitations for both cancer and burden of disease nowcasting.

Using nowcasting in cancer reporting

Cancer statistics released annually by the AIHW in Cancer data in Australia include those for incidence, mortality, survival, risk and prevalence. Most of these statistics are sourced from the Australian Cancer Database.

Why doesn’t the AIHW publish more timely cancer data?

Australian cancer data have among the highest quality and most complete population coverage in the world. Although the data are not available until around 3.5 years after the reporting period, this time-lag is consistent with reporting standards around the world.

There are several reasons for this time-lag:

• It takes considerable time for the jurisdictional cancer registries to receive and then enter cancer incidence and mortality data from notifiers (including hospitals; pathology laboratories; and the Registries of Births, Deaths and Marriages) and to follow up with notifiers and other authorised people to ensure important information about the cancer is correct. 
• Further time is then required to compile the jurisdictional data into the national dataset (including undertaking data harmonisation and quality checks) and to create the range of cancer-related statistics made available to the public.

What cancer data are nowcast?

In 2012, the AIHW started producing publicly available year-to-date (nowcast) statistics on cancer incidence (the number of new cases of cancer diagnosed in a given year) and mortality (the number of people who die from cancer in a given year) to meet the needs of stakeholders, including policy advisors, researchers and the general public. Nowcast prevalence statistics (the number of people alive who have previously been diagnosed with cancer) will be publicly released for the first time in 2024. 

These estimates support day-to-day service delivery by providing a valuable indication of how many cancer cases are expected to be diagnosed and treated, while changing (particularly increasing) trends help to identify emerging areas of need.

Actual data still needed too

Nowcasting does not negate the need for timely and accurate actual data on cancer. For instance, nowcast data are unable to predict unusual events – such as cancer clusters or other unexpected increases; as such, they are unsuitable to monitor interval cancers for screening programs or other service delivery and research where greater data accuracy is needed.

Survival rates are a key cancer statistic, but the AIHW does not produce nowcast or projected estimates for cancer survival as the most recent statistic is generally considered to provide a reasonably relevant indication of survival to today. Also, it is uncertain whether survival can be projected sufficiently well to provide an informed estimate for today.

Cancer incidence nowcasting

Given the apparent need for nowcasting, can a reliable estimate be calculated? 

In general, cancer incidence rates change gradually over time. There may be some volatility in rates between years, but there is often a trend that sees incidence moving in a general direction (for example, see Figure NP.1 to view trends for all cancers combined). 

To nowcast cancer incidence, the AIHW uses a linear regression method which looks at the last 10 years of incidence rates. Based on these rates, the AIHW then estimates current incidence rates by 5-year age-group and sex for a mutually exclusive list of over 350 cancer sites and types – which forms the building blocks for all incidence statistics. These building blocks are aggregated to report by various cancer groups and total cancer incidence. 

Full methods for nowcasting cancer incidence are described in Cancer data in Australia.

• For example, in 2019 there were 147,600 cases of cancer diagnosed with an associated rate (age-standardised to the 2001 Australian Standard Population) of 496 cases per 100,000 people. Using nowcasting methods, 164,700 new cases are expected to be diagnosed in 2023, with an estimated age‑standardised rate of 503 cases per 100,000 people (AIHW 2023b). 

The nowcast figures help to paint a picture of cancer today that may otherwise be hard to gauge from the most recent actual data. Similarly, the small degree of change in age‑standardised rates indicates that, based on recent trends, the rates for all cancers combined are relatively stable, even though the number of cases is increasing.

Figure NP.1: Cancer incidence and rates are suitable to nowcast

Prostate cancer – an exception for nowcasting

There is one exception to this methodology. Incidence rates for prostate cancer have been highly volatile as new detection methods have been introduced, making them unsuitable as a basis for nowcasting. 

Instead, nowcast prostate cancer incidence statistics use the actual incidence rates for the most recent year – rather than the most recent 10 years – applied to the relevant populations by age.

Long-term cancer incidence projections

Nowcasting incidence uses linear regression; however, there is a limit on how many years into the future it is considered appropriate to forecast cancer incidence counts and rates.

Within the AIHW’s cancer data program, nowcast cancer incidence rates are also referred to as short-term projections. Cancer incidence rates projected out to 10 years in the future (around 13–14 years beyond the actual data) are referred to as long-term projections.

To use linear regression for long-term projections would be to suggest that the cancer trends will continue to move in the same direction for this length of time. For many cancers, the cancer incidence trends indicate that this may not be a realistic assumption. For longer term cancer projections, the AIHW uses the NordPred software package to project cancer incidence. This is a specially designed cancer incidence projection program written by Harald Fekjær and Bjørn Møller at the Cancer Registry of Norway, which takes account of the impacts of individual ageing, general societal changes and different birth cohorts.

The longer the time between the last actual data and the projection, the greater the risk that new trends will emerge that projections cannot accurately account for. Changes to existing trends will affect the reliability of the estimate to some extent. 

While we produce nowcast estimates for many cancers and for many age groups, long-term projections only include the total number of cancer cases (that is, not disaggregated by age group) for those cancers with sufficient case numbers to support a projection with a degree of certainty. We elect not to provide detailed long-term estimates (such as by age group) for long-term projections because of the greater volatility of the finer-level data.

Cancer mortality nowcasting

The process to nowcast cancer mortality estimates is similar to that used for cancer incidence, but with fewer obstacles to overcome. 

• Firstly, the most recently available mortality data are closer to the year of data release (2 years). This shorter projection interval means there is less chance of unexpected changes in trends. 
• Secondly, unlike incidence (which can be subject to sudden spikes related to changes in diagnosis techniques or the introduction of screening programs), changes in cancer mortality trends are often more gradual. This is the case for prostate cancer, where the issues that affected trends in prostate cancer incidence (making them unsuitable for nowcasting) did not affect trends in prostate cancer mortality in the same way; thus, nowcasting of prostate cancer mortality can continue to use linear regression. 

Methods are being explored for long-term projections of cancer mortality; however, the AIHW is not currently producing these. 

Cancer prevalence nowcasting

The 2024 release of the AIHW publication will include nowcast prevalence statistics. Cancer prevalence is the number of people alive at a set point in time who have been diagnosed with cancer within a certain period. 

Prevalence is a function of incidence, mortality and survival, and projecting cancer prevalence is consequently more complex. A simplified explanation of how nowcast estimates for cancer prevalence are calculated is to apply survival rates to the number of people diagnosed with cancer to derive the number projected to be alive at the set point in time. Actual incidence data are used where available, and incidence projections are used where they are not. While short-term cancer incidence projections can be used to estimate the number of people diagnosed (as previously mentioned), cancer survival is not projected. 

Full methods for cancer prevalence nowcasting will be provided with the data release in 2024.

Why isn’t cancer survival nowcast?

As previously noted, the most recent survival statistic is generally considered to provide a reasonably relevant indication of current cancer survival. Also, it is uncertain whether survival can be projected sufficiently well to provide an informed estimate for today.

Figure NP.2: Without understanding the reason for a change in trend, cancer survival is difficult to nowcast

Accounting for COVID-19 in cancer projections and nowcasting

The most recent nowcast cancer incidence and projections include years during which Australia was affected by COVID-19; however, the latest actual cancer data are from before the pandemic. The impact of COVID-19 on cancer incidence and the potential for it to affect mortality have been speculated, but actual national data are not yet available for analysis. 

As well, changes in the population structure due to COVID-19 may also have an impact on future cancer estimates.

The purpose should always be considered when developing projections and nowcast statistics. When projecting cancer incidence for years impacted by COVID-19, we have not attempted to account for the potential influence the pandemic may have had on cancer incidence trends. For a start, the information base was insufficient to do so; and the resulting projections may not have fulfilled their intended purpose – to provide an indication of cancer incidence within Australia. COVID-adjusted projections would focus on the timing of cancer diagnoses (as diagnosis of cancer may be delayed during lockdown and possibly made in a later year), muddying the general trends and overall volume of cancer incidence. In producing a more complex set of projections, we would fail to meet the purpose of these projections and nowcast figures.

Difficulty of nowcasting different types of cancer

A single, consistent and easily replicated method of nowcasting is used for cancer, due to the large number of different types of cancer that must be nowcast.

Over time, various ways of projecting cancer statistics have been tested. Each method has apparent strengths and weaknesses. 

• For linear regression, it can take time to factor in emerging trends and there is more opportunity for inaccuracies where rates move more quickly. 
• For the prevalence projections, there are risks associated with projecting incidence rates combined with being unable to reliably estimate survival rates. 

Prostate cancer incidence rates

The worst case scenario for cancer incidence nowcasting and projections is where a cancer’s rates can dramatically rise and fall over a relatively short period. This happened, for instance, with prostate cancer incidence rates and the events are discussed in more detail in Cancer in Australia. Use of standard linear regression was considered inappropriate for nowcasting prostate cancer incidence as historical trends were not sufficiently stable, but the need for an estimate remained. 

In this case, rather than not provide a nowcast estimate for this cancer, the most recent rates were held constant and only population growth (which is known) applied. Essentially, the nowcast estimate for prostate cancer incidence can be interpreted as ‘if rates remain the same, the number of prostate cancer cases would be…’. 

There were 23,000 cases of prostate cancer in 2019 (the most recent actual data); by 2023, it is estimated that there will be 25,500 cases (AIHW 2023b). This large change with stable rates highlights the impact that population growth (particularly for older age groups) has on case numbers for a cancer such as prostate cancer, which often occurs at older ages where population growth is faster and incidence rates are higher.

Using nowcasting in burden of disease

The Australian Burden of Disease Study (ABDS) is a powerful resource to help understand the health of Australians over time and provides estimates of the disease burden for specific reference years since 2003. Information on burden of disease and injuries is important for monitoring population health and provides an evidence base to inform health policy and service planning. It is used to monitor progress against relevant targets in the National Preventative Health Strategy 2021–2030.

Data informing the ABDS

The data that informs the ABDS come from many different sources, including epidemiological studies, health surveys, hospital admissions, large government administrative datasets (such as the Pharmaceutical Benefits Scheme and the Medicare Benefits Schedule) and other state and national data sets, including the National Mortality Database (deaths) and the Australian Cancer Database. 

These data, which vary in both currency and completeness, are used in complex models for each disease or injury to estimate the burden from living with and/or dying from a disease or injury in a specific year. For each iteration, the reporting (or reference) year is determined based on the availability of the most recent data for key data sources. Where data are not available for the reference year, simple modelling techniques are used to provide estimates for the reference year. 

Benefits of nowcasting for burden estimates

The complex methods for deriving estimates (which can take between 2 and 3 years), together with the availability of recent data, lead to delays in releasing timely national burden estimates. 

• For example, the ABDS 2018 was released in 2021 based on data for the 2018 reference year. The 3-year gap between the reference year and the year of publication makes the estimates appear to be outdated and less relevant, especially so when rapid changes or major events occur in the interim, such as the COVID-19 pandemic. 

Nowcasting offered an opportunity to overcome these issues. In 2022, a new approach was adopted whereby burden estimates from the most recent reference year (2019) for each disease or injury were modelled forward to the year of publication (2022), taking known impacts of the COVID-19 pandemic into account where possible, to produce annual estimates (AIHW 2022). This same approach was taken in 2023 to produce 2023 disease burden estimates (AIHW 2023a). 

Figure NP.3 shows the resultant total, fatal and non-fatal disease burden from specific time points nowcast to 2023. This shows the expected continued decline of overall disease burden by 11%, driven by a strong expected decline (27%) in fatal burden and countered by an expected increase (6.3%) in non-fatal burden. 

Figure NP.3: Burden of disease estimates are suitable to nowcast

As burden of disease estimates are modelled numbers rather than ‘actual’ measures of disease or injury, projections to the publication year enable: 

• changing age structures, healthcare needs and environmental landscapes in Australia to be accounted for 
• a timely snapshot of the health challenges in Australia to be provided without substantially impacting the validity of the estimates
• more frequent and timely monitoring of progress against relevant targets in the National Preventative Health Strategy 2021–2030.

How burden estimates are nowcast

For each disease or injury, the most recent data, supplemented by information gleaned from expert consultation and from published literature, are combined with trend analysis from past years (assuming that the trend will continue) to determine the current burden. Where necessary, the reference period used to inform the trend is restricted where data are considered inappropriate for use in trend analysis. 

Methods used

Two regression models are used to accommodate different annual patterns of disease: 

• Where a decreasing trend is detected, the Poisson (log-linear) regression (which assumes that rates changed at a constant per cent annually) is applied.
• Where an increasing trend is detected, the simple linear (ordinary least-squares) regression (which assumes a constant fixed amount of change) is applied. 

Where no change is detected, the most recent rates are assumed to apply to today’s population. 

Further information on the methods used are available in the Technical notes for the Australian Burden of Disease studies 2022 and 2023. 

Accounting for COVID-19 in burden of disease nowcasting

The COVID-19 pandemic presents an important consideration for the selection of appropriate models given its impacts on the input data sources available, the health system or the disease/injury itself. 

For some diseases that were affected by COVID-19, estimates for 2020 are not included as that was not a typical mortality year, and morbidity was influenced by many factors, including, for example, the pandemic restrictions and the pause on non-essential surgeries. Other years may be excluded where data are considered inappropriate for use in trend analysis, such as due to coding changes, or where data in early years are not robust. 

While the impact of COVID-19 on other diseases and injuries could not be fully accounted for due to limited evidence and data availability, nowcasting of 2023 burden rates – including and excluding COVID-19 (direct burden) – showed that total burden rates would have been lower in 2023 than in 2018 without COVID-19 (AIHW 2023a).

Future directions for nowcasting and projections

Improving the timeliness of data presented in our reports continues to be a strategic focus of the AIHW. Timeliness of data reporting is also monitored as a performance indicator in AIHW annual reports.

This article, which provides examples of nowcast and projected estimates from 2 key AIHW reporting areas – cancer and burden of disease – presents a valuable discussion on the usefulness and considerations of now-casting and projections in AIHW reporting.

Further reading

Related topic summaries

Glossary   Data tables More articles from Australia’s health 2024 data insights