A lot can be said about forecasting and modelling, its accuracy, and how it works. Forecasting and modelling provide any decision-maker with plausible predictions or outcomes to give some kind of estimated consequence. Without this field of science, planning would be difficult, as one would simply make decisions without knowing what might potentially happen to a specific cohort, market, or product. Forecasting as a concept can be seen as a set of mathematical, statistical and/or computational tools applied to a set of assumptions about something.

This is according to Herkulaas Combrink of the Centre for Teaching and Learning at the University of the Free State (UFS) and PhD candidate in Computer Science at the University of Pretoria (UP), following the South African government’s modelling of how many people would contract COVID-19 and die, which has come under fire in recent times – with one expert saying it was “flawed and illogical and made wild assumptions”.

Combrink is of the opinion that South Africa – by using the MASHA Consortium – is using the best minds that the country has to offer. The fact that the leadership took a pragmatic stance and reached out to the scientific community has mitigated a medical disaster in a healthcare system that was not ready a few months ago.

“The government is looking at as many models as they can, but is working very closely with MASHA and the CSIR” says Combrink, who has been involved in clinical surveillance, and also forms part of the modelling team during his secondment to the Free State Department of Health.

Prof Shabhir Madhi, the former head of the SA National Institute for Communicable Diseases (NICD), recently said that the initial modelling and fatality estimates were “back-of-envelope calculations”.

According to a news report, the government’s initial model also predicted that 600 COVID-19 patients would need treatment in intensive care units (ICU) in SA by April 1. But by April 18, the last publicly released figures showed that there were 32 COVID-19 patients in ICU.

Tried-and-tested models

The models currently used are tried-and-tested epidemiological models, mathematical models, and economic forecasting models that have been used in the past. It has now been calibrated to the specifications that we know of this disease, which come from publications. The reason why you would use more than one model is to compare models retrospectively, so that you can see what is going on.

“The government immediately reached out to the best minds in the country, and with the aid of the consortium, took a stance to throw scenarios at the different models and stress test them so that they could know that they are using the best possible models to assist in resource management and decision-making. If government responded in a different way and didn’t reach out, we might not have had a lockdown and subsequently would probably have been in a different position where the country wouldn’t be ready.” “We can say with a high degree of confidence that the lockdown really helps to ease and flatten the curve in the country. In light of flattening the curve, the right decisions have been made,” says Combrink.

COVID-19 still new

Unfortunately, says Combrink, during the COVID-19 pandemic, there was not enough information related to the disease assumptions and it lacked the rigour and perfection associated with the already existing prediction models. Although it may feel like a lifetime, the first COVID-19 case was only reported in December 2019. Add to this that not all the parameters related to the disease were known in January, it was challenging to determine all the ‘ins and outs’ of this disease.

“Luckily, the mathematics and statistics of an outbreak have been extensively studied, and as a result, we only needed to use the correct parameters to estimate the spread of the disease in some of the outbreak models. The Minister of Health, Dr Zweli Mkhize, and the national modellers led by Dr Harry Moultrie, were transparent with not only their projections, but also how they derived their conclusions and what parameters they used,” says Combrink.
The most important thing in modelling is to calibrate according to what is known about the disease and people, explains Combrink. “It is impossible to predict people and a disease100% accurately, because you don’t always know how a virus will react to every single person’s body and you can’t predict human behaviour.”

“So, there is a certain degree of error and a certain degree of confidence that lies within each model, and that is why you evaluate these models on a regular basis. And this is important. You will never be able to say this is the exact number. Just like the weather. If the weather patterns were predicted to be 12 degrees tomorrow, and it turns out to be 16 degrees, you at least packed a jersey. You knew it was going to be cold. The chances that the weather predicts that it will be 12 and it turns out to be 57 degrees, is virtually zero. It gives you more or less an indication what to prepare for.”

Models are useful, but can also be wrong

Combrink says if you want to apply any model, you need to understand the assumptions and the limitations of the models. Given a certain set of criteria – what are the assumptions you are making and what are the expected outcomes – you can only act according to that. He says, as time goes by, we can now see that there are some models that yield much better results because we can now compare what was predicted two months ago and what is actually happening.

“Some models are useful. We can get a better understanding of the pandemic’s possible trajectories or gain an understanding of the impact that different interventions have made. Models are used for decision-making. These decision-making strategies can save lives. That is the purpose of models and modelling during these times.”

Combrink uses the weather forecast to explain how modelling works and that models can be wrong. “Yes, models are wrong all the time. Take the concept of weather as an example. How many times has the weather forecast predicted that there is an 80% chance of rain, and then it doesn’t rain? Models can give you a certain degree of confidence in an outcome related to a specific event or scenario, so that you, with some degree of confidence, can go forth and plan accordingly.”

“However, models can’t tell you what exactly will happen tomorrow, or the day after. It is not a crystal ball, and it is not a mirror into the future, but it can give you an indication of what is likely possible related to a specific scenario if you used the right variables. Let us consider that there is an 80% chance of rain in the weather forecast; will you a) go to work without an umbrella or b) with an umbrella? If it doesn’t rain, you are at least prepared for the rain because you took your umbrella. If you didn’t take the umbrella and it does rain, you may run into trouble because you did not appreciate the warning of the weather forecast. I think it is this concept that makes modelling so powerful. You can use it as a tool to prepare for things, in the event that it does happen, with a certain degree of confidence. Just like the previous example, there is also a 20% chance that it might not rain, but wouldn’t you want to be prepared?” explains Combrink.

Models are tools that can be used to base decisions on

No one truly knows how the pandemic will play out, and according to Combrink, it can be said with a high degree of confidence that if nothing is done about the pandemic, we know how it would turn out from a healthcare perspective.
“If you look at some of the global projections they gave months ago (in January and February) and compare it to what they said for March and April, you can see that they predicted, with a fairly good degree of confidence, what actually happened in certain countries. We have a good idea in terms of numbers and how it will play out, but what we will never know is what the impact will be on the socio-economic status of a person, the economy, and the impact on other diseases.”
“We do not know what is going to happen when it comes to mental health and COVID-19, for example. This is why modelling is a multidimensional approach, requiring inputs from various fields. Models can help us in the same way the weather forecast does. It is a tool that we can use to base certain decisions on, to be more prepared, because without it we won’t know to pack an ‘umbrella’ if it is predicted to rain or pack a ‘jersey’ if it is projected to cool down