The Data Science and Engineering During the Pandemic

During the most critical times of the COVID-19 pandemic, large data processing and analysis became fundamental to understand if the measures taken were appropriate and what we should do in the next few days, weeks and months. In comparison with other pandemics that occurred a century ago, like the Spanish flu, we were able to go through this period in a better shape than ever before. All thanks to the huge effort from Doctors, data and the scientists that worked on it.

The marriage between data and science has always been of great significance from the very first science to emerge, astronomy. However, in the last few years data science has taken a whole new height with the use of innovative technologies and huge processing power available in the public cloud. Furthermore, the vast amounts of data being produced every second created another profession, the Data Engineer. As a result, Data Scientists and Data Engineers usually work together to achieve incredible results.

What had enabled data science in public health?

A wide range of techniques that were not available in the past facilitated data science in public health during the pandemic. We are talking about human mobility, contact tracing, medical imaging, virology, drug screening, bioinformatics, electronic health records, among others [1].

All of the different sources from billion of connected devices real-time plus the extra computing capabilities provided in the cloud that allow you to elastically increase and decrease computing resources as needed when needed; plus the previously mentioned techniques enabled data science to provide enormous amount of help during the pandemic.

Difference between data science and data engineering

Data scientists see data as a tool to construct questions and then use data analytics to find patterns, create predictive models, and develop insights that guide decisions making within businesses. Meanwhile, Data Engineers are concerned about storing, retrieving, sharing, extracting value out of data for business decisions.

Applications across industries

Each industry has different needs. For instance, as mentioned in the pandemic example, the healthcare industry requires data to be available swiftly to make quick decisions that save millions of lives. This is particularly challenging since there are privacy laws and different regulations that need to be taken into account. Furthermore, other disciplines within healthcare have also benefited, such as medical imaging analysis, genomics and genetics, pharmaceutical research and development, among others.

In the same way, banks and insurers require prompt data in an ever increasing competitive market with FinTechs growing quickly. One of the most complex yet exciting fields is the real-time processing to catch illegal or fraudulent trading activity in the financial markets. Likewise, companies like Tesla are using large amounts of data to understand drivers patterns and improve self-driving efficiency.

Conversely, communications and social media companies need to process varied data sources in large quantities by leveraging mobile and social media content to display advertisements and recommendations; or collecting, analyzing and utilizing customers’ insights to improve products and services.

Finally, the Australian government released in 2021 the Digital Government Strategy (DGS) with the plan to deliver a digital government that meets and exceeds the expectations of Australians. One of the critical enablers is data. And the government has to deal with large volumes of information and data every day, via individual interactions and transactions, geographic mapping, strategic intelligence and machine learning.

Looking ahead

Most of the time, data scientists’ job is to predict the future using data and advanced technologies. Paradoxically, can they predict their own future? Who would have guessed the good position data scientists are today.

People keep demanding data and with a valid reason, we use it in every decision. Not only humans, machines also use data to learn and improve on the machine assisted decisions. Even to the point where we have Artificial Intelligence taking decisions on real-time systems like self-driving vehicles on the road.

The future will find data scientists collaborating more and more with other disciplines to achieve things that we never thought were even possible.

References

[1] Data science approaches to confronting the COVID-19 pandemic: a narrative review

Disclaimers

This is a personal article. The opinions expressed here represent my own and not those of my employer.