Data Product is an exciting new concept - which means that every software vendor in the data space has their version of what data products are and the best way to build them. Each type of data product makes sense in a context - it solves specific problems for specific organizations with specific pre-existing capabilities and requirements. This talk will explore a few of the popular types of data products in those contexts and show example architectures that I’ve seen built to deliver these types of data products. By the end of the conference, you will be able to have a sensible conversation with your team and vendors; You will be discussing requirements, constraints, complexity, and costs - not buzzwords and hype.

Unfortunately, the majority of data projects fail. Yet, they fail for the same reasons. Most management and data teams don’t know the reasons a project succeeds or fails. It just appears to be random, hard work, or luck. To help understand the reasons teams succeed, we will introduce the who, what, when, where, and how of data projects. By answering these questions, teams will understand what they’re trying to accomplish far better.

Who: Data teams all start with people. This needs to be the right people, with the right skills and at the right ratios. You will need data scientists, data engineers, and operations all working together.

What: Just saying you want AI isn’t enough. You need to know what business value will be generated. There should be a clear and attainable path to value creation. You have to clearly state what you are going to do to create value.

When: Unattainable timelines aren’t feasible and neither are “when it’s ready” timeframes. Data projects need to deliver value on a sane timeline. This will include delivering in tranches so the team can gain velocity.

Where: Clusters need to be spun up somewhere. Data needs to be stored somewhere. The data needs to come from somewhere. Data teams need to have a clear plan and architecture of where each piece will be done.

How: Data teams need a clear plan that they are executing on. This plan needs a singular focus or the work will go in different directions. There need to be clear technical choices and specific technologies chosen.

Workflow orchestration has traditionally been closely coupled to the concept of Directed Acyclic Graphs (DAGs). Building data pipelines involved registering a static graph containing all the tasks and their respective dependencies. During workflow execution, this graph would be traversed and executed. The orchestration engine would then be responsible for determining which tasks to trigger based on the success and failure of upstream tasks.

This system was sufficient for standard batch processing-oriented data engineering pipelines but proved to be constraining for some emerging common use cases. Data professionals would have to compromise their vision to get their workflow to fit in a DAG. For example,

How do I re-run a part of my workflow based on a downstream condition? How do I execute a long-running workflow? How do I dynamically add tasks to the DAG during runtime? This has led to the development of Prefect Orion (Prefect 2.0), a DAG-less workflow orchestration system that emphasizes runtime flexbility and an enhanced developer experience. By removing the DAG constraint, Orion offers an interface to workflow orchestration that feels more Pythonic than ever. Developers only need to wrap as little code as they want to get observability into a specific task of the workflows.

Data pipelines are a challenging part of working in data science. Netflix Conductor is a battle tested workflow orchestration tool invented at Netflix to automate millions of microservice workflows. Using a workflow tool like Conductor as a part of your data analysis pipeline can help streamline and improve the reproducibility of your results. In this presentation we will use Conductor to automate a set of microservices to build an automated workflow for a data pipeline workflow, automating the data acquisition, cleaning and analysis.

This talk looks at Spark + Koalas, Dask, and Modin + Ray all of which attempt to provide the "holy grail" of big data – distributed pandas. We'll kick it old-school starting with "Sparkling Pandas", one of the OG distributed pandas (with the terrible performance to show for it).

No talk like this would be complete without a conflict of interest disclosure, which for me includes being one of the two original co-authors for Sparkling Pandas (and some funny stories), being a Spark committer, but this balances out with my current work on co-writing books on Dask and Ray.

At the end of this talk you will be questioning if you really want to scale pandas given all of the duct-tape involved, and have a good idea of how to choose which particular duct-taped-together solution is going to involve the least amount of rusty spoons in your eyeballs.

We all need a little optimism when it comes to climate change, but what does it look like when a team optimizes for reduced emissions? There's a huge amount of potential for data teams to benefit in surprising ways.Joining a cross functional data team at Culture Amp last year opened my mind to possibilities around communicating, monitoring and acting on climate change at work. In this talk I’ll unpack my journey, sharing ideas, anecdotes and tools to bring Carbon thinking onto the agenda.

Many scientific applications currently rely on the use of brute-force numerical methods performed on high-performance computing (HPC) infrastructure. However, these methods have their limits in many applications, e.g. climate prediction. Can artificial intelligence (AI) methods augment or even entirely replace these brute-force calculations to obtain significant speed-ups? Anima will present exciting recent advances that build new foundations in AI, viz. neural operators that are independent of the resolution or grid of training data and allow for zero-shot generalization to higher resolution evaluations. They have recently yielded 4-5 orders of magnitude speedups over standard numerical weather models.

Historically, data engineering has been relegated to batch processes running in the data warehouse or data lake - if the data didn't exist in the warehouse, it effectively didn't exist. In fact, operational systems - OLTP databases, messaging and streaming systems, caches, microservices, and so on - are not just primary sources of data, but destinations for processed data as well. In-app analytics, recommendation systems, online ML/AI infrastructure, automation, and a host of domain-specific services are just as much a part of a company's data platform as the data warehouse and BI tools. The common thread that ties these systems together is streaming and real-time data processing that handles both event-driven services and real-time data integration. While streaming systems have historically been the land of low level code and complex distributed systems, modern real-time data platforms and their associated patterns are making it just as easy to build data pipelines in SQL, on the stream, to power both operational and analytical applications. In this session we'll discuss the unified operational/analytical data platform architecture, the most common patterns, and how data engineering works in real-time.

The landscape of applied machine learning (ML) is becoming polarised. At one end are giants, complete with hundreds of people and GPUs (or TPUs) building impressive systems at scale. On the opposite end are startups, less burdened by the complexity of integrating with an existing product or the economics of turning a profit. But in the middle, established and growth companies using ML to create exceptional customer experiences walk a delicate tightrope. To succeed, they must engineer enough to match their existing products' scale and research algorithms that deliver a product experience not easily copied — all without breaking the bank.This talk explores the unique challenges of delivering impactful ML products in companies that are neither giants nor startups. We'll examine the economic constraints on innovation, how engineering choices shape the success of an ML product, and how to approach researching applied ML in new domains.

Genomics identifies your risk of developing disease and detects infectious diseases. However, to extract these clinical actionable insights, unprecedented data volumes need to be processed and done so within clinically relevant turn-around-times. The transformational bioinformatics group at CSIRO has built Big-Data and Cloud-computing solutions using machine learning and automated distribution channels to build health solutions that support globally connected health networks.

This talk outlines how we developed novel bioinformatics approaches to cater for new applications, such as genomic new-born screening and tracking emerging strains of COVID-19. The talk concludes by touching on how increasingly privacy-conscious patients inspire a re-thing of data ownership and data transactions, where distributed dynamic consent borrows from ideas from the crypto space such as decentralized autonomous organization, and immutable tokens.

Operationalising Machine learning models, particularly scaling MLOps capability across teams within an organization is a difficult feat. In this session, we show how easily you can accelerate your MLOps journey using Metaflow and Amazon SageMaker. You will learn how Carsales keeps up with an increased demand in building and productionising AI models, and their strategy to democratise AI across the whole development teams. This allows any developer to be a citizen Data Scientist and ML Engineer.

Bigger isn't always better and is definitely more likely to be irresponsible when it comes to datasets. But datasets are the unsung heroes of modern machine learning and AI systems, just as much as the algorithms, advanced hardware, and models that support them. In this talk, I share some tips and tricks from over 20 years of building and using datasets for search, natural language processing, and ML applications more generally. I discuss the importance of understanding your application task, gotchas with personalisation, the benefits of human diversity, a couple of patterns for dealing with too much or too little data, and last but not least some responsible AI considerations.