liveBook · Manning

Flyte

As explained in Machine Learning System Design

Flyte is an open-source workflow orchestration platform for defining, executing, and monitoring data and machine-learning workflows.

Flyte serves as the orchestration layer for recurring machine-learning data jobs, especially when production data must be fetched and prepared on a schedule. section 6.7.2

A practical Flyte-managed pipeline can:

  • Fetch training data from production databases daily in batch mode. section 6.7.2
  • Store the gathered data separately as Parquet files for simpler downstream reads. section 6.7.2
  • Support full-dataset processing while allowing subsampling when class balance requires it. section 6.7.2

PhotoStock training-data pipeline

A PhotoStock workflow uses Flyte to orchestrate daily extraction from production databases, preserving identifiers such as user_id, seller_id, session_id, and purchase_id for later joins. section 6.7.2

How Machine Learning System Design covers it

How deep it goes
Positions Flyte as the orchestration choice without detailing workflow definitions, execution internals, or monitoring.
Examples
Uses daily training-data extraction, identifier preservation, Parquet storage, and class-balance subsampling as concrete pipeline decisions.
What it assumes
Assumes access to production databases and an existing batch-job environment, while keeping preprocessing requirements simple.
What you can do after
Operate/configure scheduled data-gathering jobs that produce separately stored training datasets.
How it connects
Connects Flyte with ETL, data preparation, batch processing, production databases, and Parquet files.
Cover of Machine Learning System Design
liveBookmanning.com
Read Machine Learning System Design on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.