Skip to content

Introduction

Welcome to SynaFlow! Here's how the documentation is organized so you can find what you need.

If you're new here

Start with the Tutorial — a step-by-step guide that builds a real pipeline from scratch across four levels:

  1. Hello World — split a message into characters
  2. Lowercase Transform — convert each character with EACH mode
  3. Character Counter — count frequencies into a dictionary
  4. Key-Value Printer — iterate and print the results

Every page has Sync / Async tabs so you can follow along in either style.

Understanding the concepts

Once you've built something, dive into Core Concepts:

  • How the DAG is Wired — step names, parameter names, and type hints: the three rules that build the graph.
  • Build vs Run — the strict separation between DAG compilation and execution that enables custom runners, external orchestrator export, and deterministic behavior.
  • Lockstep Data Flow — how one item flows entirely through the pipeline before the next is produced, guaranteeing extreme memory efficiency.
  • Max In Flight — how to let a producing stream get a bounded number of items ahead of the next consumer stage without giving up lazy streaming, including real sync and async HTTP examples for I/O-bound workloads.
  • Event-Based Processing — how lazy streaming makes the framework idempotent by default and naturally suited for processing events individually or in time windows.
  • Materialization & Error Policies — when data is collected into memory, how to force materialization, and how on_error=STOP / on_error=CONTINUE affect the data flow.
  • Semantic Naming — smart binding: using singular, plural, and suffixes naturally without exact name matching.
  • DAG Construction — the build-time validation, JSON export, and execution levels.
  • Sync & Async Parity — identical semantics for sync and async, and how to convert between them.
  • Examples — auto-generated diagrams and source code for every pipeline in the test corpus.

Going deeper

The Advanced section covers:

  • Custom Materializers — write your own disk, database, or cloud-backed collectors.
  • Custom Observers — monitor pipeline and step lifecycle events for logging, metrics, or tracing.
  • Export Guidance — compile SynaFlow DAGs into Airflow, Prefect, or custom orchestrators.

If you're coming from another ecosystem, the Comparisons section maps SynaFlow concepts to Java Streams and LINQ.

Quick reference

from collections.abc import Generator, Iterator
from typing import NamedTuple
from synaflow import pipeline, step, run

class Params(NamedTuple):
    count: int

def producer(count: int) -> Generator[int, None, None]:
    yield from range(count)

def consumer(producer: Iterator[int]) -> None:
    for x in producer:
        print(x)

p = pipeline(
    name="quickstart",
    params=Params,
    steps=[
        step("producer", fn=producer),
        step("consumer", fn=consumer),
    ],
)

run(p, Params(count=5))

Next

Jump into the Tutorial and build your first pipeline.