# brevis.sh

> Open source data infrastructure in Go, in four pieces: the **Core** orchestrates
> (workflows, a persistent queue, a scheduler and an operational interface, in
> one binary that never touches the data); the **SDK** extracts and loads; the
> **Gateway** receives events over HTTP; and **SQL** models tables. A step runs
> as its own Kubernetes pod with its own image, or on a machine you already
> run. MIT licensed.

Every page below is also served as Markdown: append `index.md` to any docs URL.
`https://brevis.sh/llms-full.txt` is this entire documentation set in one file.

How the pieces fit: a **workflow** is a YAML file of **steps**; the
**scheduler** turns schedules into **runs** and a **queue** executes them; each
step becomes a **pod**, or is sent to a **host** you keep running. A step is any
command in any image — the **Go SDK**
(`brevis/sdk`) is for extract-and-load work, and the **client libraries**
(`pip install brevis`) give a step the run's context, clock and metrics through
two environment variables.

There is a third piece with a version of its own: the **ingestion gateway**
(`areteacademy/brevis-gateway`), an HTTP endpoint that lands data which arrives
instead of going to fetch it — configured entirely by YAML, delivering to
Pub/Sub, Postgres, MySQL, BigQuery, Redshift or object storage. What joins all
three is the `ingestion_id`, a frozen UUID v5 over
`provider|entity|source_key|record_ts`: a row the gateway lands and a row an SDK
fetcher lands for the same record are the same row.

The fourth piece, also versioned on its own, is **SQL**
(`areteacademy/brevis-sql`, 0.1.0): plain `.sql` files that stay valid SQL,
built into tables and views in the order their references imply, on Postgres
or BigQuery — `brevis-sql compile`, `graph`, `build` and `test`. No templating.

The same documentation in Portuguese: https://brevis.sh/docs/ (append `index.md` likewise).

## Start here

- [Ecosystem](https://brevis.sh/en/docs/ecosystem/index.md): The four brevis.sh pieces, what each one does, and where the data goes through.
- [Integrations](https://brevis.sh/en/docs/integrations/index.md): What connects to what: where data comes from, where it lands, where it is modelled and where it all runs.
- [Introduction](https://brevis.sh/en/docs/introduction/index.md): What brevis.sh is, the problem it solves, and when it is not the right choice.
- [Installation](https://brevis.sh/en/docs/installation/index.md): How to get the binary, what it requires, and how to confirm the install is correct.
- [Quickstart](https://brevis.sh/en/docs/quickstart/index.md): From nothing to a live DAG in the interface, in four steps.

## Building a workflow

- [Workflows](https://brevis.sh/en/docs/workflows/index.md): The complete YAML format — steps, dependencies, images, resources and schedule.
- [Parameters](https://brevis.sh/en/docs/parameters/index.md): What changes between two runs of the same workflow, without editing the file.
- [Step runtime](https://brevis.sh/en/docs/runtime/index.md): What a step declares about the language and tools it drives — and why you rarely need to.
- [Context between steps](https://brevis.sh/en/docs/context/index.md): What one step tells the next — two environment variables and a file.

## How it executes

- [Core](https://brevis.sh/en/docs/core/index.md): The orchestrator: schedules runs, keeps the queue in Postgres and starts one pod per step — and never touches the data.
- [Scheduler and queue](https://brevis.sh/en/docs/scheduler-and-queue/index.md): Two independent loops — who creates runs, who executes them, and why they are separate.
- [Pod per step](https://brevis.sh/en/docs/pod-per-step/index.md): Why each step runs in its own pod, and what that changes in cost and isolation.

## Command line and configuration

- [CLI](https://brevis.sh/en/docs/cli/index.md): The engine's twelve subcommands, with flags, arguments and what each one requires.
- [Configuration](https://brevis.sh/en/docs/configuration/index.md): Every environment variable, with default, format and what happens when it is missing.

## Writing a step in code

- [Go SDK](https://brevis.sh/en/docs/sdk/index.md): Writing a fetcher — extract from a source, transform, and load into a destination.
- [Client libraries](https://brevis.sh/en/docs/libraries/index.md): How a step in any language reads the context and the run's clock — and which languages already have a library.
- [Python](https://brevis.sh/en/docs/python/index.md): pip install brevis — context between steps, the run's clock and your pipeline's own metrics, with no dependencies at all.

## Receiving data over HTTP

- [Ingestion](https://brevis.sh/en/docs/ingestion/index.md): An HTTP endpoint that lands data — validate, identify, batch, deliver. Its own image, its own version.
- [Ingestion sinks](https://brevis.sh/en/docs/ingestion-sinks/index.md): The six destinations, how each database writes a record, and why the image carries only what you use.
- [Benchmarks](https://brevis.sh/en/docs/benchmarks/index.md): How we measure the gateway, what the numbers say, and the three witnesses that have to agree.

## Modelling in SQL

- [SQL](https://brevis.sh/en/docs/sql/index.md): Plain .sql files become tables and views, in dependency order — as a workflow step or on their own.

## Running it in production

- [Console](https://brevis.sh/en/docs/console/index.md): The Core's web interface: what is running, what will run, and what happened to each run.
- [Kubernetes](https://brevis.sh/en/docs/kubernetes/index.md): How to deploy the engine, with what permissions, and what to check afterwards.
- [Observability](https://brevis.sh/en/docs/observability/index.md): Prometheus metrics from both processes, what to watch, and what is deliberately absent.
- [White label](https://brevis.sh/en/docs/white-label/index.md): Changing the interface's title, motto, logo and palette through one YAML file.

## The project

- [Philosophy](https://brevis.sh/en/docs/philosophy/index.md): Where brevis.sh comes from: Seneca, Aretê Academy, and what that decides in the code.
- [FAQ](https://brevis.sh/en/docs/faq/index.md): The questions that come first, with short answers and where to go next.

## Optional

- [Repository](https://github.com/AreteAcademy/brevis): source and issues
- [Go SDK reference](https://pkg.go.dev/github.com/AreteAcademy/brevis/sdk): the generated API docs
- [Aretê Academy](https://areteacademy.com.br/): the group the project belongs to