Kuration documentation¶
Kuration turns sequential source data into selected, inspectable, training-ready dataset artifacts. This documentation is organized around what you are trying to accomplish rather than the package's module layout.
Start here¶
| If you are… | Read |
|---|---|
| Evaluating kuration | Core concepts and product model |
| Building a first dataset | Guided quickstart |
| Looking for a specific operation | Pipeline guide |
| Integrating an internal workflow | Task cookbook |
Build datasets¶
- Schemas and semantics — declare fields, dimensions, media, rates, and meaning.
- Adapters and inputs — CSV, JSONL, Parquet, arrays, Hugging Face, video, and custom sources.
- Metrics and the closed DSL — derive validated per-window signals.
- Selection strategies — filter, rank, bin, cover, split, and budget.
- Outputs and artifact integrity — modalities, containers, manifests, checksums, resume, and reading data back.
Extend kuration¶
- Building operators — package a reusable curation decision, connect bounded computation, test its contract, and distribute it as a plugin.
- Customization and discovery — registries, descriptors, metrics, scorers, analyzers, adapters, encoders, methods, filters, containers, and presets.
- Architecture — internal layers, execution boundaries, and design decisions.
Operate and audit¶
- Jobs and CLI reference — validate, explain, run, inspect, compare, and gate JSON jobs.
- Cookbook: scale and reliability — streaming, parallelism, bounded checkpoints, Ray, cloud paths, failure behavior, and resume.
- Compatibility policy — supported surfaces, beta evolution, migrations, and deprecation.
- Release notes — what changed in the current release candidate and any migration considerations.
- Security policy — private reporting and runtime trust boundaries.
- Releasing — build, smoke-test, tag, and trusted-publishing process.
Reference¶
- Python API — supported public names and exact signatures.
- Pipeline operators — method forms and validation behavior.
- Job specification — declarative keys and command behavior.
- Documentation contract — audience and public-boundary rules.
Runnable examples¶
The examples/ directory contains download-free demos plus bounded
public-data workflows. Start with:
python examples/quickstart.py --out quickstart-out
python examples/05_custom_operators.py
python examples/06_gameplay_operator_methods.py
Every example used as a primary documentation path is exercised by CI.