CORE WORKFLOW / FITTED PIPELINE

From raw table to reusable artifact.

This is the canonical DATADOC path. Each stage has a clear input, output, and failure mode so a developer can move from exploration to a repeatable ML preparation job.

profile → plan → fit → transformoptional evaluateJSON artifact

See every CLI option →

1. Profile the source

Start with facts, not mutations. Profiles are useful in code review because they preserve the evidence behind a transformation decision.

Terminalprofile.json is optional
datadoc profile data.csv \
  --target churn \
  --output profile.json

Profile field	Why it matters
`roles`	Explains whether a column is a target, feature, datetime, identifier, text, or constant.
`findings`	Surfaces all-null fields, duplicates, infinite values, unsupported text, and leakage clues.
`schema_fingerprint`	Lets a run record which input contract produced an artifact.

2. Create and review a plan

Plans are deterministic and do not modify data. In a team, save the plan beside the experiment configuration so reviewers can see why a column changed.

Terminalplan.json
datadoc plan data.csv \
  --target churn \
  --task classification \
  --output plan.json

Destructive behavior is explicit.

Outlier clipping is opt-in. Identifier dropping requires --drop-identifiers. High-cardinality fields produce findings and are not silently discarded.

3. Fit only on training data

The fit step learns reusable statistics and writes a JSON artifact. Use a real training split here when a model will be evaluated later.

Terminaltrain-only state
datadoc fit train.csv \
  --target churn \
  --task classification \
  --scaling auto \
  --output artifacts/churn-pipeline.json

What is learned?

Numeric medians and missingness-indicator decisions.
Categorical vocabularies or frequency maps.
Datetime parsing and stable calendar feature rules.
Optional IQR clipping bounds.
Optional standard or robust scaling centers and spreads.

4. Transform validation, test, or inference

Transform validates the artifact’s input contract and applies the saved state. It handles unseen categories without changing the output schema.

Terminalsame artifact, different rows
datadoc transform validation.csv \
  --pipeline artifacts/churn-pipeline.json \
  --output validation-features.parquet

datadoc transform new_customers.csv \
  --pipeline artifacts/churn-pipeline.json \
  --output new-customer-features.csv

Do not refit for inference.

Loading a new file into a new pipeline and calling fit changes the learned state. Use DataDocPipeline.load(...).transform(...) or the CLI artifact path.

5. Understand pipeline.json

The artifact is intentionally inspectable. It is JSON rather than a hidden binary model object.

JSONabridged artifact
{
  "artifact_version": 2,
  "datadoc_version": "0.5.0",
  "config": {"target": "churn", "task": "classification"},
  "input_schema": {"age": "Int64", "plan": "String"},
  "output_schema": {"age": "Float64", "plan__pro": "UInt8"},
  "provenance": {"schema_fingerprint": "74cfba9cbc7242be", "rows": 500, "deduplicated_rows": 0},
  "state": {
    "numeric": {"age": {"median": 35.0}},
    "categorical": {"plan": {"kind": "one_hot"}}
  }
}

Artifact loading accepts v1 and v2 and rejects anything else; transform rejects missing required source columns. Keep the artifact with the model run that consumed it.

6. Evaluate the preparation choice

Evaluation requires the ML extra and a declared target. DATADOC compares a minimal baseline with the configured candidate using cross-validation and a holdout.

Terminalclassification example
python -m pip install "datadoc-cli[ml]"
datadoc evaluate train.csv \
  --target churn \
  --task classification \
  --estimator linear \
  --output evaluation.json

Task	Primary metric	Default split
Classification	Balanced accuracy	Stratified CV + holdout
Regression	RMSE	K-fold CV + holdout
Time-aware	Same metric	Ordered split with `--time-column`
Grouped	Same metric	Group-aware split with `--group-column`

Read the whole report.

A positive score difference is not enough. Check the split strategy, estimator family, feature count, warnings, and whether an untouched external test set agrees.

7. Use the guided run

run is convenient for local experiments. It creates a profile, plan, fitted artifact, transformed Parquet, and manifest in one directory.

Terminalruns/churn-v1/
datadoc run data.csv \
  --target churn \
  --task classification \
  --output-dir runs/churn-v1 \
  --evaluate

Use fit and transform separately when you have a deliberately reserved final test set or a production inference boundary.