INTERFACES / PYTHON SDK

Build the pipeline into your application.

The SDK exposes the same safe lifecycle as the CLI, but returns typed Python objects and Polars DataFrames for notebooks, services, and scheduled jobs.

from datadoc import DataDocPipelinePolars DataFrameJSON save/load

Copy the quickstart →

1. Install

Terminalcore + optional ML
python -m pip install datadoc-cli
python -m pip install "datadoc-cli[ml]"

The core SDK does not import scikit-learn until evaluate is called. This keeps preparation usable offline and keeps optional dependencies explicit.

2. Quickstart

Pythonprofile → plan → fit → transform
import polars as pl
from datadoc import DataDocPipeline, PipelineConfig

train = pl.read_csv("train.csv")
validation = pl.read_csv("validation.csv")

config = PipelineConfig(
    target="churn",
    task="classification",
    scaling="standard",
    clip_outliers=False,
    random_seed=42,
)

pipeline = DataDocPipeline(config)
profile = pipeline.profile(train)
plan = pipeline.plan(train)
pipeline.fit(train)

features = pipeline.transform(validation)
pipeline.save("artifacts/churn-pipeline.json")
print(profile.to_dict())
print(plan.to_dict())
print(features.shape)

The target stays available.

DATADOC protects the target from feature operations but leaves it in the returned frame. Separate X and y in the model layer when you are ready.

3. Configure deliberate behavior

Pythoncommon controls
config = PipelineConfig(
    target="churn",
    task="classification",       # or regression
    scaling="robust",             # none, standard, robust, auto
    drop_identifiers=False,
    deduplicate=False,
    clip_outliers=False,
    rare_category_min_frequency=0.0,  # e.g. 0.02 groups rare into __RARE__
    datetime_cyclical=False,
    datetime_extract_hour=True,
    categorical_threshold=20,
    estimator_family="linear",    # linear | tree | both
    time_column=None,
    group_column=None,
    strict_schema=True,
    random_seed=42,
)

Config	Default	Use it when
`target`	`None`	You have a supervised label to protect.
`scaling`	`auto`	Linear or distance-based models need normalized features.
`drop_identifiers`	`False`	You have reviewed the profile and want explicit removal.
`strict_schema`	`True`	You want inference failures to be visible rather than silently coerced.

4. Save and load across processes

Pythonartifact boundary
pipeline.save("artifacts/pipeline.json")

loaded = DataDocPipeline.load("artifacts/pipeline.json")
features = loaded.transform(new_rows)

Artifacts (v2) include the configuration, input schema, output schema, plan, profile, train provenance (schema fingerprint, row counts, deduplicated rows, package version), and fitted state. Loading accepts v1 and v2 and fails only on unsupported versions. Extra helpers: explain_plan(df), drift_report(df), evaluate_ablation(df), profile_to_html(df) for notebooks, and export_sklearn_artifact(path) for joblib portability.

5. Evaluate from Python

Pythonrequires datadoc-cli[ml]
report = DataDocPipeline(
    PipelineConfig(
        target="churn",
        task="classification",
        estimator_family="linear",
        time_column=None,
        group_column=None,
    )
).evaluate(train, target="churn")

print(report.to_dict())

The report names the task, metric, estimator family, split strategy, baseline score, selected score, improvement, feature count, selected pipeline, and warnings.

Use an external final test set.

Built-in evaluation gives structured evidence on the supplied data. Keep the final production test set untouched until you have selected the preparation strategy.

6. AI Advisory in Python

Pythonhypotheses & missingness audit
from datadoc.ai import AIExplainer, explain_dataset

# One-liner analysis:
explanation = explain_dataset("train.csv", target="churn", model="gpt-4o-mini")
print(explanation.summary)
for hyp in explanation.feature_hypotheses:
    print(f"Feature: {hyp.feature_name} = {hyp.formula}")
    print(f"Rationale: {hyp.rationale}")

# Or via explainer instance:
explainer = AIExplainer(model="gemini/gemini-2.0-flash")
res = explainer.explain_file("train.csv", target="churn")
print(res.missingness_mechanisms)

Requires datadoc-cli[ai] (LiteLLM) and valid API credentials. Returns structured, validated dataclass schemas without executing unvetted code.

7. Handle SDK errors

Pythonactionable failure
from datadoc import DataDocPipeline, PipelineConfig
from datadoc.core.pipeline import DataDocError

try:
    pipeline = DataDocPipeline(
        PipelineConfig(target="churn")
    ).fit(train)
    features = pipeline.transform(inference)
except DataDocError as error:
    print(f"DATADOC input contract failed: {error}")

Missing target: pass the correct target name or use an unsupervised preparation flow.
Missing source columns: compare the input file with pipeline.input_schema_.
Unseen category: safe by default; the output schema remains stable.
Unsupported file: read it into Polars, then pass the DataFrame to the SDK.