CORE CONCEPTS / HOW IT WORKS

A pipeline that can explain itself.

DATADOC separates observation, decision, learning, and application. That separation is the product: users can see what will happen, inspect the learned state, and apply it consistently later.

Read-only profile firstTrain-only fitVersioned JSON

Follow the full pipeline →

Profile: observe the dataset

Profiling is a factual pass. It does not impute, encode, scale, clip, or drop your columns. It records enough context to make the next decision inspectable.

SCHEMA

Types and fingerprint

Column names, Polars dtypes, row/column counts, and a stable schema fingerprint.

QUALITY

Findings

Nulls, duplicates, constants, infinite values, mixed types, and unsupported text warnings.

ROLES

Confidence + rationale

Target, numeric, categorical, datetime, identifier, text, ignored, and constant roles.

Terminalread-only
datadoc profile data.csv \
  --target churn \
  --output profile.json

Identifier decisions are not guesses.

A numeric column is not an ID just because every value is unique. DATADOC combines names and values, reports confidence, and only drops identifiers when drop_identifiers is explicitly enabled.

Plan: review proposed work

A plan converts findings into ordered operations and reasons. It is safe to show to a teammate or include in a pull request before fitting anything.

Finding	Default operation	Policy
Numeric missing values	Training median + indicator	Fit only on training rows.
Categorical values	Training vocabulary + unseen-safe encoding	High cardinality is reported, not silently deleted.
Datetime strings	Calendar features if parse confidence passes	Preserve parse warnings.
Outliers	No clipping by default	Enable IQR clipping deliberately.
Identifiers	Retain by default	Drop only by explicit policy.

JSON shapeplan.json
{
  "operations": [
    {"operation": "numeric_imputation", "column": "age",
     "reason": "Median learned from training data."}
  ],
  "findings": [],
  "protected_columns": ["churn"]
}

Fit and transform: learn once, apply many times

fit learns state. transform consumes state. That boundary is the rule that prevents test-set leakage.

ATraining rowsCalculate medians, vocabularies, bounds, and scaling centers.

Bpipeline.jsonSerialize configuration, input schema, output schema, and fitted state.

CNew rowsValidate compatibility and apply exactly the saved state.

PythonSDK lifecycle
pipeline = DataDocPipeline(config)
pipeline.fit(train_df)          # learns state
validation = pipeline.transform(validation_df)  # applies state
pipeline.save("pipeline.json")

Target columns are protected.

If you declare target="churn", DATADOC does not impute, encode, scale, clip, or drop it as a feature. The transformed frame keeps the target so your code can separate labels explicitly.

Leakage safety in plain language

Leakage happens when a model gets information it would not have at prediction time. A common example is calculating an imputation median from train and test rows together.

Avoidtest data influences fit
all_rows = pl.concat([train, test])
median = all_rows["income"].median()
train = train.with_columns(pl.col("income").fill_null(median))
test = test.with_columns(pl.col("income").fill_null(median))

Usefit state is frozen
pipeline = DataDocPipeline(
    PipelineConfig(scaling="none")
).fit(train)
test_features = pipeline.transform(test)

Evaluation is evidence, not a guarantee.

DATADOC compares a baseline and candidate under a declared split. It can report “no improvement found,” which is often the correct conclusion.