INTERFACES / CLI DEEP DIVE & REFERENCE

The Command Line Interface, Completely.

DATADOC provides a local-first, zero-leakage CLI for tabular dataset preparation. Explore parameters, execution mechanics, command breakdowns, and interactive demo testing.

datadoc --help Polars Powered Zero Data Leakage v0.6.1

0. Interactive Demo Sandbox & Test Drive

Download our official 500-row demo.csv test dataset and watch how DATADOC profiles, plans, fits, transforms, and evaluates the dataset step-by-step.

Download Test Dataset (demo.csv)

Includes missing values, scale mismatches, UUID identifiers, string dates, and extreme outliers (age=999).

๐Ÿ“ฅ Download demo.csv (46 KB)

Raw Input Dataset: demo.csv (First 5 Rows)

Notice the dirty features requiring engineering before machine learning model training:

customer_id (ID) name (Text) age salary department city signup_date churn (Target)
dba51b99... Person_0 23 57,098 Engineering Hyderabad 2020-07-10 0
4f57a639... Person_1 56 83,878 Engineering Bangalore 2021-11-20 0
10eb978b... Person_2 60 30,695 Support Chennai 2023-07-11 1
e22b64ba... Person_3 null 62,930 Sales Chennai 2022-01-28 1
b88ae03d... Person_4 999 (Outlier) 47,431 Sales Bangalore 2023-02-14 0

Step 1: datadoc profile demo.csv --target churn

Profiles dataset health, missing value counts, cardinality, column roles, and generates a unique schema fingerprint.

JSONdatadoc profile output
{ "rows": 500, "columns": 10, "schema": { "customer_id": "String", "name": "String", "age": "Int64", "salary": "Int64", "department": "String", "city": "String", "signup_date": "String", "churn": "Int64" }, "null_counts": { "age": 25, "salary": 20, "department": 15 }, "roles": [ { "name": "customer_id", "role": "identifier" }, { "name": "name", "role": "identifier" }, { "name": "age", "role": "feature_numeric" }, { "name": "salary", "role": "feature_numeric" }, { "name": "department", "role": "feature_categorical" }, { "name": "churn", "role": "target" } ], "schema_fingerprint": "a3f890e12d..." }

Step 2: datadoc plan demo.csv --target churn --drop-identifiers

Formulates an explicit, explainable sequence of transformation operations with analytical rationales before modifying data.

JSONdatadoc plan output
{ "target": "churn", "operations": [ { "name": "DropIdentifiers", "columns": ["customer_id", "name"], "rationale": "High-cardinality string IDs overfit models and carry zero predictive signal." }, { "name": "ImputeNumeric", "columns": ["age", "salary"], "strategy": "median", "rationale": "Fill missing numerical values with training median (outlier-resistant)." }, { "name": "ImputeCategorical", "columns": ["department"], "strategy": "mode", "rationale": "Fill missing categorical entries with the most frequent value." }, { "name": "DatetimeExtract", "columns": ["signup_date"], "features": ["month", "day", "dayofweek"], "rationale": "Extract temporal sub-components from date strings." }, { "name": "OneHotEncode", "columns": ["department", "city"], "rationale": "Convert categorical strings into numeric binary indicator vectors." } ] }

Step 3: datadoc fit demo.csv --target churn --scaling standard --clip-outliers --output pipeline.json

Learns medians, modes, Z-score parameters, and IQR bounds strictly from training data and saves pipeline.json.

JSONpipeline.json artifact
{ "artifact_version": 1, "config": { "target": "churn", "drop_identifiers": true, "scaling": "standard", "clip_outliers": true }, "state": { "dropped": ["customer_id", "name"], "medians": { "age": 36.0, "salary": 78500.0 }, "modes": { "department": "Engineering" }, "scaling": { "age": { "mean": 36.4, "std": 11.2 }, "salary": { "mean": 79200.0, "std": 31500.0 } }, "outliers": { "age": { "lower": 12.0, "upper": 75.0 } } } }

Step 4: datadoc transform demo.csv --pipeline pipeline.json --output clean_demo.csv

Before vs After summary showing how dirty features transformed into clean, model-ready numerical columns:

Feature Column Raw Input State Transformed Output State Transformation Applied
customer_id / name UUIDs & Names Dropped Identified as string IDs and removed
age Nulls + Outlier (999) -1.19 (Z-score) Null โ†’ Median (36.0), Clipped (75.0), Standardized
salary Nulls present -0.68 (Z-score) Null โ†’ Median (78,500), Standardized
department "Engineering" dept_Engineering=1, dept_Sales=0 One-Hot Encoded into binary indicator vectors
signup_date "2020-07-10" month=7, day=10, dayofweek=4 Extracted numeric temporal features
churn (Target) 0 or 1 0 or 1 (Untouched) Protected target column preserved exactly

Step 5: datadoc evaluate demo.csv --target churn --task classification --estimator tree

Runs 80/20 train/test evaluation benchmark to measure clean vs raw accuracy improvement:

JSON & Terminaldatadoc evaluate benchmark
{ "task": "classification", "metric": "balanced_accuracy", "baseline_score": 0.542, "selected_score": 0.815, "improvement": +0.273, "selected_pipeline": "candidate", "feature_count": 16 } Baseline Score : โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡ 0.542 Candidate Score: โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡ 0.815 (+27.3% boost!)

1. Core Concepts & Terminology

Before running CLI commands, it is essential to understand the key parameters and arguments that govern DATADOC's behavior.

Target Column (--target)

The target is the column you intend to predict (e.g., churn, salary, price). When you declare --target <column>, DATADOC applies a protection boundary to that column. The target is never imputed, scaled, encoded, or clipped. It passes through unchanged so your evaluation metrics and model training remain uncorrupted.

Example: In a subscription dataset, churn contains 0 (customer stayed) or 1 (customer left). Passing --target churn prevents DATADOC from scaling or modifying these class labels.

Task Type (--task)

Defines the nature of the supervised learning task. DATADOC uses this to determine evaluation metrics and optimal default preprocessing routines:

  • auto (Default): Automatically infers task type. If the target has โ‰ค 20 unique values or is boolean/string, it selects classification; otherwise, regression.
  • classification: Supervised binary or multi-class prediction (evaluates via balanced accuracy / log loss).
  • regression: Continuous numeric prediction (evaluates via RMSE / Rยฒ).

Identifier Dropping (--drop-identifiers)

High-cardinality string IDs (such as customer_id, uuid, user_name) carry no generalizable predictive signal and cause severe overfitting if one-hot encoded. Passing --drop-identifiers instructs DATADOC to detect string IDs and remove them during transformation.

Feature Scaling Strategies (--scaling)

Tabular numeric features often vary by orders of magnitude (e.g., age spans 18โ€“80 while income spans 20,000โ€“500,000). Scaling normalizes these ranges:

Mode Mathematical Formula Best Used For
auto Picks standard for linear models; none for tree models Default โ€” recommended for most workflows
none Xout = X Tree-based algorithms (RandomForest, XGBoost, LightGBM)
standard Z = (X - ฮผ) / ฯƒ Linear Regression, Logistic Regression, Neural Networks, SVMs
robust Z = (X - Q2) / IQR Datasets containing unclipped, heavy-tailed outliers

Outlier Clipping (--clip-outliers)

When enabled, DATADOC calculates upper and lower Interquartile Range (IQR) bounds during fit and clips extreme values during transform:

Lower Bound = Q1 - (1.5 ร— IQR)
Upper Bound = Q3 + (1.5 ร— IQR)

Values below the lower bound are clamped to the lower bound; values above the upper bound are clamped to the upper bound.

Split Strategies (--time-column & --group-column)

  • --time-column <col>: Performs an ordered temporal split (oldest rows in train, newest in test) to prevent future-data leakage in time-series tabular problems.
  • --group-column <col>: Performs group-aware splitting so that all records belonging to a single group (e.g., all visits by patient_id) reside strictly in train or test.

2. The Pipeline Lifecycle

DATADOC operates on a strict 4-stage lifecycle designed to guarantee zero data leakage between training, validation, and production inference sets.

flowchart LR A["Raw Dataset (CSV/Parquet)"] --> B["datadoc profile"] B --> C["datadoc plan"] C --> D["datadoc fit (Train Data Only)"] D --> E["pipeline.json (Artifact)"] E --> F["datadoc transform"] F --> G["Clean Dataset"]
01 Profile Inspect schema, missingness, cardinality, data roles, and compute a schema fingerprint.
02 Plan Formulate a deterministic sequence of operations with explicit analytical rationales.
03 Fit Learn parameters (medians, mode vocabularies, scaling means/stds) strictly from training rows and persist to JSON.
04 Transform Apply the saved JSON artifact immutably to validation, test, or production inference data.

3. Read-Only Commands

PROFILE datadoc profile <file_path>

Inspects a CSV or Parquet file without making any modifications. Generates detailed health reports, column role assignments, missing value counts, and schema fingerprints.

TerminalInspect dataset quality
datadoc profile demo.csv --target churn --output profile.json

Options

OptionTypeDefaultDescription
file_pathArgumentRequiredPath to CSV or Parquet input file
--targetOptionNoneDeclare prediction target column to apply protection
--outputOptionNonePath to write profile output JSON
PLAN datadoc plan <file_path>

Generates a transparent, step-by-step transformation plan without modifying the dataset or learning statistics.

TerminalGenerate transformation plan
datadoc plan demo.csv --target churn --drop-identifiers --output plan.json

Options

OptionTypeDefaultDescription
file_pathArgumentRequiredInput CSV or Parquet dataset
--targetOptionNoneProtected target column name
--taskOptionautoauto, classification, or regression
--drop-identifiersFlagFalseInclude identifier dropping in transformation operations
--outputOptionNonePath to save plan JSON file
REPORT datadoc report <file_path>

Generates an automated, standalone, interactive HTML data health and preparation audit report for sharing with teammates. Completely self-contained with health grade scoring, distribution histograms, null rates, and transformation plans.

TerminalGenerate standalone HTML report
datadoc report train.csv --target churn --output report.html

Options

OptionTypeDefaultDescription
file_pathArgumentRequiredPath to CSV or Parquet input dataset
--targetOptionNoneDeclare target column for protection and analysis
--outputOption<stem>_report.htmlPath to write standalone HTML report
--titleOptionNoneCustom title for report header
--presetOptionNonePreset to base recommendations on
--aiFlagFalseInclude AI Executive Summary & Feature Hypotheses in report
--ai-modelOptionNoneModel to use for AI report analysis
--open / --no-openFlagTrueOpen in browser automatically
COMPARE datadoc compare <raw_path> <transformed_path>

Visually compares raw vs. transformed datasets side-by-side. Displays dimension changes, missing value reduction (100% resolution tracking), column lifecycle (retained, dropped, engineered), and numeric distribution shifts.

TerminalCompare datasets side-by-side
datadoc compare raw.csv clean.csv --target churn --html comparison.html

Options

OptionTypeDefaultDescription
raw_pathArgumentRequiredPath to raw CSV or Parquet file
transformed_pathArgumentRequiredPath to transformed CSV or Parquet file
--targetOptionNoneTarget column name
--htmlOptionNonePath to save visual HTML comparison page
--jsonFlagFalseOutput comparison data as structured JSON
--openFlagFalseOpen HTML comparison in browser
EXPLAIN datadoc explain <file_path>

AI-powered dataset feature engineering hypotheses, semantic role inference, missingness mechanism analysis (MCAR/MAR/MNAR), and target leakage audit using LiteLLM (OpenAI, Gemini, Anthropic, Ollama).

TerminalAI Feature Hypotheses & Audit
datadoc explain raw.csv --target churn --model gpt-4o-mini --recommend-config

Options

OptionTypeDefaultDescription
file_pathArgumentRequiredPath to CSV or Parquet file to analyze
--targetOptionNoneTarget column for predictive task
--modelOptionNoneLLM model (e.g., gpt-4o-mini, gemini/gemini-2.0-flash, ollama/llama3)
--outputOptionNoneSave AI markdown explanation to file
--recommend-configFlagFalseGenerate and save recommended datadoc.toml
HEALTH datadoc health <file_path>

Fast dataset health audit displaying 0โ€“100 quality score, letter grade (A+ to F), and detected data quality findings.

TerminalQuick health check
datadoc health demo.csv --target churn

4. Build Commands

FIT datadoc fit <file_path>

Fits transformation rules (imputation medians, encodings, clipping bounds) strictly on the provided training split and saves a serializable pipeline.json artifact.

TerminalFit pipeline on training data
datadoc fit train.csv \ --target churn \ --task classification \ --drop-identifiers \ --scaling standard \ --clip-outliers \ --output artifacts/pipeline.json

Options

OptionTypeDefaultDescription
file_pathArgumentRequiredPath to training CSV/Parquet file
--targetOptionNoneSupervised target column
--taskOptionautoauto, classification, regression
--drop-identifiersFlagFalseDrop string IDs during pipeline execution
--scalingOptionautoauto, none, standard, robust
--clip-outliersFlagFalseOpt into upper/lower IQR outlier clipping
--outputOptionpipeline.jsonPath to save fitted JSON artifact
TRANSFORM datadoc transform <file_path>

Applies a saved, fitted pipeline artifact to validation, test, or production inference data. Guarantees that validation rows are transformed using training-derived statistics.

TerminalApply pipeline to validation data
datadoc transform test.csv \ --pipeline artifacts/pipeline.json \ --output clean_test.parquet

Options

OptionTypeDefaultDescription
file_pathArgumentRequiredPath to dataset to transform
--pipelineOptionRequiredPath to fitted pipeline.json artifact
--outputOptionRequiredOutput file path (.csv or .parquet)
EXPORT datadoc export

Exports a standalone, self-contained Python script (pipeline.py) wrapping the fitted artifact for zero-dependency inference in production environments.

TerminalExport standalone Python script
datadoc export \ --pipeline artifacts/pipeline.json \ --output my_pipeline.py

Options

OptionTypeDefaultDescription
--pipelineOptionRequiredPath to fitted pipeline.json artifact
--outputOptionpipeline.pyPath to write standalone Python script

5. Measure & Orchestration Commands

EVALUATE datadoc evaluate <file_path>

Performs a strict 80/20 train/test evaluation comparing a baseline model on raw data against a candidate model trained on DATADOC-cleaned data.

flowchart TD Data["Raw Dataset"] --> Split["80 / 20 Train-Test Split"] Split --> RawTrain["80% Raw Train"] Split --> RawTest["20% Raw Test"] RawTrain --> BaseFit["Train Baseline Model"] RawTest --> BaseEval["Evaluate Baseline Score"] RawTrain --> PipeFit["Fit Pipeline (Leakage-Safe)"] PipeFit --> CleanTrain["Transform 80% Train"] PipeFit --> CleanTest["Transform 20% Test"] CleanTrain --> CandFit["Train Candidate Model"] CleanTest --> CandEval["Evaluate Candidate Score"] BaseEval --> Comp["Compare & Report Delta"] CandEval --> Comp
TerminalRun evaluation benchmark
datadoc evaluate demo.csv \ --target churn \ --task classification \ --estimator tree \ --output eval_report.json

Options

OptionTypeDefaultDescription
file_pathArgumentRequiredInput CSV or Parquet file
--targetOptionRequiredSupervised target column to evaluate
--taskOptionautoauto, classification, regression
--estimatorOptionlinearlinear, tree, or both
--time-columnOptionNoneColumn name for ordered temporal splits
--group-columnOptionNoneColumn name for group-aware splits
--outputOptionNonePath to save evaluation report JSON
RUN datadoc run <file_path>

Orchestrates a complete, one-shot local run: profiles, plans, fits, transforms, and generates a reproducible run directory with manifest files.

TerminalExecute reproducible local run
datadoc run demo.csv \ --target churn \ --output-dir datadoc-run \ --evaluate

Options

OptionTypeDefaultDescription
file_pathArgumentRequiredInput dataset path
--targetOptionNoneTarget column name
--taskOptionautoauto, classification, regression
--output-dirOptiondatadoc-runDirectory path to persist run artifacts
--evaluateFlagFalseInclude evaluation benchmark in run output

6. Dashboard & Utility Commands

UI datadoc ui <file_path>

Launches a local FastAPI backend server and opens the interactive Web Dashboard in your default browser.

TerminalLaunch Web Dashboard
datadoc ui demo.csv --port 8000

Options

OptionTypeDefaultDescription
file_pathArgumentRequiredDataset file to load in dashboard
--portOption8000HTTP port for local dashboard server
VERSION datadoc version

Displays the current installed version of datadoc-cli.

TerminalCheck DATADOC version
datadoc version
WIZARD datadoc wizard <file_path>

Guided setup: asks for target, preset, identifier/dedup policy, scaling, clipping, and rare-category frequency, writes datadoc.toml, then runs the full pipeline.

TerminalGuided first run
datadoc wizard train.csv --output-dir datadoc-run

Options

OptionTypeDefaultDescription
file_pathArgumentRequiredTraining CSV or Parquet file
--output-dirOptiondatadoc-runDirectory for run outputs
INIT datadoc init

Writes a starter datadoc.toml so runs are reproducible without retyping flags. Settings can also live under [tool.datadoc] in pyproject.toml; explicit flags always win.

TerminalRepeatable config
datadoc init --preset balanced --output datadoc.toml
LINT datadoc lint <file_path>

Lints a dataset for leakage and prep pitfalls: target duplication, null targets, infinite values, duplicate rows, constants, and profile findings.

TerminalLeakage lint
datadoc lint train.csv --target churn
DIFF datadoc diff <file_a> <file_b>

Diffs two profile, plan, or pipeline JSON artifacts top-level key by key, plus added/removed plan operations.

TerminalCompare artifacts
datadoc diff plan-v1.json plan-v2.json
PLUGINS datadoc plugins list

Lists the 7 registered deterministic plugins in priority order, including third-party entry-point plugins. Use datadoc plugins show <Name> for detail.

TerminalRegistry introspection
datadoc plugins list

7. Step-by-Step Hands-On Tutorial

Follow this sequence using the downloadable demo.csv file to test all features of the DATADOC CLI. Prefer guidance? Start with datadoc wizard demo.csv instead of steps 1โ€“3.

BashFull CLI workflow test sequence
# Step 1: Inspect dataset quality and health findings datadoc profile demo.csv --target churn # Step 2: Preview the proposed transformation plan datadoc plan demo.csv --target churn --drop-identifiers # Step 3: Fit the pipeline on training data and save artifact datadoc fit demo.csv \ --target churn \ --drop-identifiers \ --scaling standard \ --clip-outliers \ --output artifacts/pipeline.json # Step 4: Transform new validation dataset using saved artifact datadoc transform demo.csv \ --pipeline artifacts/pipeline.json \ --output clean_demo.csv # Step 5: Benchmark candidate vs baseline ML performance (with ablation) datadoc evaluate demo.csv \ --target churn \ --task classification \ --estimator tree \ --ablation # Step 6: Export executable Python wrapper script datadoc export \ --pipeline artifacts/pipeline.json \ --output my_pipeline.py # Step 7: Run complete reproducible local run in one command datadoc run demo.csv \ --target churn \ --output-dir full-run \ --evaluate # Step 8: Open interactive Web Dashboard datadoc ui demo.csv

8. Master Options Reference Table

Below is a quick reference matrix of all options across DATADOC CLI commands:

Flag / Option Commands Supported Allowed Values Default Description
--target profile, plan, fit, evaluate, run Column Name None Declares protected prediction target column
--task plan, fit, evaluate, run auto, classification, regression auto Specifies machine learning problem type
--drop-identifiers plan, fit, run Boolean Flag False Detects and removes high-cardinality string IDs
--identifier-column plan, fit, run Column Name (repeatable) None Forces a column into the identifier role (e.g. PassengerId)
--ignore-column plan, fit, run Column Name (repeatable) None Excludes a column from features entirely
--scaling fit, run auto, none, standard, robust auto Selects numeric feature scaling strategy
--clip-outliers fit, run Boolean Flag False Clamps numerical outliers to IQR bounds
--estimator evaluate linear, tree, both linear Selects model family; both fits linear and tree and keeps the stronger validation score
--preset profile, plan, fit, evaluate, run quick, balanced, linear, tree, time, robust None One-word policy bundle for scaling, clipping, threshold, and estimator family
--deduplicate fit, run Boolean Flag False Drops duplicate rows from training data at fit time
--rare-frequency fit, run Float 0โ€“1 0.0 Groups categories below this frequency into __RARE__
--cyclical fit, run Boolean Flag False Adds sin/cos pairs for datetime parts
--no-hour fit, run Boolean Flag False Skips hour extraction even when time data is present
--categorical-threshold fit, run Integer 20 Max one-hot width before falling back to frequency encoding
--ablation evaluate, run Boolean Flag False Compares full, no-clip, no-scaling, and minimal variants
--validate transform Boolean Flag False Runs schema validation and median-shift drift checks
--explain profile, plan Boolean Flag False Renders human-readable tables instead of raw JSON
--compare profile File Path (.json) None Compares against another profile JSON
--diff plan File Path (.json) None Diffs against another plan JSON
--config profile, plan, fit, evaluate, run File Path (.toml) Auto-discovered Explicit datadoc.toml path; explicit flags override it
--format export python, joblib python Executable wrapper script or portable joblib artifact dict
--time-column evaluate Column Name None Enforces ordered temporal train/test split
--group-column evaluate Column Name None Enforces group-aware train/test split
--pipeline transform, export File Path (.json) Required Path to fitted pipeline JSON artifact
--output profile, plan, fit, transform, evaluate, export File Path Varies Path to write command output artifact
--output-dir run Directory Path datadoc-run Directory path for reproducible run outputs
--port ui Integer 8000 HTTP port for local dashboard server (--no-browser skips auto-open)

9. Error Recovery & Best Practices

Holdout Score Warning in Evaluation

Evaluation scores are computed via an 80/20 holdout split. Use an external final test set for critical production decisions.

Missing Input Columns during Transform

If datadoc transform fails with a schema mismatch error, ensure the input dataset contains all feature columns present when datadoc fit was executed.