The Command Line Interface, Completely.
DATADOC provides a local-first, zero-leakage CLI for tabular dataset preparation. Explore parameters, execution mechanics, command breakdowns, and interactive demo testing.
0. Interactive Demo Sandbox & Test Drive
Download our official 500-row demo.csv test dataset and watch how DATADOC profiles, plans, fits, transforms, and evaluates the dataset step-by-step.
Download Test Dataset (demo.csv)
Includes missing values, scale mismatches, UUID identifiers, string dates, and extreme outliers (age=999).
Raw Input Dataset: demo.csv (First 5 Rows)
Notice the dirty features requiring engineering before machine learning model training:
| customer_id (ID) | name (Text) | age | salary | department | city | signup_date | churn (Target) |
|---|---|---|---|---|---|---|---|
dba51b99... |
Person_0 | 23 | 57,098 | Engineering | Hyderabad | 2020-07-10 | 0 |
4f57a639... |
Person_1 | 56 | 83,878 | Engineering | Bangalore | 2021-11-20 | 0 |
10eb978b... |
Person_2 | 60 | 30,695 | Support | Chennai | 2023-07-11 | 1 |
e22b64ba... |
Person_3 | null | 62,930 | Sales | Chennai | 2022-01-28 | 1 |
b88ae03d... |
Person_4 | 999 (Outlier) | 47,431 | Sales | Bangalore | 2023-02-14 | 0 |
Step 1: datadoc profile demo.csv --target churn
Profiles dataset health, missing value counts, cardinality, column roles, and generates a unique schema fingerprint.
JSONdatadoc profile output{ "rows": 500, "columns": 10, "schema": { "customer_id": "String", "name": "String", "age": "Int64", "salary": "Int64", "department": "String", "city": "String", "signup_date": "String", "churn": "Int64" }, "null_counts": { "age": 25, "salary": 20, "department": 15 }, "roles": [ { "name": "customer_id", "role": "identifier" }, { "name": "name", "role": "identifier" }, { "name": "age", "role": "feature_numeric" }, { "name": "salary", "role": "feature_numeric" }, { "name": "department", "role": "feature_categorical" }, { "name": "churn", "role": "target" } ], "schema_fingerprint": "a3f890e12d..." }
Step 2: datadoc plan demo.csv --target churn --drop-identifiers
Formulates an explicit, explainable sequence of transformation operations with analytical rationales before modifying data.
JSONdatadoc plan output{ "target": "churn", "operations": [ { "name": "DropIdentifiers", "columns": ["customer_id", "name"], "rationale": "High-cardinality string IDs overfit models and carry zero predictive signal." }, { "name": "ImputeNumeric", "columns": ["age", "salary"], "strategy": "median", "rationale": "Fill missing numerical values with training median (outlier-resistant)." }, { "name": "ImputeCategorical", "columns": ["department"], "strategy": "mode", "rationale": "Fill missing categorical entries with the most frequent value." }, { "name": "DatetimeExtract", "columns": ["signup_date"], "features": ["month", "day", "dayofweek"], "rationale": "Extract temporal sub-components from date strings." }, { "name": "OneHotEncode", "columns": ["department", "city"], "rationale": "Convert categorical strings into numeric binary indicator vectors." } ] }
Step 3: datadoc fit demo.csv --target churn --scaling standard --clip-outliers --output pipeline.json
Learns medians, modes, Z-score parameters, and IQR bounds strictly from training data and saves pipeline.json.
JSONpipeline.json artifact{ "artifact_version": 1, "config": { "target": "churn", "drop_identifiers": true, "scaling": "standard", "clip_outliers": true }, "state": { "dropped": ["customer_id", "name"], "medians": { "age": 36.0, "salary": 78500.0 }, "modes": { "department": "Engineering" }, "scaling": { "age": { "mean": 36.4, "std": 11.2 }, "salary": { "mean": 79200.0, "std": 31500.0 } }, "outliers": { "age": { "lower": 12.0, "upper": 75.0 } } } }
Step 4: datadoc transform demo.csv --pipeline pipeline.json --output clean_demo.csv
Before vs After summary showing how dirty features transformed into clean, model-ready numerical columns:
| Feature Column | Raw Input State | Transformed Output State | Transformation Applied |
|---|---|---|---|
customer_id / name |
UUIDs & Names | Dropped | Identified as string IDs and removed |
age |
Nulls + Outlier (999) | -1.19 (Z-score) |
Null โ Median (36.0), Clipped (75.0), Standardized |
salary |
Nulls present | -0.68 (Z-score) |
Null โ Median (78,500), Standardized |
department |
"Engineering" | dept_Engineering=1, dept_Sales=0 |
One-Hot Encoded into binary indicator vectors |
signup_date |
"2020-07-10" | month=7, day=10, dayofweek=4 |
Extracted numeric temporal features |
churn (Target) |
0 or 1 |
0 or 1 (Untouched) |
Protected target column preserved exactly |
Step 5: datadoc evaluate demo.csv --target churn --task classification --estimator tree
Runs 80/20 train/test evaluation benchmark to measure clean vs raw accuracy improvement:
JSON & Terminaldatadoc evaluate benchmark{ "task": "classification", "metric": "balanced_accuracy", "baseline_score": 0.542, "selected_score": 0.815, "improvement": +0.273, "selected_pipeline": "candidate", "feature_count": 16 } Baseline Score : โโโโโโโโโโโโโ 0.542 Candidate Score: โโโโโโโโโโโโโโโโโโโ 0.815 (+27.3% boost!)
1. Core Concepts & Terminology
Before running CLI commands, it is essential to understand the key parameters and arguments that govern DATADOC's behavior.
Target Column (--target)
The target is the column you intend to predict (e.g., churn, salary, price). When you declare --target <column>, DATADOC applies a protection boundary to that column. The target is never imputed, scaled, encoded, or clipped. It passes through unchanged so your evaluation metrics and model training remain uncorrupted.
churn contains 0 (customer stayed) or 1 (customer left). Passing --target churn prevents DATADOC from scaling or modifying these class labels.
Task Type (--task)
Defines the nature of the supervised learning task. DATADOC uses this to determine evaluation metrics and optimal default preprocessing routines:
auto(Default): Automatically infers task type. If the target has โค 20 unique values or is boolean/string, it selectsclassification; otherwise,regression.classification: Supervised binary or multi-class prediction (evaluates via balanced accuracy / log loss).regression: Continuous numeric prediction (evaluates via RMSE / Rยฒ).
Identifier Dropping (--drop-identifiers)
High-cardinality string IDs (such as customer_id, uuid, user_name) carry no generalizable predictive signal and cause severe overfitting if one-hot encoded. Passing --drop-identifiers instructs DATADOC to detect string IDs and remove them during transformation.
Feature Scaling Strategies (--scaling)
Tabular numeric features often vary by orders of magnitude (e.g., age spans 18โ80 while income spans 20,000โ500,000). Scaling normalizes these ranges:
| Mode | Mathematical Formula | Best Used For |
|---|---|---|
auto |
Picks standard for linear models; none for tree models |
Default โ recommended for most workflows |
none |
Xout = X |
Tree-based algorithms (RandomForest, XGBoost, LightGBM) |
standard |
Z = (X - ฮผ) / ฯ |
Linear Regression, Logistic Regression, Neural Networks, SVMs |
robust |
Z = (X - Q2) / IQR |
Datasets containing unclipped, heavy-tailed outliers |
Outlier Clipping (--clip-outliers)
When enabled, DATADOC calculates upper and lower Interquartile Range (IQR) bounds during fit and clips extreme values during transform:
Upper Bound = Q3 + (1.5 ร IQR)
Values below the lower bound are clamped to the lower bound; values above the upper bound are clamped to the upper bound.
Split Strategies (--time-column & --group-column)
--time-column <col>: Performs an ordered temporal split (oldest rows in train, newest in test) to prevent future-data leakage in time-series tabular problems.--group-column <col>: Performs group-aware splitting so that all records belonging to a single group (e.g., all visits bypatient_id) reside strictly in train or test.
2. The Pipeline Lifecycle
DATADOC operates on a strict 4-stage lifecycle designed to guarantee zero data leakage between training, validation, and production inference sets.
3. Read-Only Commands
Inspects a CSV or Parquet file without making any modifications. Generates detailed health reports, column role assignments, missing value counts, and schema fingerprints.
TerminalInspect dataset qualitydatadoc profile demo.csv --target churn --output profile.json
Options
| Option | Type | Default | Description |
|---|---|---|---|
file_path | Argument | Required | Path to CSV or Parquet input file |
--target | Option | None | Declare prediction target column to apply protection |
--output | Option | None | Path to write profile output JSON |
Generates a transparent, step-by-step transformation plan without modifying the dataset or learning statistics.
TerminalGenerate transformation plandatadoc plan demo.csv --target churn --drop-identifiers --output plan.json
Options
| Option | Type | Default | Description |
|---|---|---|---|
file_path | Argument | Required | Input CSV or Parquet dataset |
--target | Option | None | Protected target column name |
--task | Option | auto | auto, classification, or regression |
--drop-identifiers | Flag | False | Include identifier dropping in transformation operations |
--output | Option | None | Path to save plan JSON file |
Generates an automated, standalone, interactive HTML data health and preparation audit report for sharing with teammates. Completely self-contained with health grade scoring, distribution histograms, null rates, and transformation plans.
TerminalGenerate standalone HTML reportdatadoc report train.csv --target churn --output report.html
Options
| Option | Type | Default | Description |
|---|---|---|---|
file_path | Argument | Required | Path to CSV or Parquet input dataset |
--target | Option | None | Declare target column for protection and analysis |
--output | Option | <stem>_report.html | Path to write standalone HTML report |
--title | Option | None | Custom title for report header |
--preset | Option | None | Preset to base recommendations on |
--ai | Flag | False | Include AI Executive Summary & Feature Hypotheses in report |
--ai-model | Option | None | Model to use for AI report analysis |
--open / --no-open | Flag | True | Open in browser automatically |
Visually compares raw vs. transformed datasets side-by-side. Displays dimension changes, missing value reduction (100% resolution tracking), column lifecycle (retained, dropped, engineered), and numeric distribution shifts.
TerminalCompare datasets side-by-sidedatadoc compare raw.csv clean.csv --target churn --html comparison.html
Options
| Option | Type | Default | Description |
|---|---|---|---|
raw_path | Argument | Required | Path to raw CSV or Parquet file |
transformed_path | Argument | Required | Path to transformed CSV or Parquet file |
--target | Option | None | Target column name |
--html | Option | None | Path to save visual HTML comparison page |
--json | Flag | False | Output comparison data as structured JSON |
--open | Flag | False | Open HTML comparison in browser |
AI-powered dataset feature engineering hypotheses, semantic role inference, missingness mechanism analysis (MCAR/MAR/MNAR), and target leakage audit using LiteLLM (OpenAI, Gemini, Anthropic, Ollama).
TerminalAI Feature Hypotheses & Auditdatadoc explain raw.csv --target churn --model gpt-4o-mini --recommend-config
Options
| Option | Type | Default | Description |
|---|---|---|---|
file_path | Argument | Required | Path to CSV or Parquet file to analyze |
--target | Option | None | Target column for predictive task |
--model | Option | None | LLM model (e.g., gpt-4o-mini, gemini/gemini-2.0-flash, ollama/llama3) |
--output | Option | None | Save AI markdown explanation to file |
--recommend-config | Flag | False | Generate and save recommended datadoc.toml |
Fast dataset health audit displaying 0โ100 quality score, letter grade (A+ to F), and detected data quality findings.
TerminalQuick health checkdatadoc health demo.csv --target churn
4. Build Commands
Fits transformation rules (imputation medians, encodings, clipping bounds) strictly on the provided training split and saves a serializable pipeline.json artifact.
TerminalFit pipeline on training datadatadoc fit train.csv \ --target churn \ --task classification \ --drop-identifiers \ --scaling standard \ --clip-outliers \ --output artifacts/pipeline.json
Options
| Option | Type | Default | Description |
|---|---|---|---|
file_path | Argument | Required | Path to training CSV/Parquet file |
--target | Option | None | Supervised target column |
--task | Option | auto | auto, classification, regression |
--drop-identifiers | Flag | False | Drop string IDs during pipeline execution |
--scaling | Option | auto | auto, none, standard, robust |
--clip-outliers | Flag | False | Opt into upper/lower IQR outlier clipping |
--output | Option | pipeline.json | Path to save fitted JSON artifact |
Applies a saved, fitted pipeline artifact to validation, test, or production inference data. Guarantees that validation rows are transformed using training-derived statistics.
TerminalApply pipeline to validation datadatadoc transform test.csv \ --pipeline artifacts/pipeline.json \ --output clean_test.parquet
Options
| Option | Type | Default | Description |
|---|---|---|---|
file_path | Argument | Required | Path to dataset to transform |
--pipeline | Option | Required | Path to fitted pipeline.json artifact |
--output | Option | Required | Output file path (.csv or .parquet) |
Exports a standalone, self-contained Python script (pipeline.py) wrapping the fitted artifact for zero-dependency inference in production environments.
TerminalExport standalone Python scriptdatadoc export \ --pipeline artifacts/pipeline.json \ --output my_pipeline.py
Options
| Option | Type | Default | Description |
|---|---|---|---|
--pipeline | Option | Required | Path to fitted pipeline.json artifact |
--output | Option | pipeline.py | Path to write standalone Python script |
5. Measure & Orchestration Commands
Performs a strict 80/20 train/test evaluation comparing a baseline model on raw data against a candidate model trained on DATADOC-cleaned data.
TerminalRun evaluation benchmarkdatadoc evaluate demo.csv \ --target churn \ --task classification \ --estimator tree \ --output eval_report.json
Options
| Option | Type | Default | Description |
|---|---|---|---|
file_path | Argument | Required | Input CSV or Parquet file |
--target | Option | Required | Supervised target column to evaluate |
--task | Option | auto | auto, classification, regression |
--estimator | Option | linear | linear, tree, or both |
--time-column | Option | None | Column name for ordered temporal splits |
--group-column | Option | None | Column name for group-aware splits |
--output | Option | None | Path to save evaluation report JSON |
Orchestrates a complete, one-shot local run: profiles, plans, fits, transforms, and generates a reproducible run directory with manifest files.
TerminalExecute reproducible local rundatadoc run demo.csv \ --target churn \ --output-dir datadoc-run \ --evaluate
Options
| Option | Type | Default | Description |
|---|---|---|---|
file_path | Argument | Required | Input dataset path |
--target | Option | None | Target column name |
--task | Option | auto | auto, classification, regression |
--output-dir | Option | datadoc-run | Directory path to persist run artifacts |
--evaluate | Flag | False | Include evaluation benchmark in run output |
6. Dashboard & Utility Commands
Launches a local FastAPI backend server and opens the interactive Web Dashboard in your default browser.
TerminalLaunch Web Dashboarddatadoc ui demo.csv --port 8000
Options
| Option | Type | Default | Description |
|---|---|---|---|
file_path | Argument | Required | Dataset file to load in dashboard |
--port | Option | 8000 | HTTP port for local dashboard server |
Displays the current installed version of datadoc-cli.
TerminalCheck DATADOC versiondatadoc version
Guided setup: asks for target, preset, identifier/dedup policy, scaling, clipping, and rare-category frequency, writes datadoc.toml, then runs the full pipeline.
TerminalGuided first rundatadoc wizard train.csv --output-dir datadoc-run
Options
| Option | Type | Default | Description |
|---|---|---|---|
file_path | Argument | Required | Training CSV or Parquet file |
--output-dir | Option | datadoc-run | Directory for run outputs |
Writes a starter datadoc.toml so runs are reproducible without retyping flags. Settings can also live under [tool.datadoc] in pyproject.toml; explicit flags always win.
TerminalRepeatable configdatadoc init --preset balanced --output datadoc.toml
Lints a dataset for leakage and prep pitfalls: target duplication, null targets, infinite values, duplicate rows, constants, and profile findings.
TerminalLeakage lintdatadoc lint train.csv --target churn
Diffs two profile, plan, or pipeline JSON artifacts top-level key by key, plus added/removed plan operations.
TerminalCompare artifactsdatadoc diff plan-v1.json plan-v2.json
Lists the 7 registered deterministic plugins in priority order, including third-party entry-point plugins. Use datadoc plugins show <Name> for detail.
TerminalRegistry introspectiondatadoc plugins list
7. Step-by-Step Hands-On Tutorial
Follow this sequence using the downloadable demo.csv file to test all features of the DATADOC CLI. Prefer guidance? Start with datadoc wizard demo.csv instead of steps 1โ3.
BashFull CLI workflow test sequence# Step 1: Inspect dataset quality and health findings datadoc profile demo.csv --target churn # Step 2: Preview the proposed transformation plan datadoc plan demo.csv --target churn --drop-identifiers # Step 3: Fit the pipeline on training data and save artifact datadoc fit demo.csv \ --target churn \ --drop-identifiers \ --scaling standard \ --clip-outliers \ --output artifacts/pipeline.json # Step 4: Transform new validation dataset using saved artifact datadoc transform demo.csv \ --pipeline artifacts/pipeline.json \ --output clean_demo.csv # Step 5: Benchmark candidate vs baseline ML performance (with ablation) datadoc evaluate demo.csv \ --target churn \ --task classification \ --estimator tree \ --ablation # Step 6: Export executable Python wrapper script datadoc export \ --pipeline artifacts/pipeline.json \ --output my_pipeline.py # Step 7: Run complete reproducible local run in one command datadoc run demo.csv \ --target churn \ --output-dir full-run \ --evaluate # Step 8: Open interactive Web Dashboard datadoc ui demo.csv
8. Master Options Reference Table
Below is a quick reference matrix of all options across DATADOC CLI commands:
| Flag / Option | Commands Supported | Allowed Values | Default | Description |
|---|---|---|---|---|
--target |
profile, plan, fit, evaluate, run | Column Name | None |
Declares protected prediction target column |
--task |
plan, fit, evaluate, run | auto, classification, regression |
auto |
Specifies machine learning problem type |
--drop-identifiers |
plan, fit, run | Boolean Flag | False |
Detects and removes high-cardinality string IDs |
--identifier-column |
plan, fit, run | Column Name (repeatable) | None |
Forces a column into the identifier role (e.g. PassengerId) |
--ignore-column |
plan, fit, run | Column Name (repeatable) | None |
Excludes a column from features entirely |
--scaling |
fit, run | auto, none, standard, robust |
auto |
Selects numeric feature scaling strategy |
--clip-outliers |
fit, run | Boolean Flag | False |
Clamps numerical outliers to IQR bounds |
--estimator |
evaluate | linear, tree, both |
linear |
Selects model family; both fits linear and tree and keeps the stronger validation score |
--preset |
profile, plan, fit, evaluate, run | quick, balanced, linear, tree, time, robust |
None |
One-word policy bundle for scaling, clipping, threshold, and estimator family |
--deduplicate |
fit, run | Boolean Flag | False |
Drops duplicate rows from training data at fit time |
--rare-frequency |
fit, run | Float 0โ1 | 0.0 |
Groups categories below this frequency into __RARE__ |
--cyclical |
fit, run | Boolean Flag | False |
Adds sin/cos pairs for datetime parts |
--no-hour |
fit, run | Boolean Flag | False |
Skips hour extraction even when time data is present |
--categorical-threshold |
fit, run | Integer | 20 |
Max one-hot width before falling back to frequency encoding |
--ablation |
evaluate, run | Boolean Flag | False |
Compares full, no-clip, no-scaling, and minimal variants |
--validate |
transform | Boolean Flag | False |
Runs schema validation and median-shift drift checks |
--explain |
profile, plan | Boolean Flag | False |
Renders human-readable tables instead of raw JSON |
--compare |
profile | File Path (.json) |
None |
Compares against another profile JSON |
--diff |
plan | File Path (.json) |
None |
Diffs against another plan JSON |
--config |
profile, plan, fit, evaluate, run | File Path (.toml) |
Auto-discovered | Explicit datadoc.toml path; explicit flags override it |
--format |
export | python, joblib |
python |
Executable wrapper script or portable joblib artifact dict |
--time-column |
evaluate | Column Name | None |
Enforces ordered temporal train/test split |
--group-column |
evaluate | Column Name | None |
Enforces group-aware train/test split |
--pipeline |
transform, export | File Path (.json) |
Required | Path to fitted pipeline JSON artifact |
--output |
profile, plan, fit, transform, evaluate, export | File Path | Varies | Path to write command output artifact |
--output-dir |
run | Directory Path | datadoc-run |
Directory path for reproducible run outputs |
--port |
ui | Integer | 8000 |
HTTP port for local dashboard server (--no-browser skips auto-open) |
9. Error Recovery & Best Practices
Evaluation scores are computed via an 80/20 holdout split. Use an external final test set for critical production decisions.
If datadoc transform fails with a schema mismatch error, ensure the input dataset contains all feature columns present when datadoc fit was executed.