Architecture & Codebase

1. Plugins Subsystem: The Math & Logic

The datadoc/plugins/ directory contains seven isolated, stateless advisor modules using Polars expressions. They inherit from BasePlugin and are discovered via plugins/registry.py (see section 4). The production transforms live in core/pipeline.py with frozen train-only state.

duplicates.py (Duplicate Detection, priority 05)

Logic: Counts exact duplicate rows with is_duplicated().sum(). With deduplicate=True, the pipeline drops them from training data at fit time so medians and vocabularies are not skewed. Validation data is never deduplicated.

rare.py (Rare-Category Grouping, priority 42)

Logic: Categories below rare_category_min_frequency (e.g. 0.02) are grouped into __RARE__ before one-hot encoding, capping output width while keeping the signal. Unseen inference values map to the same token.

outliers.py (IQR Outlier Clipping)

This plugin detects and clips extreme values to prevent them from skewing machine learning models (like Linear Regression) that are sensitive to extreme variances.

Logic: If a numeric column has a high standard deviation relative to its mean, the plugin uses the polars.Series.quantile() function to find Q1 and Q3. It applies a with_columns(pl.col(c).clip(lower, upper)) operation.

scaling.py (Standardization)

Machine learning models that use distance metrics (KNN, SVM) or gradient descent (Neural Networks) require features to be on the same numerical scale. This plugin applies Z-score normalization.

Logic: The plugin checks the ratio of max standard deviation to min standard deviation across columns. If the ratio > 10x, it applies scaling. The Polars execution is (pl.col(c) - pl.col(c).mean()) / pl.col(c).std().

missing_values.py (Imputation)

Models crash when they encounter null or NaN. This plugin safely fills holes.

Logic: Median is used for numerics instead of Mean because Mean is highly sensitive to outliers. Mode (most frequent value) is used for strings/categoricals. Polars code: pl.col(c).fill_null(pl.col(c).median()).

encoders.py (Categorical One-Hot)

Algorithms only understand numbers, not text like "Red" or "Blue".

Logic: If a column has too many unique values (e.g., UUIDs), it's dropped to avoid the "curse of dimensionality". Otherwise, it iterates through unique strings and does (pl.col(c) == val).cast(pl.Int8).alias(f"{c}_{val}").

target_encoder.py (Bayesian Target Encoding, priority 41)

For high-cardinality categoricals where one-hot encoding would produce hundreds of sparse columns, target encoding replaces each category with an empirical Bayes smoothed estimate of the target mean.

Logic: $n_i$ is category row count, $\bar{y}_i$ is category target mean, $\bar{y}_{global}$ is global target mean, and $m$ is smoothing weight (default 10.0). Categories with few samples shrink toward the global mean to avoid overfitting.

polynomial.py (Interaction & Polynomial Features, priority 44)

Linear models cannot capture feature interactions without explicit multiplicative terms. This plugin generates pairwise products and quadratic features for top numeric predictors.

Logic: Detects numeric features with high correlation or non-linear relationship to target, computes pairwise interactions up to max_interaction_terms (default 10) to avoid feature explosion.

datetime_feat.py (Time Series Extraction)

Logic: Takes a string column like "2024-05-12 14:00" and uses str.to_datetime(strict=False). It then extracts dt.year(), dt.month(), dt.day(), dt.weekday() — plus dt.hour() only when training data has a real time component — and drops the original column. With datetime_cyclical=True, month/day/weekday/hour also get sin/cos pairs.

2. CLI Commands: Background Execution Trace

When you type commands into the terminal, cli/app.py acts as the orchestrator. Here is exactly what happens step-by-step in the background.

datadoc profile

• Load: The CSV/Parquet file is loaded lazily into a Polars DataFrame.
• Pipeline Init: A DataDocPipeline is instantiated with default PipelineConfig.
• Analyze: pipeline.profile() is called. This triggers a massive parallel Polars scan measuring null percentages, uniqueness (cardinality), and regex matching to guess column types (Target vs Identifier vs Numeric).
• Render: The resulting DatasetProfile dataclass is passed to the Rich library, which draws the beautiful terminal table you see on screen.

datadoc plan

• Profile: Runs the profile step (above) internally to understand the data.
• Formulate: pipeline.plan() is called. It looks at the profile. If it sees numeric nulls, it appends a "missing_values" step to the TransformPlan. If it sees a categorical column with 5 unique values, it appends an "encoder" step.
• Serialize: The plan is printed to the screen as a sequence of deterministic steps.

datadoc wizard (guided entry point)

• Inspect: Loads the dataset and lists columns, row counts, and detected shape.
• Ask: Prompts for target column, preset (quick | balanced | linear | tree | time | robust), identifier policy, scaling, clipping, and rare-category frequency.
• Persist: Writes the answers to datadoc.toml so the run is reproducible, then delegates to datadoc run.

AI policy: there is no code-generating agent. The optional datadoc-cli[ai] extra (litellm) is constrained to explaining or ranking already-registered deterministic transformations. Provider-generated Python is never executed by the pipeline.

datadoc evaluate

• Split: Splits 80% train / 20% holdout — stratified for classification, ordered for --time-column, group-aware for --group-column.
• Cross-validate: Runs 3-fold CV on the train split comparing a minimal baseline config (no clipping, no scaling) against the configured candidate.
• Holdout: Refits the CV winner on full train and scores both on the untouched holdout. Classification uses balanced accuracy (LogisticRegression and/or RandomForest); regression uses negative RMSE (Ridge and/or RandomForestRegressor).
• Report: Emits an EvaluationReport with metric, split strategy, both scores, improvement, selected pipeline, feature count, and warnings. --ablation additionally compares full, no-clip, no-scaling, and minimal variants.

datadoc report

• Profile & Score: Calculates dataset profile, transformation plan, and 0–100 data health quality score with letter grade (A+ through F).
• Optional AI Advisory: When --ai is passed, queries LiteLLM for executive data summary, missingness etiology, and feature hypotheses.
• Self-Contained HTML: Renders an interactive, standalone HTML report with embedded CSS, distribution bars, and health audit findings.

datadoc compare

• Dual Profiling: Loads raw and transformed datasets and runs parallel profiling.
• Differential Delta: Measures dimension shifts, null resolution percentage (tracking 100% fixes), column lifecycle (retained, dropped, engineered), and numeric distribution shifts.
• Visual Report: Outputs an interactive side-by-side HTML comparison or structured JSON summary.

datadoc explain

• Grounding: Profiles data, inspects target, data types, null rates, and high-cardinality signals.
• Prompt Synthesis: Builds strict system prompt instructing model on tabular statistics, missingness mechanisms (MCAR/MAR/MNAR), and mathematical transformation syntax.
• Structured LLM Call: Calls provider via LiteLLM (OpenAI, Gemini, Anthropic, Ollama) requesting valid JSON adhering to AIExplanation schema.
• Display & Config: Renders rich terminal panels with feature hypotheses and can persist a recommended datadoc.toml.

datadoc health

• Fast Audit: Scans dataset for null rates, duplicates, constant columns, and extreme variances.
• Grade: Computes 0–100 health score with letter grade (A+ to F) and outputs concise terminal summary.

3. Core Logic: core/pipeline.py

This is the heart of DATADOC. It defines the DataDocPipeline class and the state models (DatasetProfile, TransformPlan). The pipeline enforces a strict 4-phase lifecycle.

The code is strictly designed to avoid Data Leakage. For example, if you fit a pipeline to training data, the medians calculated during fit() are hardcoded. When transform() is called on validation data, it injects those previously calculated medians rather than recalculating them on the validation data.

4. Plugin Registry: plugins/registry.py

Nine deterministic plugins (DuplicateRemover 05, MissingValue 10, Outlier 20, Datetime 30, CategoricalEncoder 40, TargetEncoder 41, RareCategory 42, PolynomialFeatures 44, Scaling 45) implement the BasePlugin lifecycle analyze → recommend → apply as stateless advisors. The registry auto-discovers them plus any third-party plugin registered under the datadoc.plugins entry-point group, sorted by priority. The fitted pipeline in core/pipeline.py — not the plugins — is the production source of truth: it re-implements each transformation with frozen train-only state so validation data can never influence fitted statistics.

5. AI Advisory Engine: datadoc/ai/

DATADOC integrates a zero-execution AI advisory subsystem that turns raw tabular metrics into actionable modeling recommendations, semantic missingness classification, and domain feature engineering hypotheses.

Execution Boundary & Safety

DATADOC maintains a strict zero-execution security boundary. The AI advisory engine is purely observational and analytical: it inspects statistical distributions, column headers, and quality findings to generate natural language explanations and mathematical formula hypotheses. It never writes or executes arbitrary code inside the transformation pipeline. All transformations applied to data must strictly pass through the deterministic, frozen-state DataDocPipeline.

Core AI Modules

• client.py: Multi-provider client wrapper. Automatically discovers active environment keys (OPENAI_API_KEY, GEMINI_API_KEY, ANTHROPIC_API_KEY) or connects to local Ollama endpoints (http://localhost:11434).
• prompts.py: System prompts enforcing structured JSON output for missingness classifications (MCAR, MAR, MNAR), semantic roles, target leakage risks, and domain-specific feature engineering formulas.
• explainer.py: Houses the AIExplainer class and explain_dataset() function. Extracts tabular signals, orchestrates API calls, validates JSON against expected schema, and generates clean markdown outputs.

6. Removed Legacy: core/engine.py and core/agent.py

Earlier versions shipped a DATADOC facade (engine.py) and an LLM code-executing agent (agent.py). Both were deleted in the 0.4.0 leakage-safe refactor: the facade fitted and transformed the same data (a leakage risk), and the agent executed provider-generated Python (a safety risk). New code uses DataDocPipeline directly; see MIGRATION.md for the upgrade path.

7. Dependencies Explained

Every dependency in pyproject.toml has a distinct, justified purpose.