OctOpus Documentation

OctOpus automates recurring data-science work for enterprise teams: give it data and a business objective, and it builds, validates, and delivers the model and report without requiring a data scientist to operate it. Under the hood it is an autonomous AI data scientist — an AI agent that profiles data, writes training code, runs machine-learning experiments as subprocesses, diagnoses failures, validates winners on a held-out set, and deploys the best model. This document is the complete reference: engine internals, supported model families, supported tasks, data connectors, the prediction API, deployment options, and a glossary of every term used inside the product.

1. The autonomous research engine

The OctOpus engine is a closed-loop research agent. Where a traditional AutoML platform runs a fixed pipeline — pick from a model catalog, hyperparameter-search inside fixed bounds, ensemble the top-K — OctOpus operates the way a senior data scientist would: profile, plan, code, run, diagnose, iterate, validate, ship. Each round is a fresh decision informed by the prior round's results.

The loop has four explicit phases, all visible to the user in real time:

1. Profile — type inference, missingness, leakage detection, target imbalance, time-aware split eligibility, suggested features.
2. Discover — frame the task: target column, task type, validation strategy, headline metric. The agent infers these from the user's plain-English goal and the data profile, then surfaces them for confirmation.
3. Decide & write — the agent writes program.md (the research spec) and train.py (the actual training code) for the next experiment. It picks the model family, the feature pipeline, and the hyperparameter search bounds.
4. Execute & reflect — the experiment runs as a sandboxed subprocess. The agent reads stdout, parses metrics, and decides what to try next. Errors are diagnosed and fed back into the next round.

Mandatory model-family rotation prevents the agent from getting stuck on one architecture. If a given family (e.g. LightGBM) has been used twice in the last eight experiments, the rotation guard injects a directive forcing the next experiment to use a different family.

2. The end-to-end workflow

A full OctOpus run produces a deployable model from a CSV in 5 to 60 minutes depending on data size and model family. The user-visible flow:

1. Drop a CSV or connect a warehouse — see connectors.
2. Describe the goal — plain English: "predict churn", "forecast revenue", "classify support tickets".
3. Confirm the framing — OctOpus shows the inferred target, task, metric, and validation strategy. Edit any line.
4. Say go — the research loop runs: baseline → tuned GBM → modern tabular → foundation model → ensemble.
5. Use the model — score new data via the Predict tab, hit the API, or download model.pkl.

See the step-by-step guide for screenshots and the user-facing walkthrough.

3. Supported model families

OctOpus chooses the model family per experiment based on the data shape, the task type, and the rotation guard. The full catalog:

Gradient boosting (the workhorses)

Modern tabular deep learning

Time-series neural networks (NeuralForecast)

Foundation models (zero-shot)

Classical ML (small-data fallback)

For datasets under ~500 rows, OctOpus prefers Ridge, ElasticNet, RandomForest, ExtraTrees, kNN, or SVM over deep nets. These avoid overfitting on small samples and produce stable cross-validated estimates.

NLP

• TF-IDF + Logistic Regression / SVM — strong baseline for short-text classification.
• SetFit — few-shot fine-tuning of a sentence-transformer; works well with hundreds of labels.
• HuggingFace transformers — fine-tune of any encoder on the HF hub via the transformers library.
• Sentence-transformers + classifier head — frozen encoder, lightweight head.

Image classification

• ResNet18 / ResNet50 — torchvision pretrained backbones with a fine-tuned head.
• EfficientNet / EfficientNet-V2 — better accuracy/parameter trade-off.
• ViT (Vision Transformer) — via timm, for larger labelled datasets.

4. Supported tasks

• Binary classification — predict 0/1 outcomes (churn, fraud, conversion). Headline metric: ROC-AUC, PR-AUC, F1, log-loss.
• Multi-class classification — predict one of N classes. Headline metric: accuracy, macro-F1, log-loss.
• Multi-label classification — predict any subset of N labels per row. Per-label metrics aggregated.
• Regression — predict a continuous number (revenue, price, score). Headline metric: RMSE, MAE, R², MAPE.
• Time-series forecasting — single-series, panel/hierarchical, with optional exogenous variables. Headline metric: WAPE, sMAPE, RMSE.
• NLP classification & regression — text inputs of any length, single or multi-document.
• Anomaly detection — unsupervised (isolation forest, autoencoder, OneClassSVM) or supervised (rare-class classification).
• Image classification — single-label or multi-label, fine-tune of a pretrained backbone.

Recommendation, ranking, and survival analysis are on the roadmap.

5. Data connectors

OctOpus reads data from any of the following sources. Credentials are encrypted at rest with Fernet using a per-install random secret and never shown to the LLM-authored code.

• File upload — CSV, TSV, Parquet, JSON-Lines, XLSX, ZIP archives.
• Cloud warehouses — BigQuery, Snowflake, Redshift, Databricks SQL, ClickHouse, Vertica.
• Operational databases — Postgres, MySQL, SQL Server, MongoDB, Supabase.
• Object storage — Amazon S3, OneDrive, SharePoint, Google Drive.
• SaaS / collaboration — Google Sheets, Notion.
• Marketing — Google Ads, Meta Ads.
• Messaging — Slack.
• BI — Power BI.

6. Feature engineering

OctOpus generates a prepare.py per run that captures every preprocessing step:

• Type inference — categoricals, numerics, timestamps, IDs auto-detected.
• Missing-value imputation — median for numeric, most-frequent for categorical, learned indicator columns.
• Categorical encoding — target encoding, one-hot, ordinal, embedding (deep models).
• Date features — day-of-week, day-of-month, week-of-year, hour, is-weekend, is-holiday, lag features (time-series only).
• Text features — TF-IDF, character n-grams, sentence-transformer embeddings.
• Numeric transforms — log, square-root, quantile-uniform, standardization, robust scaling.
• Interaction features — pairwise products / ratios when the data shape supports it.
• Leakage detection — drops columns with a near-perfect correlation to the target or with a future-date condition.

The full preprocessing pipeline is reproducible: the same prepare.py applied to a new CSV produces the same features.

7. Validation & holdout governance

Every run uses an agent-blind holdout: a subset of the data is split off before any feature engineering or model code is shown to the LLM. The holdout is stored in a separate filesystem location that the agent's subprocess cannot read. The "winner" of a run is the model that generalizes to this holdout — not the one that scored best on cross-validated training data.

Validation strategies:

• Stratified k-fold — default for classification, k=5 by default.
• K-fold — default for regression on iid data.
• Time-series split — no future bleeds into training; mandatory when the engine detects a timestamp column.
• Group k-fold — when an explicit group column is present (user_id, customer_id), to prevent within-group leakage.
• Walk-forward validation — for forecasting, expanding-window or rolling-window.

8. Hyperparameter tuning

Experiment 2 of every research run is a mandatory tuned-GBM round with Optuna:

• Sampler — Tree-structured Parzen Estimator (TPE)
• Trials — minimum 30, scaled with dataset size
• Objective — cross-validated headline metric
• Search space — chosen per family with sensible priors (log-uniform for learning rates, integer for tree depth, categorical for boosting type)
• Pruning — Hyperband-style early-stop on slow trials

9. Stacking & ensembles

From experiment 5 onward, OctOpus considers StackingRegressor / StackingClassifier with three or more diverse base learners. Diversity is enforced by family (one GBM, one deep model, one classical), and the meta-learner is Ridge for regression / LogisticRegression for classification. Bagging is also produced when single-model variance is high.

10. Prediction API

Every completed run exposes a prediction endpoint:

POST /api/run/{run_id}/predict-file
Content-Type: multipart/form-data
file: <your CSV with the same feature columns>

→ 200 OK
{
  "predictions": [0.812, 0.034, ...],
  "total_rows": 1024
}

Authentication uses the workspace token shown in the run's Predict tab. Self-hosted serving is supported by downloading model.pkl + train.py from the Artifacts tab and loading them with joblib.load.

11. Run artifacts & reproducibility

Every run captures, per experiment:

• program.md — the research plan
• train.py — the actual code that trained this experiment
• prepare.py — the preprocessing pipeline
• model.pkl — the trained artifact (joblib for sklearn / GBM, pickled wrapper for deep / foundation models)
• metrics.json — CV + holdout metrics
• logs.txt — full subprocess stdout / stderr
• shap_values.npy + feature_importance.csv
• report.md + report.pdf — full markdown + PDF report

The artifact bundle is downloadable as a .zip from the Artifacts tab, and is keyed by an immutable run hash so re-running with the same seed reproduces the result.

12. Deployment options

Three deployment modes:

• Web app (octoopus.dev/app) — fastest way to try OctOpus. Free tier, no credit card. Data is processed on a secure server with at-rest encryption.
• Desktop (octoopus.dev/desktop) — Mac app that runs fully on-device. Your data never leaves your machine. Ideal for HIPAA / PCI / AML workloads.
• Enterprise (octoopus.dev/enterprise) — private VPC install in your AWS / GCP / Azure account. SSO/SCIM, audit logging, SOC 2 / GDPR / HIPAA / PCI-aligned.

13. Security & compliance

Full security posture is documented at /security. Highlights:

• Encryption — TLS 1.2+ in transit; AES-256 / Fernet at rest.
• Key isolation — research-run subprocesses launch with a scrubbed environment so LLM-authored code cannot read provider API keys.
• Holdout governance — holdout data lives outside the agent's workspace; the LLM cannot peek.
• Secret redaction — all SSE / SQLite log streams pass through a redaction regex before reaching disk or UI.
• SSO/SCIM — SAML 2.0 + OIDC + SCIM 2.0 (Enterprise).
• Audit log — every run, every experiment, every error, queryable per workspace.

14. OctOpus Desktop

OctOpus Desktop is the fully on-device version of the autonomous AI data scientist. It runs entirely on your Mac — no server round-trips, no external API calls except to the LLM provider you choose. Ideal for sensitive data that cannot leave the device. See /desktop for the download.

15. OctOpus Enterprise

OctOpus Enterprise deploys inside your AWS, GCP, or Azure account as a private / VPC install. Includes SSO/SCIM, audit logging, custom roles, dedicated workspaces, and the same agent-blind holdout governance as the public app. Compliance documentation (SOC 2, GDPR, HIPAA, PCI) available under NDA — email sales@octoopus.dev. See /enterprise.

16. OctOpus MCP server

OctOpus ships a Model Context Protocol (MCP) server at mcp/octopus_mcp_server.py. Register it with Claude Code, Cursor, or any MCP-aware client to expose project tools (health, profile_csv, discover_direction, start_research_run, get_run_status, get_artifact_url, bootstrap_project, bootstrap_and_export_handoff) as callable tools.

// .mcp.json
{
  "mcpServers": {
    "octopus": { "command": "python3", "args": ["mcp/octopus_mcp_server.py"] }
  }
}

17. Command-line usage

The web app is the primary interface, but every operation is also available via the underlying Python modules. Quick syntax-check:

python3 -c "import engine; import server; print('OK')"

Run the demo regression suite:

make demo-test

Start the server locally:

PORT=8019 python3 server.py
# → http://localhost:8019/app

18. Explainability

Every run produces:

• Feature importance — gain / split for GBMs, learned weights for linear models.
• SHAP values — per-row contribution per feature, summarized as a bar plot and a beeswarm.
• Partial-dependence plots — average model output as one feature varies, with the others held at their distribution.
• Per-prediction reasoning — for the Predict tab, the agent explains the top-3 features driving each prediction.

19. Monitoring & drift

Once a model is deployed, OctOpus monitors:

• Input drift — KS test on numeric features, chi-square on categoricals, alerts when a feature distribution has shifted significantly from training.
• Prediction drift — distribution of model outputs vs the training-time distribution.
• Performance drift — when ground truth labels arrive, rolling-window holdout-equivalent metrics.
• Schema drift — alerts when the input CSV gains, loses, or renames columns.

20. Glossary

Terms used inside the product, in alphabetical order.

Agent-blind holdout

A test set kept outside the agent's workspace so the LLM-authored experiment code cannot read it. The validated winner of a run is the model that holds up on this set.

AutoML

"Automated machine learning" — historically a fixed pipeline (model catalog → grid search → ensemble). OctOpus uses the term loosely; the engine is autonomous, not just automated.

Autonomous AI data scientist

An AI agent that runs the full data-science loop — profile, frame, code, experiment, diagnose, validate, deploy — with minimal human input. Distinct from AutoML in that each round is a fresh decision, not a fixed step in a pipeline.

Baseline experiment

The first experiment of every run. Domain-matched single model + heavy feature engineering. Tabular → CatBoost / LightGBM. Time series → LightGBM with lag features. NLP → TF-IDF + Logistic Regression. Images → ResNet18.

Cross-validation (CV)

Estimating model performance by training on K-1 folds and validating on the K-th, rotated K times. OctOpus uses stratified k-fold by default for classification, k-fold for regression, time-series split for forecasting.

Discovery phase

The step where OctOpus picks the target column, task type, validation strategy, and headline metric from the user's plain-English goal and the data profile. Surfaced for confirmation before any code runs.

Experiment

One model-building attempt: a complete train.py + a subprocess run + parsed metrics. A research run produces 5 to 15 experiments.

Foundation model

A model pretrained on a very large dataset that can be applied zero-shot to a new task. Examples used by OctOpus: TabPFN (tabular), Chronos / TiRex / TimesFM / Moirai (time series).

Headline metric

The single number that defines "best" for a given run. Picked by OctOpus based on the task: ROC-AUC for binary classification, RMSE for regression, WAPE for forecasting, etc.

Holdout governance

The discipline of keeping the holdout set outside the agent's reach so it can't be peeked at during experiment generation. See agent-blind holdout above.

Leakage

When a feature accidentally encodes the answer (a column populated AFTER the event you're predicting, an ID with target information, a time-leaking aggregate). OctOpus's profiling stage flags these.

Optuna

The hyperparameter-optimization library OctOpus uses for the mandatory tuned-GBM round. TPE sampling, ≥30 trials, CV objective.

Plan / program.md

The research spec written by OctOpus before any experiment runs. Editable inline. The next experiment reads it back.

Research run

A complete loop: profile → discover → experiments → validate → ship. Identified by a unique run ID.

Rotation guard

The engine policy that prevents the agent from getting stuck on one model family. If the same family is used twice in the last eight experiments, the next experiment must use a different family.

SHAP (Shapley Additive exPlanations)

Per-row, per-feature contribution to the model's prediction. OctOpus computes SHAP for every run and ships it as part of the artifact bundle.

Stacking

An ensemble where a meta-learner is trained on the out-of-fold predictions of multiple base learners. OctOpus uses Ridge (regression) or LogisticRegression (classification) as the meta-learner.

train.py

The Python file the agent writes for each experiment. Fresh each time, never copied from a template. Contains the full pipeline: data load, prepare, fit, validate, save.

Validated winner

The single best model from a run, judged on the agent-blind holdout. The artifact downloaded as model.pkl.