DeciShift

Behavioral regression testing for ML decision systems.

DeciShift is an open-source, CPU-first framework for comparing two executable versions of a machine-learning decision system before deployment. It shows which final operational decisions changed, which action transitions occurred, which versioned components contributed, and whether observed changes stay inside explicit Decision Contracts.

GitHub repository · PyPI · Discussions · Help test v0.3.1

Why it exists

A candidate model can improve aggregate accuracy or AUC while downstream operational actions still change because models interact with features, policies, thresholds, and rules.

baseline + candidate
        ↓
execute both on the same records
        ↓
compare final actions and transitions
        ↓
software-counterfactual attribution
        ↓
verify saved evidence
        ↓
Decision Contract → PASS / BLOCK

DeciShift complements experiment tracking, model registries, evaluation/monitoring, orchestration, and CI/CD rather than replacing them.

Public reproducible example

The Digits/XGBoost case study uses a fixed held-out split of 719 public records:

• accuracy: 90.26% → 91.79%
• ROC AUC: 0.9429 → 0.9472
• final actions changed: 30 / 719 (4.17%)
• global 5% shift contract: PASS
• actual_digit=6 cohort: 9.41% shift against an 8% limit → BLOCK

This is descriptive evaluation evidence, not a claim of production safety, fairness, compliance, or real-world causality.

Read the evaluation study

Start here

python -m pip install --upgrade decishift==0.3.1
decishift demo --rows 1000 --no-save

From a source checkout:

decishift graph examples/triage/flow.yaml
decishift compare examples/triage/flow.yaml
decishift verify RUN_ID
decishift gate RUN_ID --contract examples/triage/decision-contract.yaml

Getting started

Documentation map

• Architecture — execution abstractions, evidence layers, trust boundaries
• Decision flows — structured row-aligned DAGs
• Flow attribution — exact/sampled software-counterfactual attribution
• Decision Contracts — deterministic release limits and exit codes
• Evidence integrity — manifests and SHA-256 verification
• Public API reference — generated top-level import inventory
• Compatibility — Python/OS/dependency verification matrix
• Failure modes — explicit negative-path behavior
• Related categories — complementary tooling and information matrix
• Community — independent validation and contribution paths
• Roadmap — direction and non-goals

Current integration directions

• MLflow — proposed decision-level release-gating example: issue #15
• ZenML — proposed pipeline integration example: issue #16

These are active integration directions, not claims of built-in production integrations today.

Join the project

The most valuable current contribution is an independent reproducibility run on a different environment. Successful reproductions, failures, installation friction, and counterexamples are all useful.

Community and contribution paths · Independent testing issue · GitHub Discussions

Keywords: ML behavioral regression testing, machine learning regression testing, decision-level model evaluation, MLOps release gating, model change impact analysis, ML decision comparison, model governance, behavioral testing, decision drift, machine learning testing.