DeciShift

Evaluation study: public flows, reproducible benchmarks, and decision-change evidence

Scope

This study evaluates the current DeciShift object of analysis: changes in the final action produced by a versioned executable decision system. It does not introduce a new attribution algorithm.

The evaluation has three parts:

  1. Public-data flows — one genuinely trained scikit-learn flow and one genuinely trained XGBoost flow.
  2. Reproducible DecisionFlow benchmarks — 10K and 100K records by default, with an optional 1M run.
  3. Decision-change case study — a model metric improves while aggregate action counts stay close, but individual actions still move.

The existing synthetic equipment-maintenance example remains unchanged.

Public-data examples

Wine / scikit-learn

examples/public_wine_sklearn/run.py uses scikit-learn’s bundled Wine dataset and trains deterministic logistic-regression models. A score for class_2 versus other classes drives the three-action routing policy route_not_class2, manual_review, route_class2.

The script runs five comparisons against the same baseline: feature-only, model-only, policy-only, rule-only, and all changes together.

Digits / XGBoost

examples/public_digits_xgboost/run.py uses scikit-learn’s bundled Digits dataset and trains deterministic XGBoost boosters for digit 8 versus other digits. The score drives auto_not_8, manual_review, and auto_8.

The same five change-isolation scenarios are executed. The feature-only scenario changes pixel preprocessing; the model-only scenario changes the fitted booster; the policy-only scenario changes action thresholds; the rule-only scenario changes the manual-review override.

Decision-change case study

The Digits script also creates a deliberately narrow case study:

The output records:

First published machine run

The first committed snapshot was generated by GitHub Actions run 36093691597 for source commit 76bbf63abb7fdbba26739f3c8f96f94e0110bfc8.

The candidate XGBoost model improved from 90.26% to 91.79% accuracy and from 0.9429 to 0.9472 ROC AUC on the fixed evaluation split. Final action counts remained close (auto_8 66→74, auto_not_8 530→526, manual_review 123→119), but 30 of 719 individual actions changed (4.17%).

Those changes were auto_not_8→manual_review (13), manual_review→auto_8 (8), and manual_review→auto_not_8 (9). Exact software-counterfactual attribution assigned 48.86% of absolute attribution mass to the model, 46.59% to policy, and 4.55% to rules.

The overall 5% action-shift limit passed, but the Decision Contract blocked the candidate because the actual_digit=6 cohort shifted at 9.41%, above its declared 8% cohort limit. This is the intended monitoring-gap example: aggregate model/action summaries can look acceptable while record-level transitions and one cohort still require review.

Committed snapshot: case-studies/digits-xgboost-76bbf63abb7fdbba26739f3c8f96f94e0110bfc8.md.

Benchmark protocol

benchmarks/evaluate_flow_attribution.py runs the existing 5-node linear and 8-node branched DecisionFlow benchmark cases at:

For each flow/size it records both exact and sampled attribution with:

The sampled protocol uses a deterministic seed and bounded adaptive sampling of 16–64 permutations, in batches of 8, with a target CI width of 0.10. The table records whether that bounded run actually reaches the requested convergence target rather than assuming that it did.

Memory is explicitly tracemalloc peak Python allocation, not process RSS.

First published benchmark snapshot

The first committed table is benchmarks/results/76bbf63abb7fdbba26739f3c8f96f94e0110bfc8.md, generated by the same Actions run. It contains the measured 10K/100K wall time, memory, changed-node counts, cache reuse, exact-vs-sampled attribution diagnostics and convergence status.

A useful result from that run is that exact attribution was faster for these 3–4 changed-node flows. The bounded sampled runs used all 64 permutations and did not meet the configured CI-width convergence target. That outcome is retained as measured rather than being tuned away.

Reproducibility and publication discipline

Every benchmark payload records the source commit SHA, Python version, NumPy version, pandas version and runner platform. The current workflow explicitly checks out the exact source SHA it reports, never writes to main, and never overwrites versioned release assets. It uploads commit-scoped artifacts.

The first published run predated that explicit-checkout hardening and used GitHub’s pull-request merge ref. Git inspection confirmed that the merge ref and recorded source head had the same tree SHA (b46035d26dc9c7b93eb7092a6768edc6f2d9b86d), so the executed file tree was identical. The workflow was then corrected so future provenance does not rely on that equivalence.

Committed benchmark snapshots live under benchmarks/results/ and retain the source SHA recorded by the generating job. README documentation should link to those generated snapshots rather than hand-entering timing or memory numbers.

For the first snapshot, the durable Actions artifact was 10846941370 with digest sha256:c80b98c0fd2188d403d7a79c4a39efe934ea1b2fd859074fdd3c531272508e84.

Interpretation limits

These examples show software behavior over fixed public evaluation splits. They do not establish real-world causality, fairness, safety, compliance or deployment fitness. Model metrics and aggregate action distributions are not substitutes for record-level decision-change evidence.