Open Evaluation Suite

SulcusBench Suite

Twenty Deterministic Tasks

An open, reproducible benchmark evaluating AI agent memory layers across contradiction resolution, temporal ordering, cross-session persistence, long-horizon scale, and grounded calibration.

Empirical Boundary

Memory Territory Gap

Raw prompt stuffing hits a hard ceiling at 57.9% because context windows vanish when sessions close and degrade under long-horizon noise.

Context Ceiling57.9%
Memory Territory42.1%

Context Window Ceiling

Handles single-session calibration, local contradiction updates, and short-range temporal sequence checks while the prompt window remains open.

Dedicated Memory Territory

Requires persistent Apache AGE graphs, cross-session entity recall, thermodynamic salience decay, and O(1) Synapse Map pointer paging.

Reference Baselines

Overall Leaderboard

Inspect verified baseline runs across all twenty tasks or isolate any evaluation dimension.

Verified Runs2 baseline adapters recorded
Rank 1

Context Window Baseline

Official2026-03-14
58%
Contradiction100%
Temporal75%
Sessions0%
Scale0%
Calibration100%
58%

10 of 20 passed

Rank 2

Zero Memory Floor

Official2026-03-14
0%
Contradiction0%
Temporal0%
Sessions0%
Scale0%
Calibration0%
0%

0 of 20 passed

Reproducible Harness

Execute Benchmark Suite

SulcusBench is open source and runs locally in Python. Submit verified adapter runs via pull request; suites include adversarial distractors for engineering credibility.

Install Runner

pip install "git+https://github.com/digitalforgeca/sulcus.git#subdirectory=sdks/python"
cd sulcus/packages/sulcusbench

Execute Baselines

python -m sulcusbench --adapter no-memory
python -m sulcusbench --adapter in-context

Evaluate Adapter

python -m sulcusbench --adapter sulcus --api-key sk-...
python -m sulcusbench --adapter mem0 --api-key ...
python -m sulcusbench --adapter sulcus --api-key sk-... --categories recall temporal