The Observable Garden

Did the agent find an edge, or just search long enough?

When an AI agent searches market data, its best backtest always looks good, because it is the best of many tries. Quixote records every step of the agent’s search, replays it on null data, and certifies a strategy only if it beats what the search would find by luck.

Run it on your own search, read the docs, or read the paper.

Eucalyptus, coming soon: the same test for agents that search raw data freely.

Continue

Run your own agent

Your model as the agent

Bring a data panel, a class of rules, an API key and a budget. Your model runs the search here in your browser, and the gate tests what it finds.

What the model sees, and where your files and key go

Your panel and class files are read in this page and never uploaded. Your key goes from this page straight to the provider. The model receives the prompt and tools below, plus what the harness shows any agent: the feature names, the class name, the rule it holds, the scores of the rules it tries, the stopping-rule information and the move budget. It never sees the returns or the feature values. You get a log the command line can audit, and a verdict you can download or share.

Try it on three bundled searches →

Data

A small sample panel and class are already loaded, so you can try a run with only a key.

Reading the sample panel and class…

Model

Sent to the Messages API as output_config.effort. All documented levels are offered for every model. If a model rejects one, the provider's error is shown word for word.

Key

Your key is held in this page's memory for one run and sent only from your browser to the provider. It is never stored and never appears in a URL, the log or the result. It is cleared when the run ends or you leave the page.

You can verify this. Open your browser's developer tools, Network tab, during a run: the only hosts contacted are this site and api.anthropic.com. The site's content security policy allows connections only to this site, api.anthropic.com, api.openai.com, openrouter.ai, and localhost or 127.0.0.1 on any port. It forbids the browser from sending anything to any other destination. The code that handles the key is public: app/web/src/gate.worker.ts.

Use a key you can delete afterwards.

Budget

How many times the search is repeated on resampled data.

The tools the model is given

The model's moves

No run yet.

    NewIf a run fails, the error shows the provider's message. Please report problems in the repository's issues.

    SanchoSancho, the fixed model, doesn't run in the browser. Set it up in the Sancho section.

    AdviceThis is not investment advice.

    RecordRuns here are not registered results and never enter the research record.

    Continue

    Docs

    Run the gate yourself

    Install #

    git clone --depth 1 https://github.com/ben06kn-droid/observable-garden
    pip install -e .                  # Python 3.10 or later

    One minute #

    quixote example

    This audits a bundled example, a scripted search on a small synthetic panel, and prints the verdict. Every verdict on this site comes with the command that reproduces it. For example:

    quixote audit examples/logs/scripted_session.json --panel examples/data/tiny_panel.npz --class examples/class.json --B 2000 --seed 0 --json

    The panel format #

    A panel is a single .npz file (see quixote/panel.py). quixote run, quixote audit and quixote sancho all read it.

    arrayshapemeaning
    returns(T, M)simple returns; NaN = not tradable in that row
    features(T, M, K)what a rule may use in row t (data up to row t only)
    feature_names(K,)names of the K features
    periods_per_yearscalar252 for daily rows
    delayscalar, default 1features at row t earn returns[t + 1 + delay]
    cost_bpsscalar, default 5one-way cost per unit of turnover
    borrow_bps_yrscalar, default 50yearly borrow cost on short weight
    row_labels, asset_names(T,), (M,)optional; split accepts a row label

    A rule's book is dollar-neutral with gross exposure 1 over the tradable assets. quixote.panel.save(...) writes the file. Sancho reads the same file. It builds its return-structure inputs from returns, treats features as its 40 registered features only when K is 40, and needs a few years of rows: with the default levers, its first scored row is 763.

    What a verdict means #

    PASSp is below α: the submitted rule beats what the search would find on noise.
    FAILp is at or above α, and the test had the power to tell: the result is what this search finds by luck.
    INADMISSIBLEp is at or above α, and the search was too wide to certify anything.
    UNDECIDABLEthe record alone cannot license a correction.
    DEGENERATEthe test statistic broke on this menu. Do not read its p.

    The code reports a pass as CERTIFIED; this site shows it as PASS.

    How this site runs it #

    Nothing is computed on a server. Your browser downloads Python (Pyodide 314.0.7), numpy, pandas and the observable-garden package, built from the repository's own source. All are served from this site, with their hashes recorded. A Web Worker then runs quixote.audit.audit_log, the function behind quixote audit. Two functions that the package normally takes from compiled libraries are replaced in the browser by ports, which tests hold equal to the originals: the Politis–White block length (from arch) and the normal distribution functions (from scipy). An equivalence test runs the browser build against the command line on every example.

    The record on a result #

    The audit id is the SHA-256 of a JSON object with the keys B, alpha, class_sha256, log_sha256, panel_sha256, seed and wheel, written in code-point order with no spaces. It covers the hashes of the inputs, the replicate count, the seed, α and the exact package build. Two runs with the same audit id test the same search on the same data with the same code. Sancho's config hash is different: it is the hash of its lever file, shown in the Sancho section and in the viewer.

    The paper: The Observable Garden: fixed and adaptive candidate sets under the DSR #

    Knaster (2026). On SSRN: papers.ssrn.com/sol3/papers.cfm?abstract_id=7616559.

    A backtest is usually the best of many attempts, so its Sharpe ratio has to be judged against what the best of many attempts would find on pure noise. When the search is adaptive, testing against the list of strategies it happened to try gives the wrong error rate. A class of strategies declared before the search, or a log of every decision that can be re-run on noise, restores a valid test. Quixote is the gate built on that answer.

    Where it stands #

    The gate kept its error rate in every registered test. Its limit is power. On real data it refused nearly everything, including 240 LLM agent sessions on liquid ETFs. It issued one certificate, on a long history of industry portfolios, and that certificate's lower bound held over the six years that followed. This is evidence that the gate works, not a source of trading signals.

    Tracing a claim #

    Every claim in the paper maps to its registration, result and read in research/NOTE_MAP.md. The archived release: doi:10.5281/zenodo.23332377.

    Licence #

    The paper is licensed CC BY; the code is MIT.

    Continue

    Sancho · the fixed model

    Sancho

    Sancho doesn't run in the browser. You run it on your own machine with the package.

    Why, and what this section does

    Sancho's model uses LightGBM, which has no browser build, and a re-implementation would not be the registered Sancho. Use this section to set the levers, download the config and copy the command. Then open the output in the viewer, at the end of this section.

    What Sancho is

    Sancho is the fixed model that agents are compared against. It is one prediction model, declared in advance, so it counts as a single strategy. That lets the gate test a non-linear strategy at the cost of one trial.

    How it was designed and how its stream is built

    Its design (the penalty grids and training windows) was settled on synthetic returns, never on the returns of a panel it was later tested on. Every real test was registered first. Its stream is built walk-forward: it refits every 252 rows on past rows only, with an embargo of h + 1 + d rows between the training rows and the row scored. Inside are ridge regressions and small gradient-boosted trees (LightGBM, depth 2) over price, cross-asset and volume inputs, stacked with non-negative weights.

    The registered result

    Sancho's results with the default levers in the registered real-data tests (the paper's Table 3). These are read from the research record, not recomputed here. Config hash: …

    PanelVerdictIn-sampleHoldoutRecord
    French 49 industries, 1926–1989 Certified in-sample under the recency weights at the 96% level: weighted net Sharpe 1.084, p 0.0010, weighted lower bound +0.674. On the holdout, 1990–1995, the bound held: realised net Sharpe +1.646. read, grading
    Liquid ETFs, 2015–2024 Refused weighted net Sharpe 0.246, p 0.1970, weighted lower bound −0.123. On the holdout, 2025-01-02 to 2026-10-09 (1.76 years, weak), the bound held. read
    Binance USDT perpetuals, 4h bars, 2021 – Mar 2025 Refused (stream at d = 0): net Sharpe 0.198, p 0.3611. The holdout was not opened. read

    Each claim, its registration and its commits: research/NOTE_MAP.md, rows 31 and 33 to 35.

    The levers

    All of Sancho's levers are in one file, sancho/config.json. The defaults are the registered version 2, and the registered results apply only to those defaults.

    What changing a lever does

    Change any lever below and this page writes the config file that quixote sancho reads, with its hash. Nothing you set here is stored or leaves the page.

    The registered Sancho. Config hash (SHA-256 of the canonical JSON, the hash the run records): …

    A run with a tailored config is a different model from the registered Sancho, and is labelled by its config hash.

    Run it on your machine

    
      

    It writes sancho_out/sancho.json: the net stream from the first scored row, its in-sample net Sharpe and turnover, and the config hash.

    Coming soon in Eucalyptus: agents that can modify Sancho as they search, with no fixed feature panel.

    Continue

    Eucalyptus

    Coming soon