Benchmarks.

A result will be published here only when it comes from data described under Datasets, was produced under the holdout rule below, and, where a retailer's data is involved, the retailer has agreed in writing to its publication.

Three kinds of data can say something about Kelynto. They answer different questions, and results will always name which kind they come from.

For any retailer dataset the publication will state the number of stores, categories and weeks, which optional files were supplied (in-stock, promotions, local signals), and which signal sources were switched on. The retailer will be named only with its permission.

What is being measured is the method described on the Methodology page: variance decomposition, the gate a cause must pass, the noise guard, the confidence bands and the direction-only radar. Results will be produced by running the product as a customer would run it, through the same loaders and the same weekly run.

• Every setting that differs from its default will be listed beside the result.
• The commentary writer will be named: the deterministic template, or a language model and which one.
• Planner verdicts will be the planners' own. A verdict entered by Kelynto staff on a planner's behalf will be counted separately and labelled.

The rule is that no evaluated week may influence the settings it is evaluated under.

• Holdout in time. Thresholds are fixed before the evaluated weeks are run. Confirmation rates for a week are learned only from verdicts on earlier weeks, which is how the product works in normal use.
• No tuning on the test weeks. If a setting is changed after seeing results, the weeks already seen stop counting as holdout and will be reported as such.
• Generated data. Published runs will use data generated from seeds that were not used while the engine was being developed.
• All weeks reported. A week will not be dropped because its result is poor. Weeks excluded for a data fault will be listed with the reason.

The four measures are the ones on the product's own scorecard, so a published figure and a customer's figure mean the same thing. Each is reported with its sample size and is withheld below a minimum.

Reported beside them: the share of material variance that was explained and the share left unexplained, how often radar items were new to the planner, and the minutes a review took against the team's own stated starting point. On generated data, two more: how many planted causes were found at the planted size, and how many findings appeared in weeks in which nothing happened.

A number means little without something to hold it against. Each result will be reported next to the baseline that fits it.

• The team's own commentary. In a backtest, the causes named in the draft are compared with the causes the retailer's team named for the same week.
• A fixed threshold with no gate. The same data run through a plain rule: flag every store and category beyond a fixed percentage of plan and credit the nearest signal. This shows what the gate and the noise guard add, or do not.
• Chance, for direction. Radar direction accuracy is read against a coin toss, and against always predicting the usual direction for that type of signal.
• The pilot targets. 20 percent or less of the draft rewritten, 60 percent or more of causes confirmed, 75 percent or more of radar calls right, and 4 of 5 or higher on would-use. These are targets set before a pilot. They are not results.

These will apply to anything published here, and each publication will repeat the ones that bear on it.

• Generated data proves that the logic behaves as designed. It says nothing about accuracy on a real business.
• A planner's verdict is a judgement, not ground truth. A confirmed cause can still be wrong, and a rejected one right.
• The first retailers will be few and will have chosen to take part. Their results will not be a sample of retail.
• A result from one retailer does not transfer to another with different categories, signals or data quality.
• Minutes saved is self-reported by the planner against a starting point the team states itself.
• The radar's weather signals are only as good as the public forecast inside its horizon.

Every result will carry the engine version that produced it, and results from different versions will not be pooled. The engine version is stored on every weekly run, so a figure can always be traced to the code that produced it.

The current engine version is 2.1.0. No results have been published for it or for any earlier version.

Three sets of figures exist. All three come from synthetic data on a development machine, and none is a benchmark.

• Automated tests on generated weeks. They pass or fail: planted causes must be found, and a week of pure noise must produce no causal finding. They are a check on the logic, described on the Methodology page.
• Timing measurements. How long loading, the weekly run and page reads took at 500 and 2,000 synthetic stores on one small development machine. They guide engineering and are shared during a technical review. A development machine is not a benchmark environment, so they are not quoted here.
• The sample workspace scorecard. Its figures come from a synthetic retailer with simulated planner reviews. They show what the screen looks like. They are not results.