LongMemEval-S: measuring memory retrieval

Retrieving the evidence for an answer.

Enkeon’s recorded isolated evaluation reports 99.0% Recall@15 on 499 scored LongMemEval-S questions. The reported mean context size is 481 tokens. Recall measures whether the retrieved context contains the required evidence; it does not measure the correctness of a generated answer.

Recall by category.

Each point is a reported category result. The line between points shows the difference between the ANY and ALL retrieval criteria.

Method and measurement boundary.

The recorded configuration ingested conversations session by session, disabled category hints, and evaluated retrieval at a budget of 15 results with aggregation. Of the 500 questions in the source suite, one with no gold session was excluded from recall scoring.

The historical warm retrieval measurement was 16 ms p50 on isolated benchmark infrastructure. It is not a hosted API or end-to-end response-time claim. The landing page’s server measurement of 313 ms p50 and 376 ms p95 uses a different workload and boundary.

What this result establishes.

• This is an internal retrieval evaluation on a public dataset, not an independent certification.
• Recall@15 and answer accuracy are different metrics. Published competitor answer scores are not plotted against this retrieval score.
• These results do not establish a current leaderboard rank or performance on every customer workload.
• The recorded summary does not publish a versioned raw run artifact or hardware specification. Those details are needed for independent reproduction.
• No repeated-run uncertainty interval is reported.

Dataset and further reading.

LongMemEval benchmark repository · LongMemEval paper

See the research index for evaluations of answer correctness, model spend and memory integrity.