摘要
閱讀答案“Because vendors optimise for public benchmarks, and a high score on questions the vendor has seen tells you little about behaviour on your documents.
Thinkerwave AITech · Testing ai financial accuracy
thinkerwave.ai
閱讀原始來源，再探索採用它的已發佈答案與指南。
採用這份來源的答案
1 篇內容引用這份文件的已發佈公司答案及指南。
摘要
閱讀答案“Correct refusals.
摘要
閱讀答案“Twenty is enough to be useful in a day, including four malformed ones.
完整已發佈資料
Thinkerwave AITech
這份來源提供的資料
Why not just use a public benchmark score?
在公司資料中查看“Because vendors optimise for public benchmarks, and a high score on questions the vendor has seen tells you little about behaviour on your documents. Build a set from filings your own team already knows the answers to. Treat any headline accuracy number as a starting question, not an answer.”
What is the single most important thing to measure?
在公司資料中查看“Correct refusals. A system that answers a malformed or unanswerable question with a confident number is the one that loses you money in production. Most evaluations never score refusals at all, which is why they miss the failure mode that matters most in a regulated setting.”
How many questions do I need?
在公司資料中查看“Twenty is enough to be useful in a day, including four malformed ones. The point is not statistical power. It is to observe how the system behaves on questions where you already know the truth, and on questions that should be refused.”
關於 Thinkerwave AITech
ThinkerWave: “ThinkerWave AITech is a deep-tech AI company founded by Vikas Goel, building AI for complex, contested, high-stakes problems. Its first product is a self-evolving, multi-agent reasoning engine.”
來源與資料詳情
- 原始文件
- Thinkerwave AITech: /guides/testing-ai-financial-accuracy