Idan Schwartz
Posts by Idan Schwartz on the Aigency blog.
- Can We Use the KV Cache for Retrieval?
Prompt caching is treated as a per conversation trick. The measurements point somewhere more interesting: encode a catalog once, keep all of it, and retrieve from it instead of re-encoding. A thousand product images fit in 7.7 GB.
· SCIENCE
- How Many Rules Can a Model Follow?
A web page enforces your return notice by keeping it in the template. An agent has to remember it, one shopper at a time. We measured how many rules four models can hold at once.
· SCIENCE