I realised I didn't mention it in the article, so in case you're curious it cost about $116 to run the 100k token version 100 times.
So, half that for batch processing [1], which presumably would be just fine for this task?
thank you, I was going to ask about this. It's not a crazy amount...
Do we know how that relates to actual operating cost? My understanding is that this is below cost price because we're still in the investor hype part of the cycle where they're trying to capture market share by pumping many millions into these companies and projects
Does this really reflect the resource cost of finding this vulnerability?
It sounds like a crazy amount to me. I can run code analyzers/sanitizers/fuzzers on every commit to my repo at virtually no cost. Would they have caught a problem like this? Maybe not, certainly not without some amount of false positives. Still this LLM approach costs many millions of times more than previous tooling, and might still have brought up nothing (we just don't read the blog posts about those attempts).