kevinherron 2 days ago

Submit for batch processing using the OpenAI batch API?

1
sprobertson 2 days ago

This in combination with making the tool call / json response itself also "batched" is a good pattern. Instead of returning a single `{english, foul}` object per record, pass in an array of records and have it return an array of `[{english, foul}]`. Adjust the inner batch size depending on your record size and spread the rest over the batched API.