Even with an agenda, nobody wants to be the guy that interrupts people to get them back on topic.
I frequently ask people whether they can discuss the diversion later with whoever is interested. It's not just me but part of company culture.
In other contexts I'm notorious for keeping people on the agenda. It's generally appreciated.