This is described using Zeus terminology.
There are two types of walker:
Random walkers which you control by the length of their walk and road blocks, and destination walkers which go from the generating building to the destination building and ignore roadblocks.
Gymnasiums generate both types of walker. Athletes are random walkers, and give housing gymnasium access, but they also generate competitors which are destination walkers. They are generated at gyms and go to the stadium which can be much further away.
You need a competitor to pass a manor to turn it into an estate, but the stadium does not have to be in the estate block as long as you contrive to have destination competitors passing the estates.
To complicate matters, stadiums themselves generate random walking competitors
Generally speaking I'd have a stadium in an estate block as it only needs one tile access and you have to put it somewhere. But I might also have a second estate block which cannot have a second stadium but with care you can place two gyms in it to provide the competitor access.
Thus you can have lots of lovely charioteers.
On any map on the culture tab click the "see competitors" button to see where the competitors come from and go to.
Poseidon is broadly similar except for using Museum and University (generates random scholars and destination curators) instead of gym and stadium. In addition the university generates destination astronomers which go to the Observatories.
If there is something else going on then I don't know about it, but someone else might.
[This message has been edited by Serrataur (edited 01-26-2022 @ 07:58 AM).]