A number of factors go into the rate of settling. The factors that concern us are the size of the particles, the density of the particles, and the density of the liquid.
The particles are denser that the liquid - we know this because they settle rather than floating. The greater the difference between the density of the particles and the density of the liquid the faster the particles will settle. Colder liquids are denser (water is densest at 4C, which is why ice floats), so particles will settle slower in colder liquid.
Larger particles settle faster than smaller particles. Clarifying agents work by grabbing onto lots of little particles and creating bigger particles out of them. Ths will also happen naturally, but at a much slower rate, and the attractive forces are much weaker without the clarifier.
When we add the clarifier we stir it in. This allows it to come into contact with the solids, which stick to the clarifier, forming larger particles. If we were to stir wine that did not get a clarifier, we would break apart particles that have stuck together, because stirring imparts too much energy for the weak attraction.
What helps with natural (no clarifying agent) clarification is regular variations in temperature. The temp variation sets up gentle convection currents in the carboy, allowing the particles to gently bump into each other and stick together. They get bigger and settle faster.