Laborers are your most flexible resource. While it's good to keep a small pool unassigned, don't be afraid to temporarily reassign them to critical tasks. For example, if you're running low on firewood, pull some laborers to the Woodcutter. If a new building needs to be constructed quickly, assign more laborers to building. Once the task is done, return them to their original jobs or leave them as general laborers.
Use laborers to clear specific resource nodes (stone, iron, logs) that are far from your main production buildings. This allows you to gather these resources without committing permanent workers to a Mine or Quarry too early. This micro-management can be the difference between survival and collapse in early game crises.