A collection job may work on ten targets and fail at thousands. Failure classification and deduplication, rate limiting and concurrency, resumability, egress failures, consistency checks, and a few key monitoring metrics are the engineering issues that emerge at scale.
A collection script can run smoothly against ten targets, then start losing success rate when expanded to thousands. You add retries, rotate proxies, tune concurrency, and the same problems keep coming back. Dig deeper and the bottleneck is often not parsing logic, but several missing engineering layers. At small scale, these issues may never show up.
Classify failures first so retries actually help
Failures are inevitable in data collection. The key is to classify them: network jitter and connection resets can be retried immediately; temporary rate limits should be retried after backoff; if a page structure change makes parsing return empty results, ten thousand retries will not help, so record it and alert; if the target simply does not exist, mark it complete and move on; if an environment or egress cannot start, switch to another one and retry.
Retrying everything is one of the easiest mistakes to make. It hides problems that need human intervention behind loops while wasting quotas and egress capacity. Backoff matters too: increase the retry interval, otherwise a batch of jobs may all hit the target again in the same time window and make rate limiting even worse.
Retries also lead directly to deduplication. A job may execute multiple times because of retries, so every job needs a stable unique identifier, such as a normalized URL value, and database writes should be idempotent on that identifier. Otherwise, more retries simply produce more dirty data.
Rate limiting and concurrency are different things
Increasing concurrency does not guarantee higher throughput. Three constraints act at the same time: how much the target site can tolerate before rate limiting reduces total throughput, local memory and CPU, and whether one environment or session can run multiple jobs concurrently.
A more stable approach is to start with low concurrency and increase load gradually while plotting success rate and response time together to find the point where performance clearly deteriorates. Rate limiting is separate: it controls the access pace to the same target and is not the same as global concurrency. When one batch spreads requests across multiple sites, each site needs its own pacing policy.
Resuming depends on persistent state
For a job that runs for hours, an interruption is normal; restarting from the beginning is often too expensive. The prerequisite is persisted state: pending, running, completed, plus retry count, next eligible run time, and error type. When the process starts, it should restore the queue from storage instead of rebuilding it from memory.
Keeping the queue only in memory is a common implementation that appears to work. Once the process dies, every queued job disappears and the counts no longer reconcile.
Handle proxy and egress failures separately
Targets blocking an egress, proxies going offline, and regional nodes drifting will keep happening at scale. They are routine conditions, not rare exceptions. Treat egress as a replaceable resource: when a job fails, first determine whether the target is rate limiting or the egress is unavailable; back off for the former, switch egress and retry for the latter. Track the failure rate of each egress as well and remove batches that clearly degrade.
Conversely, if every job shares one egress, a single job can break the path and affect everything behind it. Troubleshooting then requires working backward through logs to identify which job triggered the problem.
Data consistency checks
A successful run does not mean the data is correct. After writing to storage, you should be able to answer a few questions: does the completed-job count match the number of stored rows, what share of parse results is empty, has the missing rate for critical fields risen abnormally, and how many duplicate rows exist?
These checks do not need to be complicated. Batch-level sampling is enough, but someone must review the results. At scale, wrong data can be more troublesome than no data.
What to monitor
Do not collect too many metrics. A few that reflect system health are enough.
- Success rate and the distribution of failure types, to see which errors are increasing
- Queue length and average wait time; steadily growing backlog means intake and processing capacity are mismatched
- Number of active environments and related processes; long one-way growth often indicates leaks in resource cleanup
- Output per unit of time, to determine whether rate limiting is suppressing throughput
- Egress failure rate, to decide whether a group of nodes should be replaced
If any of these metrics keeps moving in one direction for a long time, check resource cleanup and retry logic first.
Separate the environment layer
Viewed together, these issues point to the same conclusion: the environment layer should be managed independently from scripts. Environment pooling requires centralized scheduling instead of environments being scattered across individual scripts; resource cleanup requires queryable state instead of scripts trying to recover on their own; retrying with a different environment or egress only works when environments can be scheduled independently.
Scripts should handle logic; the environment layer should handle resources and identity. In this kind of architecture, PurpleMark provides that layer with environment resources that can be created in batches, bound to independent network egresses, and queried for status.
Compliance boundaries
The ability to scale does not mean data can be collected arbitrarily. Follow the target site's robots rules and terms of service, do not collect personal information, do not bypass technical protection measures, and control request rates so normal service is not disrupted. Stability is one technical question; whether collection is permitted is another. Both must be satisfied.
