Soil Particle Tracking Pipeline
Multi-object tracking that held identity across 3,500+ particles and cut a 40-hour cycle to 2.
95%
Faster analysis
40hr to 2hr per cycle
0.82
mAP
3,500+ tracked entities
90%
Less setup time
Docker + Terraform
99.9%
Uptime
See it work
The problem
Geotechnical researchers were measuring soil particle movement by hand, frame by frame. A single experimental cycle consumed around forty hours of manual annotation, which capped how many experiments could realistically be run.
Approach
YOLOv8 handles detection and FairMOT supplies re-identification embeddings, so a particle that disappears behind another can be matched back to its original track when it reappears rather than being counted as a new object.
The pipeline runs on AWS Lambda against S3, processing segments in parallel. Splitting the work by segment is what turned a forty-hour serial annotation cycle into a two-hour parallel one.
Docker and Terraform pin the environment and the infrastructure together, which eliminated the configuration drift that had made earlier runs difficult to reproduce and cut new-environment setup time by 90%.
CloudWatch instrumentation covers every stage, giving the 99.9% uptime figure something behind it and making failed segments visible immediately rather than at the end of a run.
Architecture
Decisions and tradeoffs
Pair detection with re-identification
YOLOv8 for detection, FairMOT embeddings for identity.
- Why
- Detection alone loses a particle the moment it is occluded. Identity embeddings let the tracker recover the original track instead of fragmenting it.
- What it cost
- Higher compute per frame and a second model to tune and keep in step with the detector.
Parallelize by video segment
Independent Lambda invocations per segment.
- Why
- Segments are largely independent, which makes the throughput win close to linear and is the bulk of the 95% time reduction.
- What it cost
- Tracks that span a segment boundary need stitching, which is extra logic and a source of subtle error.
Treat reproducibility as infrastructure
Docker images plus Terraform-defined environments.
- Why
- Research conclusions are only as good as the ability to re-run them. Pinning both code and infrastructure removed drift as a variable.
- What it cost
- Slower initial setup and a requirement that collaborators work through the toolchain rather than around it.