At Ninja Van, pricing parcels accurately required reliable dimension data — but shipper-provided measurements were often wrong, causing disputes and revenue loss. So we set out to build an automated dimensioning system using depth cameras and computer vision.
Our first attempt used background subtraction on color imagery, which failed spectacularly: parcels the same color as the surface vanished, and lighting changes constantly broke calibration. We pivoted to a depth-only approach — using point cloud data to identify parcels above a calibrated reference plane, then fitting oriented bounding boxes via Principal Component Analysis (PCA) to extract accurate length, width, and height.
Step through each stage of the computer vision pipeline. Adjust the parcel dimensions and rotation to see how the system responds. Drag to orbit the camera.
An Orbbec Astra depth camera captures 640×480 frames at 30fps. Each pixel encodes the distance from the camera in millimeters, producing a 3D point cloud of the scene below.
Euclidean clustering identifies the weighing scale’s flat surface as the largest cluster that is not the floor. This establishes a reference depth plane against which parcels are measured.
Points above the calibrated surface (closer to the camera) are classified as the parcel. Everything at or below the reference plane is background. A simple depth threshold performs this split in real time.
PCA computes the covariance matrix of the foreground points, then solves for eigenvectors. These principal axes capture the parcel’s natural orientation, regardless of how it was placed on the scale.
The eigenvectors define a rotated coordinate system aligned to the parcel. Projecting points onto these axes yields a tight-fitting bounding box, giving accurate length, width, and height measurements.
Dimensions are only published when 20 consecutive measurements have a standard deviation below 1.4 cm. This filters out transient noise from operators’ hands entering the frame during barcode scanning.
A producer–consumer architecture with a dedicated camera thread, 3 parallel consumer threads for PCA computation, and a publisher thread. Thread-safe queues and CPU-core affinity kept throughput at 30 fps.
Ported from a MacBook Pro prototype to cost-effective Intel Celeron NUCs. Raspberry Pi was too slow. Multi-core optimization and 2×2 subsampling made real-time processing feasible.
Background subtraction with mixture models on color imagery failed: similar-colored parcels became invisible, and lighting changes broke calibration. Pivoting to depth-only processing solved these issues.
Deployed across warehouses in Indonesia, Philippines, and Vietnam. Each unit ran as a systemd service with VPN connectivity, remote calibration via Socket.IO, and automated crash recovery.