feat: Port KPM/NFT tracking layer (FreakMatcher) to Rust
Issue body
Summary
This issue tracks the complete porting of the KPM (Keypoint Matching) / NFT
(Natural Feature Tracking) pipeline from C++ to Rust. KPM is the subsystem
responsible for marker-less image tracking using FREAK binary descriptors and
a Difference-of-Gaussians feature detector.
The work is organized as a Strangler Fig migration: a thin FFI bridge
keeps the existing C++ running while Rust replaces it layer by layer. At every
step, a numerical regression test suite validates that the Rust output matches
the C++ output to within floating-point tolerance.
Background
The C++ source lives in
WebARKitLib under
lib/SRC/KPM/ and lib/SRC/KPM/FreakMatcher/. The Rust port lives in
this repository (crates/kpm/).
What has already been ported (in crates/core/):
| Module |
File |
Status |
| ICP pose estimation |
src/icp.rs |
✅ Complete |
| Matrix math (ARMat) |
src/math.rs |
✅ Complete |
| Core types (ARParam, ARdouble) |
src/types.rs |
✅ Complete |
What this issue covers — the new crates/kpm/ crate:
| Layer |
C++ source |
Effort |
| KPM types & structs |
kpmType.h, kpm.h |
Low |
| KPM public API |
kpmHandle.cpp, kpmResult.cpp |
Low |
| Image resize helpers |
kpmUtil.cpp (genBWImage*) |
Low |
| Ref dataset I/O |
kpmRefDataSet.cpp |
Low |
| kpmMatching orchestration |
kpmMatching.cpp |
Medium |
| Math utilities |
math/indexing.h, math_utils.h |
Low |
| Linear algebra & solvers |
math/linear_algebra.h, linear_solvers.h |
Medium |
| Homography estimation |
homography_estimation/robust_homography.h |
High |
| Hough voting + feature matcher |
matchers/hough_similarity_voting.h |
Medium |
| Binary hierarchical clustering |
matchers/binary_hierarchical_clustering.h |
High |
| FREAK descriptor |
matchers/freak.h, keyframe.h |
High |
| DoG detector + Gaussian pyramid |
detectors/DoG_scale_invariant_detector.* |
High |
| VisualDatabase + facade |
matchers/visual_database.*, facade/ |
High |
Key dependency on Eigen: Eigen is used in exactly one place —
IncrementalHomographyFromLieWeights in robust_homography.h —
for a 3×3 matrix exponential. This will be replaced with either
nalgebra's matrix-exp feature or a hand-written Padé approximation.
The original authors left a // TODO: remove Eigen comment there.
Architecture
// note this was refactored: Now we have only core and wasm crate.
// kpm is inside core -> core/kpm
crates/kpm/
├── build.rs # compiles C++ FreakMatcher as static lib (FFI phase)
├── src/
│ ├── lib.rs
│ ├── kpm_c_api.h # thin C wrapper over C++ KPM (FFI phase)
│ ├── kpm_c_api.cpp # implementation of C wrapper
│ ├── kpm_ffi.rs # bindgen-generated bindings (FFI phase)
│ ├── types.rs # KpmHandle, KpmRefDataSet, FreakFeature, etc.
│ ├── backend.rs # FreakMatcherBackend trait
│ ├── cpp_backend.rs # CppFreakMatcher: implements backend via FFI
│ ├── handle.rs # KpmHandle struct (orchestration)
│ ├── ref_data_set.rs # KpmRefDataSet: generate, save, load, merge
│ ├── matching.rs # kpm_matching() + kpm_util_get_pose_binary()
│ └── freak/ # (Milestone 3+) pure Rust FreakMatcher
│ ├── mod.rs
│ ├── math.rs
│ ├── homography.rs
│ ├── hough.rs
│ ├── clustering.rs
│ ├── descriptor.rs
│ ├── detector.rs
│ └── database.rs
└── tests/
└── regression.rs # numerical regression vs C++ baseline
Cargo features
| Feature |
Purpose |
ffi-backend (default) |
Enables C++ FreakMatcher via FFI |
dual-mode |
Runs Rust and C++ in parallel and logs divergences |
When ffi-backend is disabled and the pure Rust FreakMatcher is complete,
the C++ dependency is dropped entirely.
How to contribute
- Pick a task from the milestone that interests you.
- Each task has a corresponding prompt in
CLAUDE_CODE_PLAN.md
at the repo root — feel free to use it with Claude Code.
- Every new function must have a unit test and must not break
cargo test -p kpm.
- Numerical functions must pass the regression tests in
crates/kpm/tests/regression.rs.
feat: Port KPM/NFT tracking layer (FreakMatcher) to Rust
Issue body
Summary
This issue tracks the complete porting of the KPM (Keypoint Matching) / NFT
(Natural Feature Tracking) pipeline from C++ to Rust. KPM is the subsystem
responsible for marker-less image tracking using FREAK binary descriptors and
a Difference-of-Gaussians feature detector.
The work is organized as a Strangler Fig migration: a thin FFI bridge
keeps the existing C++ running while Rust replaces it layer by layer. At every
step, a numerical regression test suite validates that the Rust output matches
the C++ output to within floating-point tolerance.
Background
The C++ source lives in
WebARKitLib under
lib/SRC/KPM/andlib/SRC/KPM/FreakMatcher/. The Rust port lives inthis repository (
crates/kpm/).What has already been ported (in
crates/core/):src/icp.rssrc/math.rssrc/types.rsWhat this issue covers — the new
crates/kpm/crate:kpmType.h,kpm.hkpmHandle.cpp,kpmResult.cppkpmUtil.cpp(genBWImage*)kpmRefDataSet.cppkpmMatching.cppmath/indexing.h,math_utils.hmath/linear_algebra.h,linear_solvers.hhomography_estimation/robust_homography.hmatchers/hough_similarity_voting.hmatchers/binary_hierarchical_clustering.hmatchers/freak.h,keyframe.hdetectors/DoG_scale_invariant_detector.*matchers/visual_database.*,facade/Key dependency on Eigen: Eigen is used in exactly one place —
IncrementalHomographyFromLieWeightsinrobust_homography.h—for a 3×3 matrix exponential. This will be replaced with either
nalgebra'smatrix-expfeature or a hand-written Padé approximation.The original authors left a
// TODO: remove Eigencomment there.Architecture
Cargo features
ffi-backend(default)dual-modeWhen
ffi-backendis disabled and the pure Rust FreakMatcher is complete,the C++ dependency is dropped entirely.
How to contribute
CLAUDE_CODE_PLAN.mdat the repo root — feel free to use it with Claude Code.
cargo test -p kpm.crates/kpm/tests/regression.rs.