Skip to content

[Feature] KPM/NFT Tracking Port to Rust #20

@kalwalt

Description

@kalwalt

feat: Port KPM/NFT tracking layer (FreakMatcher) to Rust

Issue body

Summary

This issue tracks the complete porting of the KPM (Keypoint Matching) / NFT
(Natural Feature Tracking) pipeline from C++ to Rust. KPM is the subsystem
responsible for marker-less image tracking using FREAK binary descriptors and
a Difference-of-Gaussians feature detector.

The work is organized as a Strangler Fig migration: a thin FFI bridge
keeps the existing C++ running while Rust replaces it layer by layer. At every
step, a numerical regression test suite validates that the Rust output matches
the C++ output to within floating-point tolerance.


Background

The C++ source lives in
WebARKitLib under
lib/SRC/KPM/ and lib/SRC/KPM/FreakMatcher/. The Rust port lives in
this repository (crates/kpm/).

What has already been ported (in crates/core/):

Module File Status
ICP pose estimation src/icp.rs ✅ Complete
Matrix math (ARMat) src/math.rs ✅ Complete
Core types (ARParam, ARdouble) src/types.rs ✅ Complete

What this issue covers — the new crates/kpm/ crate:

Layer C++ source Effort
KPM types & structs kpmType.h, kpm.h Low
KPM public API kpmHandle.cpp, kpmResult.cpp Low
Image resize helpers kpmUtil.cpp (genBWImage*) Low
Ref dataset I/O kpmRefDataSet.cpp Low
kpmMatching orchestration kpmMatching.cpp Medium
Math utilities math/indexing.h, math_utils.h Low
Linear algebra & solvers math/linear_algebra.h, linear_solvers.h Medium
Homography estimation homography_estimation/robust_homography.h High
Hough voting + feature matcher matchers/hough_similarity_voting.h Medium
Binary hierarchical clustering matchers/binary_hierarchical_clustering.h High
FREAK descriptor matchers/freak.h, keyframe.h High
DoG detector + Gaussian pyramid detectors/DoG_scale_invariant_detector.* High
VisualDatabase + facade matchers/visual_database.*, facade/ High

Key dependency on Eigen: Eigen is used in exactly one place —
IncrementalHomographyFromLieWeights in robust_homography.h
for a 3×3 matrix exponential. This will be replaced with either
nalgebra's matrix-exp feature or a hand-written Padé approximation.
The original authors left a // TODO: remove Eigen comment there.


Architecture

// note this was refactored: Now we have only core and wasm crate.
// kpm is inside core -> core/kpm
crates/kpm/
├── build.rs              # compiles C++ FreakMatcher as static lib (FFI phase)
├── src/
│   ├── lib.rs
│   ├── kpm_c_api.h       # thin C wrapper over C++ KPM (FFI phase)
│   ├── kpm_c_api.cpp     # implementation of C wrapper
│   ├── kpm_ffi.rs        # bindgen-generated bindings (FFI phase)
│   ├── types.rs          # KpmHandle, KpmRefDataSet, FreakFeature, etc.
│   ├── backend.rs        # FreakMatcherBackend trait
│   ├── cpp_backend.rs    # CppFreakMatcher: implements backend via FFI
│   ├── handle.rs         # KpmHandle struct (orchestration)
│   ├── ref_data_set.rs   # KpmRefDataSet: generate, save, load, merge
│   ├── matching.rs       # kpm_matching() + kpm_util_get_pose_binary()
│   └── freak/            # (Milestone 3+) pure Rust FreakMatcher
│       ├── mod.rs
│       ├── math.rs
│       ├── homography.rs
│       ├── hough.rs
│       ├── clustering.rs
│       ├── descriptor.rs
│       ├── detector.rs
│       └── database.rs
└── tests/
    └── regression.rs     # numerical regression vs C++ baseline

Cargo features

Feature Purpose
ffi-backend (default) Enables C++ FreakMatcher via FFI
dual-mode Runs Rust and C++ in parallel and logs divergences

When ffi-backend is disabled and the pure Rust FreakMatcher is complete,
the C++ dependency is dropped entirely.


How to contribute

  1. Pick a task from the milestone that interests you.
  2. Each task has a corresponding prompt in CLAUDE_CODE_PLAN.md
    at the repo root — feel free to use it with Claude Code.
  3. Every new function must have a unit test and must not break
    cargo test -p kpm.
  4. Numerical functions must pass the regression tests in
    crates/kpm/tests/regression.rs.

Metadata

Metadata

Assignees

Projects

Status

In progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions