Auto select CAGRA build algorithm for hnsw::build

[tfeher]

Configuring HNSW graph build using CAGRA is complicated, because CAGRA offers multiple build algorithms. This PR implements an automatic algorithm selection. The goal is to have a simplified API, where the user needs to set only two parameters that control graph size and quality (M and ef_construction respectively). This shall be familiar for HNSW users, and allows easier adaption of cuvs accelerated HNSW graph building.

  hnsw::index_params params;
  params.M               = 24;
  params.ef_construction = 200;
  params.hierarchy       = cuvs::neighbors::hnsw::HnswHierarchy::GPU;

  auto hnsw_index = hnsw::build(res, params, dataset_host_view);
  cuvs::neighbors::hnsw::serialize(res, "hnsw_index.bin", *hnsw_index);

If we have enough memory (host and GPU) to do both the KNN graph building and optimization in memory, then we choose in memory build, and let cagra::index_params::from_hnsw_params derive the additional configuration parameters.

If the build would require more memory then available, then we choose ACE method and let the number of partitions derived using #1603.

For host we query the os for available memory, for GPU it is assumed that the whole device memory is available.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[mfoerste4]

I did not go over all memory estimates in detail but suggest to align predictions with real data.

Is autotuning of ACE params part of a different PR? Besides the open question on the file location we might want to at least set the number of partitions dynamically.

[mfoerste4]

Is the data expected to always reside in host memory?

[julianmi]

ACE only supports host memory right now. The main reasons is that we expect the data size to be large and memory-mapped. Further, we do the partitioning and reordering on the host since there is no benefit of moving it to the GPU only to write it to disk afterwards.

Anyways, I think we can support device datasets easily since these should not end up using ACE with this heuristic. @tfeher What do you think?

[mfoerste4]

maybe we want to rename this to something generic now that the selection is hidden from the user.

[julianmi]

I did not get a chance to fully review the memory heuristics yet. I wonder how we can test it though. Should max_host_memory_gb and max_gpu_memory_gb be optional HNSW parameters that we could use to test that the expected algorithm is used based on memory limits set?

[julianmi]

Thanks @mfoerste4. These are great improvements!

[julianmi]

I added 2GB static memory consumption for both GPU & Host. This is probably too conservative given that we already reduce the available usage to 80% of the actual value. Especially on small host memory machines this increases the number of partitions drastically. We might want to get rid of one of the limits. On the GPU I would prefer the constant 2GB (e.g. for workspace memory), but on the host side the percentage seems more natural. What do you think?

I agree that the host might stick to the percentage. We have much less control over the host which has many other processes running. Also, high memory pressure reduces the performance significantly. We might want to test with increasing the limit to 90% given the added 2 GB static memory.

[achirkin]

Looks good overall. A few small suggestions below.

[tfeher]

Fixed few smaller issues.

[KyleFromNVIDIA]

Just one small request, otherwise looks good

[KyleFromNVIDIA]

Approving as I will be out for the first few days of next week and don't want to hold this up. Please address my above comment before merging.

[achirkin]

Thanks for the updates! lgtm

[tfeher]

/merge