[question] usage in Rust/ONNX?

[louis030195]

Hi, I'm interested in using Colpali in https://github.com./mediar-ai/screenpipe

to index 100,000 images/day on consumer hardware

I'm going to try the python implementation and if it makes sense see how to make it usable in our Rust codebase through ONNX or https://github.com./huggingface/candle, if you have any advice it would be most welcome!

related: huggingface/candle#2557

[ManuelFay]

Hey !
I am not super knowledgeable about these things sadly (got to get into it) but I know that:

• https://huggingface.co/akshayballal/colpali-v1.2-merged-onnx
  Here you have an ONNX implementation

• Add PaliGemma Support ggml-org/llama.cpp#7553
  Here you have a PR to add paligemma to llama.cpp

these sort of things might point you in the right direction !

Using ColQwen2, you can also lower the resolution to get better speed and RAM usage and therefore get a better throughput trading off for performance.

Interested in whatever you come up with !
Cheers,
Manu

[ManuelFay]

Also @louis030195

In rust:

https://github.com./huggingface/candle/blob/main/candle-examples/examples/colpali/README.md