opencl: improve profiling

[lhez]

• Wait for profiling events and collect profiling data when model execution is done. This way, the displayed performance numbers are more close to the true performance.
• Generate a chrome trace in addition to csv.

[zhouwg]

sorry to bother you, how can I mark a specified PR as ready for review? thanks.
one more thing, can Qualcomm's expert help to review a PR of Qualcomm's QNN backend for llama.cpp from an independent programmer(here is me and I already bought Qualcomm's Snapdragon 8gen3 equipped Android phone for personal learning dev activity and plan to buy a Snapdragon 8 Elite equipped phone for further dev activity later)?I know your team already has a senior technical expert whom knows everything about ggml-qnn and I hope can get help from your colleague. thanks too much.

[max-krasnyansky]

Looks good, and reminds me that I wanted to integrate graph-profiler branch with opencl and cpu backends.

[max-krasnyansky]

sorry to bother you, how can I mark a specified PR as ready for review? thanks.

If you start a Draft PR there is a way to mark it ready. qnn-backend PR is not marked as draft.

one more thing, can Qualcomm's expert help to review a PR of Qualcomm's QNN backend for llama.cpp from an independent programmer(here is me and I already bought Qualcomm's Snapdragon 8gen3 equipped Android phone for personal learning dev activity and plan to buy a Snapdragon 8 Elite equipped phone for further dev activity later)?I know your team already has a senior technical expert whom knows everything about ggml-qnn and I hope can get help from your colleague. thanks too much.

I've been keep an eye on it. In general, I'd say QNN is not the right solution here but I'll take another look.

[zhouwg]

thanks for your valuable time and attention on ggml-qnn(Qualcomm's backend for llama.cpp through Qualcomm QNN SDK) and my third formal PR of ggml-qnn.

1. is "QNN is not the right solution here" means we can't utilize the Hexagon NPU maximally through Qualcomm's QNN SDK in llama.cpp? even through the "second tech approach(mapping the entire ggml cgraph to a single QNN graph)"?

2. that's why the GPU performance in that PR(or all similar PR) is so bad at the same time Intel's ggml-sycl is so good? accordingly, that's why Qualcomm provide ggml-opencl for Qualcomm's GPU backend(right solution for Qualcomm GPU backend in llama.cpp)?

3. that's why the NPU performance in that PR(or all similar PR) is so bad?

4. that's why Qualcomm didn't provide an official ggml-qnn PR through the general approach in Intel's ggml-sycl or the second tech approach(mapping the entire ggml cgraph to a single QNN graph) with QNN SDK currently?

5. from this diagram,
   

we can see the key-point of "how to utilize the Hexagon NPU maximally through QNN SDK" is that we should compose an ideal single QNN graph from an entire/complete ggml cgraph. as my personal understanding, that's exactly the core principle(Qualcomm's dedicated binary tool convert a specified LLM model to a prepared QNN graph) in QNN SampleApp or Genie stack.

I don't understand some tech questiones:

• why "QNN is not the right solution here"? is there any other AI software stack or SDK provided by Qualcomm for llama.cpp's Hexagon NPU backend?
• or the first tech approach(general approach in Intel's ggml-sycl) and the second tech approach(mapping a complete/entire ggml cgraph to a single QNN graph through QNN SDK) in that PR are both incorrect?
• or the second tech approach(mapping the entire/complete ggml cgraph to a single QNN graph through QNN SDK) is correct path but the practice in that PR is not exactly correct? I understand the shared-buffer or memory-pool also should be used in the second tech approach.

I or/and the entire llama.cpp community need your further guidance and help.

thanks so much again.

[zhouwg]

@max-krasnyansky, thanks so much for your valuable guidance/correction on direction.

I think I know something about the third tech approach of "utilize the Hexagon NPU maximally", in other words, the Hexagon DSP SDK should be used in the third tech approach, which is exactly similar to what your excellent engineering team did with ggml-opencl, or which is exactly similar to what I did with video decoding hardware acceleration many years ago(it's also a DSP chip).

my guess might-be not correct, accordingly, it's greatly appreciated that you can give me/the llama.cpp community a clear explanation or a roughly confirmation of the third tech approach.

[zhouwg]

sorry to bother you, how can I mark a specified PR as ready for review? thanks.

If you start a Draft PR there is a way to mark it ready. qnn-backend PR is not marked as draft.

one more thing, can Qualcomm's expert help to review a PR of Qualcomm's QNN backend for llama.cpp from an independent programmer(here is me and I already bought Qualcomm's Snapdragon 8gen3 equipped Android phone for personal learning dev activity and plan to buy a Snapdragon 8 Elite equipped phone for further dev activity later)?I know your team already has a senior technical expert whom knows everything about ggml-qnn and I hope can get help from your colleague. thanks too much.

I've been keep an eye on it. In general, I'd say QNN is not the right solution here but I'll take another look.

yes, you are absolutely correct, thanks so much. I'll try to test it on a rooted 8gen3 phone to get better NPU performance(closer to the default ggml backend or better then the default ggml backend).

[updated on 03/21/2025,23:54] offload to cDSP directly is really much faster than QNN-CPU/QNN-GPU/QNN-NPU backend. now GGML_OP_ADD works fine on cDSP and verified with llama-cli: the NPU performance is good.

ggml-hexagon-offload-to-cdsp-directly.mp4

I think this approach is exactly similar to ggml-opencl: we need write some "kernels" on cDSP and then offload specified ggml ops to cDSP directly. there is a question in this approach: build the entire source code is not easy because one of parts are running on AP(arm-cpu) and one of parst are running on cDSP(NPU), at the same time, opencl provide a mature mechanism to manage/compile "kernels" naturally.

hope mulmat can also works fine on cDSP and NPU performance is also good tomorrow, then that PR can be reviewed formally.

[updated on 03/22/2025,23:22] source code has been submitted in that PR. there is an unknown issue about mulmat on cDSP. another issue in that PR: I didn't find a general approach to compile/build the hexagon kernel functions in that PR at the moment and a manual method can works fine in my local dev envs.

[zhouwg]

[updated on 03/24/2025,15:32]mulmat performance between QNN-NPU and cDSP:

mulmat performance between QNN-NPU and cDSP

case-1:
mulmat with QNN-NPU:

mulmat with cDSP(with naive algorithm):

case-2:
mulmat with QNN-NPU:

mulmat with cDSP(with naive algorithm):

case-3:
mulmat with QNN-NPU:

mulmat with cDSP(with algorithm from ggml-dsp(a tiny customized ggml on cDSP), but the compute result is not correct):

mulmat with cDSP(with naive algorithm):

case-4:
mulmat with QNN-NPU:

mulmat with cDSP(with naive algorithm):

@max-krasnyansky, I think this testcase can verify what you said "QNN is not the right solution here" is absolutely correct if there is no misunderstanding or mistake in this testcase/experiment: we can clearly see that performance of mulmat with cDSP is much faster then mulmat with QNN-NPU.

unfortunately, this hand-written implementation of mulmat on cDSP is very naive(without any optimization) and can only pass various case in UT with self-made command line program, it can't works with llama-cli.

I think AI experts can have chance to improve the mulmat algorithm on cDSP if that PR can be approved: this approach is exactly what you pointed out, developers and AI experts can do a lot on the cDSP rather than based on the black box in the QNN SDK.

I'd like to make a little contribution to Qualcomm although I'm an independent programmer, thanks so much!

[zhouwg]

[updated on 03/25/2025,21:10]
now I successfully ported the original ggml_compute_forward_add(here is ggml_compute_forward_add_f32, other can be added easily) and ggml_compute_forward_mul_mat on cDSP.

case-1: small matrix ( cDSP's performance is much much faster than QNN-NPU)
./scripts/build-run-android.sh run_testop MUL_MAT 2 with inference_approach = 1 in scripts/ggml-qnn.cfg (through cDSP directly)

./scripts/build-run-android.sh run_testop MUL_MAT 2 with inference_approach = 0 in scripts/ggml-qnn.cfg (through QNN)

case-2: big matrix

mulmat UT through QNN-NPU:

mulmat UT through cDSP directly:

[zhouwg]

[updated on 03/27/2025,17:49]
NPU performance between QNN and cDSP ------ GGML_OP_ADD

case-1:
provide a special case to avoid complex algorithm(padding operation) on cDSP side:
matrix src0:fp32 1024 * 1024
matrix src1:fp32 1024 * 1024

with QNN-NPU:

with cDSP (using algorithm from original ggml):

with cDSP(using Hexagon SIMD instructions with naive algorithm, without algorithm from original ggml):

with cDSP(using Hexagon SIMD instructions with algorithm from original ggml):

with cDSP(using Hexagon SIMD instructions with multithread algorithm from original ggml)
TBD(because HVX multithread programming is not easy at the moment, and already got 10x performance gain with GGML_OP_ADD on cDSP)

[updated on 03/28/2025, 18:53] I'm sorry that the data in above testcase is not exactly correct/accurate because there is a bug in my code with HVX SIMD intrinsic:

because qfloat32 is different with standard IEEE-754 float.

the performance of GGML_OP_ADD with cDSP is still much faster than GGML_OP_ADD with QNN-NPU although Q6_Vsf_equals_Vqf32 should be used for convert qf32 to fp32.

NPU performance between QNN and cDSP ------ GGML_OP_ADD

case-1:
provide a special case to avoid complex algorithm(padding operation) on cDSP side:
matrix src0:fp32 1024 * 1024
matrix src1:fp32 1024 * 1024

with QNN-NPU:

with cDSP (using algorithm from original ggml):

with cDSP(using Hexagon SIMD instructions with naive algorithm, without algorithm from original ggml):

with cDSP(using Hexagon SIMD instructions with algorithm from original ggml):

HVX SIMD intrinsic is not easy and I'm learning them(the principle is similar to Intel's x86 MMX/SSE and I have some experiences with Intel's x86 ISA and MMX/SSE instructions), I think the peformance of GGML_OP_ADD on cDSP will more faster if Hexagon L2-cache prefetch and HVX multithreading are both enabled/utilized, might-be faster than Kryo CPU. the principle here should be same to GGML_OP_MUL_MAT/other performance-sensitive GGML op on cDSP.

[zhouwg]

1.another more obvious case:

GGML_OP_ADD performance comparison between QNN-NPU and cDSP in real LLM inference (test phone is a Snapdragon 8 Gen3 Android phone and a Snapdragon 8 Elite(aka 8 Gen4) Android phone, test model is qwen1_5-1_8b-chat-q4_0.gguf)

  git clone https://github.com./zhouwg/ggml-hexagon
  cd ggml-hexagon
  git checkout pr_to_upstream
  cd ggml-hexagon
  ./scripts/build-run-android.sh build

LLM inference through HWACCEL_CDSP(offload GGML_OP_ADD to cDSP directly)

./scripts/build-run-android.sh run_llamacli

LLM inference through HWACCEL_QNN(offload GGML_OP_ADD to QNN_NPU)(modify hwaccel_approach to 0 --- hwaccel approach through QNN--- in scripts/ggml-hexagon.cfg and then run)

./scripts/build-run-android.sh run_llamacli

2. all quantized mulmat can works fine on cDSP side in my local dev envs at the moment(since April/04/2025), I'm working on improve mulmat's performance with HVX SIMD instructions and HVX multithreading.