RPC Per Host Layer Count Specification

[sigkill]

Hello,

I have a test environment that consists of a few nvidia Xavier AGX nodes. Some have 32GB ram, some have 64GB ram. Additionally, I am running two HP Z8G4 nodes with 1.5TB of ram each, with clusters of 4060 ti cards.

Due to the memory available, I am not able to divide the environment evenly, and would obviously like to have some nodes take more layers than others.

Additionally, given all of these variables, what would be the absolute fastest system to have in front of all of these rpc hosts, provided there was a simple way to specify the number of layers for each host, and get this working in a way where 64GB nodes effectively take 2x number of layers a 32GB rpc node takes?

Additionally, I am curious if I can emulate the unified ram configuration on the HP Z8G4 nodes as well, but that is less important than being able to specify the layer count per host for me right now.

Thank you in advance!

[rgerganov]

Due to the memory available, I am not able to divide the environment evenly, and would obviously like to have some nodes take more layers than others.

You can use the -ts SPLIT, --tensor-split SPLIT command line option to specify how tensors are split across devices and RPC servers. See here for details.

[rgerganov]

It could be possible with the --override-tensor option which @slaren added, see PR #11397 for details.

[okuvshynov]

thank you!

[sigkill]

Thank you, going to try a few of these out tonight!

[B4night]

@okuvshynov Hi, I am currently working on expert parallelism with DeepSeek R1. Do you have any progress on that?

[okuvshynov]

@B4night - I did not, I haven't got the hardware where it might be useful.

[sigkill]

Would it be possible to have redudant layers?