Why does larger model loaded into GPU with MMAP option work very slow?

[alex-ie]

I saw my memory usage went to ~98% during LLM model load to GPU (I had not much free RAM at that moment) and then "Terminated" in terminal. So I've added MMAP option in koboldcpp and it loaded, albeit longer. The delay was on "Model warm up" message in terminal. But it took like ~20 seconds to generate 1st token.

Smaller model that loaded without MMAP worked equally fast loaded with and without MMAP. With MMAP RAM usage for smaller model was lower all the way (after load too). For larger model memory usage was about same as for smaller (both when loaded with MMAP). Larger worked fine (fast) before - when free RAM was abundant.

Questions: 1) why larger model so slow if should be fully in VRAM after load? If not in VRAM, why? Can I fully put LLM into VRAM with koboldcpp (or if not with other tool)? 2) seems after load into GPU (Vulkan on NVIDIA) RAM taken to model load is not released. Why? 3) What does it mean in koboldcpp GUI MMAP help says "model will not be unloadable"? How to upload models if MMAP is not used?

[LostRuins]

1. Loading speed largely depends on your storage (e.g. SSD vs HDD).

2. yes you can do a full offload if you have enough VRAM. Check in task manager. If you share your GPU we can advise.

3. if you use mmap to load, the model will stay in memory until koboldcpp.exe closes. This may not be desired if you plan to swap model at runtime e.g. using the admin endpoint, but if you don't change model while using this is not an issue.