Source
- URL: https://x.com/NOTimothyLottes/status/1671264312516829184
- Author: NOTimothyLottes (@NOTimothyLottes)
- Posted: 2023-06-20 21:10:37
Thread
1/ @NOTimothyLottes
How about a mega thread on the mechanics of CPU/GPU communication. Using AMDgpu based timing results on the SteamDeck an example, but relating to the larger picture of PC GFX APIs like Vulkan/etc. I don’t claim to know all, so experts feel free to add your wisdom :)
2/ @NOTimothyLottes
Both RADV and AMDVLK: Flush/invalidate mapped memory ranges is a NOP. So bus-crossing dGPU traffic to HOST_VISIBLE is automatically snooping CPU caches. The one without HOST_CACHED, is Write+Combined [WC] on store, and Uncached [UC] on read. The one with HOST_CACHED is non-WC/UC
3/ @NOTimothyLottes
In AMDgpu (the kernel driver), likely DEVICE_LOCAL maps to AMDGPU_GEM_DOMAIN_VRAM (also the carve out on APUs) and the non-DEVICE_LOCAL maps to AMDGPU_GEM_DOMAIN_GTT
4/ @NOTimothyLottes
AMD+RADV added {DEVICE_COHERENT_BIT_AMD,
DEVICE_UNCACHED_BIT_AMD} variations to the core 4 memory types. Likely to support GPU crash debug. But also provides a way to avoid needing to write-back (flush) GPU caches before CPU read. Likely AMDgpu kernel flag mapping below.
5/ @NOTimothyLottes
This AMDGPU_GEM_CREATE_CPU_GTT_USWC appear to toggle on WriteCombine [WC] for CPU store, and Uncached [US] for CPU reads (cases of HOST_VISIBLE without HOST_CACHED)
6/ @NOTimothyLottes
For review from https://chipsandcheese.com/2023/03/05/van-gogh-amds-steam-deck-apu/ Deck bandwidths: ~71 GB/s GPU, ~43 GB/s DMA, ~34 GB/s shader copy CPU<->GPU, ~25 GB/s CPU/CPU, and damn, brutal 0.27 GB/s CPU mapped GPU buffer reads, 0.71 GB/s CPU mapped GPU buffer writes
7/ @NOTimothyLottes
And going direct to AMDgpu instead of VK on the Deck shows these kinds of bandwidths (non-DEVICE_LOCAL, HOST_VISIBLE with HOST_CACHED and without). So using Write-Combined is amazingly painful for stores.
8/ @NOTimothyLottes
Implies that the choices one might make on dGPU PC don’t necessarily port over to APUs at all. Another challenge: it takes almost 7 seconds to zero-fill using a 64-bit store for() loop the 8-GiB of mapped memory. Hints at why load times are such a challenge even in the best case.
9/ @NOTimothyLottes
This all hints at why PC OS derived systems are lacking in stuffing GPU VRAM. Really need some kind of bus mastered DMA (zero-copy) between non-volatile storage (disk) and GPU DRAM to avoid this CPU-touching performance tax.
Branches: 2023-06-20-never_released-on-linux-nvidia-has-a-very-good-uvm-however-the, 2023-06-20-SheriefFYI-great-thread, 2023-06-20-NOTimothyLottes-or-for-an-apu-a-way-to-have-the-storage-device