On a single DGX Spark machine, generating text with the Ling-3.0-flash model was slowed by reading model weights from memory, mainly in the expert layers and the final output layer.

They swapped in faster expert kernels, stored the output layer in a smaller number format to halve its reads, and used the model's built-in multi-token prediction to draft and check several tokens at once.

In mixture-of-experts models, each expert gets a different number of tokens, known only at runtime. Padding every expert to the same size wastes compute and memory.

Two backends that turn this uneven batch of matrix multiplications into one Group-GEMM operation: one built on AMD's hipBLASLt library, and a portable one built on Triton that runs on Nvidia and AMD GPUs.

Teams that scale AI work with Ray worried that moving to a different GPU maker would mean rewriting code that touches the hardware.

A tested stack of ROCm 10.0.0, Ray 2.58.0, PyTorch 2.12 and vLLM, with an example that serves a language model across several Instinct GPUs behind an OpenAI-compatible endpoint.

General models gave wrong local answers, such as telling a French resident who cannot travel to apply for a passport by post, which is not allowed.

A workflow that uses synthetic residents based on census data, plus official procedure text, to create training conversations and fine-tune a Nemotron model with LoRA.