torch.nn.functional.grouped_mm#
- torch.nn.functional.grouped_mm(mat_a, mat_b, *, offs=None, bias=None, out_dtype=None)[source]#
Computes a grouped matrix multiply that shares weight shapes across experts but allows jagged token counts per expert, which is common in Mixture-of-Experts (MoE) layers. Both
mat_aandmat_bmust be 2D or 3D tensors that satisfy the physical layout restrictions of grouped GEMM kernels. Inputs may betorch.float16,torch.bfloat16, ortorch.float32values. CUDA execution is supported on devices with .- Parameters:
mat_a (Tensor) – Left operand. When 2D, its leading dimension is sliced into groups according to
offs. When 3D, its first dimension enumerates the groups directly andoffsmust beNone.mat_b (Tensor) – Right operand. When both operands are 2D (e.g., MoE weight-gradient updates), the trailing dimension of
mat_aand the leading dimension ofmat_bare partitioned according to the sameoffstensor. For the common forward pass (out = input @ weight.T)mat_bis 3D with shape(num_groups, K, N). If expert weights are stored in the standardnn.Linearlayout(num_groups, N, K), passweight.transpose(-2, -1)asmat_b.offs (Tensor | None) – Optional 1D tensor of monotonically increasing
int32offsets that delimit the jagged dimension of any 2D operand.offs[i]marks the end of groupiandoffs[-1]must be strictly less than the total length of that operand’s sliced dimension; elements beyondoffs[-1]are ignored.bias (Tensor | None) – Reserved for future use. It must currently be
None.out_dtype (dtype | None) – Optional output dtype. When provided, it must match the dtype of
mat_a.
- Returns:
A tensor containing the concatenated results of each per-group GEMM with shape inferred from the operands and
offs.- Return type: