torch.nn.functional.scaled_grouped_mm#
- torch.nn.functional.scaled_grouped_mm(mat_a, mat_b, scale_a, scale_recipe_a, scale_b, scale_recipe_b, swizzle_a=None, swizzle_b=None, bias=None, offs=None, output_dtype=torch.bfloat16, contraction_dim=(), use_fast_accum=False)[source]#
- scaled_grouped_mm(mat_a, mat_b, scale_a, scale_recipe_a, scale_b, scale_recipe_b, swizzle_a, swizzle_b, bias, offs,
output_dtype, use_fast_accum)
Applies a grouped scaled matrix-multiply, grouped_mm(mat_a, mat_b) where the scaling of mat_a and mat_b are described by scale_recipe_a and scale_recipe_b respectively.
When
scale_recipe_aandscale_recipe_bare each the single recipeTensorWise, each float32 scale contains either a single value shared by all groups or one value per group in a contiguous 1D tensor. Each input may use either scale layout. Both inputs must befloat8_e4m3fnorfloat8_e5m2, with at least onefloat8_e4m3fninput. At most 1024 groups are supported. TensorWise scaling requires CUDA 13.4 or later on SM9.x-SM11.x; swizzle metadata is ignored. It supports bfloat16, float16, and float32 outputs.Hopper block scaling requires CUDA 13.4 or later on SM9.x and supports recipe pairs 1x128/1x128, 1x128/128x128, and 128x128/1x128 with float32 scales and two
float8_e4m3fninputs. Each group packs its scales separately. For outer extentO(M for A, N for B), letL = ceil(K / 128). A 1x128 group storesO * Lvalues with the outer dimension contiguous; a 128x128 group storesround_up(L, 4) * ceil(O / 128)values with the padded K-block dimension contiguous. Scales are contiguous, with 16-byte-aligned group starts. A 3D operand uses shape(groups, values_per_group); a 2D operand uses a 1D tensor concatenating the flattened group scales in offset order, covering only the groups selected byoffs. M and N must be multiples of 4. These recipes support bfloat16, float16, and float32 outputs and requireuse_fast_accum=False. Swizzle metadata must be omitted orNO_SWIZZLE.Packed MNxK4 scaling uses the recipes, byte order, element count, and alignment described in
scaled_mm(), independently for each group. A 3D operand uses scale shape(groups, elements_per_group); a 2D operand concatenates each group’s flattened scale buffer in offset order. Each group starts at a 16-byte boundary. Per-group M need not be a multiple of four. K and the total N dimension must be divisible by 16, including whenoffssplits N. Group output starts must be 16-byte aligned: starting N offsets must be multiples of 8 for bfloat16/float16 output or 4 for float32 output. Invalid group output alignment triggers a device assertion. It requires cuBLASLt 13.8.1 or later (bundled with CUDA 13.4) on SM10.x/SM11.x and accepts bfloat16, float16, and float32 outputs.use_fast_accumis forwarded to cuBLASLt for MNxK4.Other recipes require bfloat16 output.
- Parameters:
scale_a (Tensor | list[Tensor]) – Tensor containing decoding scaling factors for mat_a
scale_recipe_a (_ScalingType | list[_ScalingType]) – Enum describing how mat_a has been scaled
scale_b (Tensor | list[Tensor]) – Tensor containing decoding scaling factors for mat_b
scale_recipe_b (_ScalingType | list[_ScalingType]) – Enum describing how mat_b has been scaled
swizzle_a (_SwizzleType | list[_SwizzleType] | None) – Enum describing the swizzling pattern (if any) of scale_a
swizzle_b (_SwizzleType | list[_SwizzleType] | None) – Enum describing the swizzling pattern (if any) of scale_b
bias (Tensor | None) – optional bias term to be added to the output
offs (Tensor | None) – optional offsets into the source tensors denoting group start indices
output_dtype (dtype | None) – output dtype; the cuBLASLt recipes described above (including
TensorWisealone on both inputs) accept bfloat16, float16, or float32; other recipes require bfloat16contraction_dim (list[int] | tuple[int, ...]) – describe which dimensions are in the matmul.
use_fast_accum (bool) – Request tensor-core fast accumulation for cuBLASLt recipes (including MNxK4) and CUDA RowWise scaling. Hopper
BlockWise1x128/BlockWise128x128requireFalse. MSLK recipes and ROCm ignore this flag.
- Return type: