torch.nn.functional.scaled_mm#
- torch.nn.functional.scaled_mm(mat_a, mat_b, scale_a, scale_recipe_a, scale_b, scale_recipe_b, swizzle_a=None, swizzle_b=None, bias=None, output_dtype=torch.bfloat16, contraction_dim=(), use_fast_accum=False)[source]#
- scaled_mm(mat_a, mat_b, scale_a, scale_recipe_a, scale_b, scale_recipe_b, swizzle_a, swizzle_b, bias, output_dtype,
contraction_dim, use_fast_accum)
Applies a scaled matrix-multiply, mm(mat_a, mat_b) where the scaling of mat_a and mat_b are described by scale_recipe_a and scale_recipe_b respectively.
Packed MNxK4 scaling uses matching
BlockWise1x32orBlockWise1x128recipes withSWIZZLE_MN_K4on both inputs. Inputs must befloat8_e4m3fnorfloat8_e5m2, with at least onefloat8_e4m3fninput. It requires cuBLASLt 13.8.1 or later (bundled with CUDA 13.4) on SM10.x/SM11.x, and supports bfloat16, float16, and float32 outputs. Bias requires bfloat16 or float16 output with matching dtype.use_fast_accumis forwarded to cuBLASLt for MNxK4. MNxK4 requires row-majormat_aand column-majormat_b. Scales are contiguousfloat8_e8m0fnutensors with 16-byte-aligned data pointers. For block sizeB(32 or 128) and outer extentO(M for A, N for B), the physical row-major byte order is[ceil(K / (4 * B)), round_up(O, 4), 4]: four consecutive K-block scales are packed together for each outer position. Each scale buffer has16 * ceil(O / 4) * ceil(K / (4 * B))elements. M need not be a multiple of four; K and N must be divisible by 16. Padding entries are ignored.- Parameters:
scale_a (Tensor | list[Tensor]) – Tensor containing decoding scaling factors for mat_a
scale_recipe_a (_ScalingType | list[_ScalingType]) – Enum describing how mat_a has been scaled
scale_b (Tensor | list[Tensor]) – Tensor containing decoding scaling factors for mat_b
scale_recipe_b (_ScalingType | list[_ScalingType]) – Enum describing how mat_b has been scaled
swizzle_a (_SwizzleType | list[_SwizzleType] | None) – Enum describing the swizzling pattern (if any) of scale_a
swizzle_b (_SwizzleType | list[_SwizzleType] | None) – Enum describing the swizzling pattern (if any) of scale_b
bias (Tensor | None) – optional bias term to be added to the output
output_dtype (dtype | None) – dtype used for the output tensor
contraction_dim (list[int] | tuple[int, ...]) – Must be empty or
(1, 0)(equivalent negative dimensions are also accepted).use_fast_accum (bool) – Request tensor-core fast accumulation for supported recipes. Hopper
BlockWise1x128/BlockWise128x128requireFalse; MNxK4 forwards the flag to cuBLASLt.
- Return type:
Note
SwizzleType.SWIZZLE_128is the gfx1250 layout for MX (block 32) and NVFP4 (block 16). Each block scale hasrows * round_up(K / block, 128 / block)elements: K-groups of128 / blockscales, stored group-major. WhenKis a multiple of 128 andK > 128, linear scales have that same count, so passing them withSWIZZLE_128is accepted and the result is wrong. Reorder the scales.torch._scaled_mmcannot name a swizzle; on gfx1250 use this function and passSWIZZLE_128.