Float8DynamicActivationFloat8WeightConfig¶
- class torchao.quantization.Float8DynamicActivationFloat8WeightConfig(activation_dtype: dtype = torch.float8_e4m3fn, weight_dtype: dtype = torch.float8_e4m3fn, granularity: Optional[Union[PerTensor, PerRow, List[Union[PerTensor, PerRow]]]] = None, mm_config: Optional[Float8MMConfig] = None, activation_value_lb: Optional[float] = None, activation_value_ub: Optional[float] = None, kernel_preference: KernelPreference = KernelPreference.AUTO, set_inductor_config: bool = True, version: int = 2)[source]¶
Configuration for applying float8 dynamic symmetric quantization to both activations and weights of linear layers.
- Parameters:
activation_dtype (torch.dtype) – The target data type for activation quantization. Default is torch.float8_e4m3fn.
weight_dtype (torch.dtype) – The target data type for weight quantization. Default is torch.float8_e4m3fn.
granularity (Optional[Union[FP8Granularity, List[FP8Granularity]]]) – The granularity for quantization. Can be either a single granularity (applied to both activations and weights) or a tuple of two granularities (one for activations, one for weights). If None, defaults to PerTensor for both. Currently both quantizations need to be the same type. And only PerTensor and PerRow are supported.
mm_config (Float8MMConfig) – Configuration for the matrix multiplication. Default uses fast accumulation.
activation_value_lb (Optional[float]) – the lower bound for activation value for calculating scale
activation_value_ub (Optional[float]) – the upper bound for activation value for calculating scale
kernel_preference (KernelPreference) – kernel preference for ops like matmul, grouped matmul etc. by defalut (KernelPreference.AUTO) it will be chosen for user based on hardware or other information, this only needs to be set in weight
set_inductor_config (bool) – if True, adjusts torchinductor settings to recommended values.
version (int) – the version of the config, version 1 is using AffineQuantizedTensor that we plan to deprecate/split, version 2 is using Float8Tensor (default)
