recipes/foreach_map

Run in Google Colab

Colab

Download Notebook

Notebook

View on GitHub

GitHub

Explicit horizontal fusion with foreach_map and torch.compile

Author: Michael Lazos

Horizontal fusion is a key optimization in ML compilers. In eager,

this is typically expressed using the torch._foreach* ops which parallelizes operations across a list of tensors. However, supporting all possible permutations of arguments is quite difficult (e.g. mixtures of scalars and lists). Foreach_map allows conversion of any pointwise op in torch to a horiztonally fused foreach variant. In this tutorial, we will demonstrate how to implement the Adam optimizer with foreach_map to generate a fully fused kernel.

Note

This recipe describes a prototype feature. Prototype features are typically at an early stage for feedback and testing and are subject to change.

Prerequisites

• PyTorch v2.7.0 or later

Model Setup

For this example, we’ll use a simple sequence of linear layers. We instantiate an independent copy to compare the two optimizer implementations.

import torch

# exit cleanly if we are on a device that doesn't support ``torch.compile``
if torch.cuda.get_device_capability() < (7, 0):
    print("Exiting because torch.compile is not supported on this device.")
    import sys
    sys.exit(0)

# Create simple model
model = torch.nn.Sequential(
    *[torch.nn.Linear(1024, 1024, False, device="cuda") for _ in range(10)]
)
model_copy = torch.nn.Sequential(
    *[torch.nn.Linear(1024, 1024, False, device="cuda") for _ in range(10)]
)
input = torch.rand(1024, device="cuda")

# run forward pass
output = model(input)
output_copy = model_copy(input)

# run backward to populate the grads for our optimizer below
output.sum().backward()
output_copy.sum().backward()

Helper functions for foreach_map implementation

In this section, we’ll begin our implementation of the Adam optimizer.

from torch._higher_order_ops.foreach_map import foreach_map

# Helper function to extract optimizer states from a torch.optim.Adam instance
def get_inputs(optim):
    steps = []
    params = []
    grads = []
    exp_avgs = []
    exp_avg_sqs = []
    for group in optim.param_groups:
        for p in group["params"]:
            params.append(p)
            grads.append(p.grad)
            state = optim.state[p]
            exp_avgs.append(state["exp_avg"])
            exp_avg_sqs.append(state["exp_avg_sq"])
            steps.append(state["step"])

    return steps, params, exp_avgs, exp_avg_sqs


# Functions to update the different optimizer states
def update_exp_avg_sq(exp_avg_sq, grad, beta2):
    return exp_avg_sq.mul(beta2).addcmul(grad, grad, value=1 - beta2)

def update_param(param, step, exp_avg, exp_avg_sq, beta1, beta2, lr, eps):
    bias_correction1 = 1 - torch.pow(beta1, step)
    bias_correction2 = (1 - torch.pow(beta2, step)).sqrt()
    step_size = (lr / bias_correction1).neg()
    denom = (exp_avg_sq.sqrt() / (bias_correction2 * step_size)).add(eps / step_size)
    return torch.add(param, torch.div(exp_avg, denom))

# Our full Adam implementation
def foreach_map_adam(
    steps,
    params,
    exp_avgs,
    exp_avg_sqs,
    weight_decay=0,
    beta1=0.9,
    beta2=0.999,
    lr=1e-3,
    eps=1e-8,
):
    with torch.no_grad():
        grads = [param.grad for param in params]
        # update step
        updated_steps = foreach_map(lambda x: x + 1, steps)
        torch._foreach_copy_(steps, updated_steps)

        if weight_decay != 0:
            foreach_map(torch.add, (grads,), alpha=weight_decay)

        # Higher-order operators (HOPs) cannot have multiple outputs at the moment
        # need to call foreach_map once for each output
        exp_avgs_updated = foreach_map(torch.lerp, exp_avgs, grads, 1 - beta1)
        exp_avgs_sq_updated = foreach_map(update_exp_avg_sq, exp_avg_sqs, grads, beta2)
        params_updated = foreach_map(
            update_param,
            params,
            steps,
            exp_avgs_updated,
            exp_avgs_sq_updated,
            beta1,
            beta2,
            lr,
            eps,
        )
        # Higher-order operators (HOPs) don't support input mutation today
        # so manually  update the states in-place
        torch._foreach_copy_(exp_avgs, exp_avgs_updated)
        torch._foreach_copy_(exp_avg_sqs, exp_avgs_sq_updated)
        torch._foreach_copy_(params, params_updated)
    return

Setting up and running the compiled kernel

In this section, we’ll run our Adam optimizer and compare the results

Note

torch.compile is only supported on CUDA devices that have a compute capability of 7.0 or higher.

opt_eager = torch.optim.Adam(model.parameters(), lr=torch.tensor(0.01))
opt_eager_copy = torch.optim.Adam(model_copy.parameters(), lr=torch.tensor(0.01))

# warm up the optimizer state dict
opt_eager.step()
opt_eager_copy.step()

inputs = get_inputs(opt_eager_copy)
compiled_adam = torch.compile(foreach_map_adam)

# optionally view the output code
torch._logging.set_logs(output_code=True)

# Warmup runs to compile the function
for _ in range(5):
    opt_eager.step()
    compiled_adam(*inputs)

for eager_p, compile_p in zip(opt_eager.param_groups[0]["params"], opt_eager_copy.param_groups[0]["params"]):
    torch.allclose(eager_p, compile_p)

# Benchmark performance

 # Let's define a helpful benchmarking function:
import torch.utils.benchmark as benchmark

def benchmark_torch_function_in_microseconds(f, *args, **kwargs):
    t0 = benchmark.Timer(
        stmt="f(*args, **kwargs)", globals={"args": args, "kwargs": kwargs, "f": f}
    )
    return t0.blocked_autorange().mean * 1e6

eager_runtime = benchmark_torch_function_in_microseconds(opt_eager.step)
compiled_runtime = benchmark_torch_function_in_microseconds(lambda: compiled_adam(*inputs))

assert eager_runtime > compiled_runtime

print(f"eager runtime: {eager_runtime}us")
print(f"compiled runtime: {compiled_runtime}us")

V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] Output code:
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] # AOT ID: ['0_inference']
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from ctypes import c_void_p, c_long, c_int
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] import torch
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] import math
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] import random
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] import os
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] import tempfile
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from math import inf, nan
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from cmath import nanj
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from torch._inductor.hooks import run_intermediate_hooks
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from torch._inductor.utils import maybe_profile
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from torch._inductor.codegen.memory_planning import _align as align
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from torch import device, empty_strided
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from torch._inductor.async_compile import AsyncCompile
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from torch._inductor.select_algorithm import extern_kernels
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from torch._C._dynamo.guards import copy_if_misaligned
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from torch._C import _cuda_getCurrentRawStream as get_raw_stream
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] import triton
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] import triton.language as tl
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from torch._inductor.runtime.triton_heuristics import start_graph, end_graph
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from torch._C import _cuda_getCurrentRawStream as get_raw_stream
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] aten = torch.ops.aten
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] inductor_ops = torch.ops.inductor
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] _quantized = torch.ops._quantized
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] assert_size_stride = torch._C._dynamo.guards.assert_size_stride
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] assert_size_stride_grouped = torch._C._dynamo.guards.assert_size_stride_grouped
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] assert_alignment = torch._C._dynamo.guards.assert_alignment
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] empty_strided_cpu = torch._C._dynamo.guards._empty_strided_cpu
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] empty_strided_cpu_pinned = torch._C._dynamo.guards._empty_strided_cpu_pinned
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] empty_strided_cuda = torch._C._dynamo.guards._empty_strided_cuda
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] empty_strided_xpu = torch._C._dynamo.guards._empty_strided_xpu
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] empty_strided_mtia = torch._C._dynamo.guards._empty_strided_mtia
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] reinterpret_tensor = torch._C._dynamo.guards._reinterpret_tensor
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] alloc_from_pool = torch.ops.inductor._alloc_from_pool
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] async_compile = AsyncCompile()
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] empty_strided_p2p = torch._C._distributed_c10d._SymmetricMemory.empty_strided_p2p
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] cpp_fused__foreach_copy_0 = async_compile.cpp_pybinding(['const float*', 'const float*', 'const float*', 'const float*', 'const float*', 'const float*', 'const float*', 'const float*', 'const float*', 'const float*', 'float*', 'float*', 'float*', 'float*', 'float*', 'float*', 'float*', 'float*', 'float*', 'float*'], r'''
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] #include <torch/csrc/inductor/cpp_prefix.h>
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] extern "C"  void  kernel(const float* in_ptr0,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        const float* in_ptr1,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        const float* in_ptr2,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        const float* in_ptr3,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        const float* in_ptr4,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        const float* in_ptr5,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        const float* in_ptr6,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        const float* in_ptr7,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        const float* in_ptr8,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        const float* in_ptr9,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        float* out_ptr0,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        float* out_ptr1,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        float* out_ptr2,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        float* out_ptr3,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        float* out_ptr4,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        float* out_ptr5,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        float* out_ptr6,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        float* out_ptr7,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        float* out_ptr8,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                        float* out_ptr9)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     std::atomic<int> inductor_cpu_integer_div_error{0};
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     inductor_cpu_integer_div_error_flag = &inductor_cpu_integer_div_error;
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp0 = in_ptr0[static_cast<int64_t>(0L)];
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 out_ptr0[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp0 = in_ptr1[static_cast<int64_t>(0L)];
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 out_ptr1[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp0 = in_ptr2[static_cast<int64_t>(0L)];
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 out_ptr2[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp0 = in_ptr3[static_cast<int64_t>(0L)];
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 out_ptr3[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp0 = in_ptr4[static_cast<int64_t>(0L)];
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 out_ptr4[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp0 = in_ptr5[static_cast<int64_t>(0L)];
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 out_ptr5[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp0 = in_ptr6[static_cast<int64_t>(0L)];
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 out_ptr6[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp0 = in_ptr7[static_cast<int64_t>(0L)];
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 out_ptr7[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp0 = in_ptr8[static_cast<int64_t>(0L)];
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 out_ptr8[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             {
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp0 = in_ptr9[static_cast<int64_t>(0L)];
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]                 out_ptr9[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     inductor_cpu_integer_div_error_flag = nullptr;
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     inductor_cpu_throw_if_integer_div_error(inductor_cpu_integer_div_error);
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] }
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] ''')
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] # kernel path: /tmp/torchinductor_ci-user/dq/cdqpmdn3fm5zhlh3xrxfrnkppgk4c2gpvcboqp7u4dpc3mwaigrd.py
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] # Unsorted Source Nodes: [], Original ATen: []
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] # Source node to ATen node mapping:
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] triton_for_fused_1 = async_compile.triton('triton_for_fused_1', '''
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] import triton
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] import triton.language as tl
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from torch._inductor.runtime import triton_helpers, triton_heuristics
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from torch._inductor.runtime.triton_helpers import libdevice, math as tl_math
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] from torch._inductor.runtime.hints import AutotuneHint, ReductionHint, TileHint, DeviceProperties
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] @triton_heuristics.foreach(
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     filename=__file__,
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     triton_meta={'signature': {'in_ptr0': '*fp32', 'in_ptr1': '*fp32', 'in_ptr2': '*fp32', 'in_ptr3': '*fp32', 'in_ptr4': 'fp32', 'in_ptr5': '*fp32', 'in_ptr6': '*fp32', 'in_ptr7': '*fp32', 'in_ptr8': '*fp32', 'in_ptr9': 'fp32', 'in_ptr10': '*fp32', 'in_ptr11': '*fp32', 'in_ptr12': '*fp32', 'in_ptr13': '*fp32', 'in_ptr14': 'fp32', 'in_ptr15': '*fp32', 'in_ptr16': '*fp32', 'in_ptr17': '*fp32', 'in_ptr18': '*fp32', 'in_ptr19': 'fp32', 'in_ptr20': '*fp32', 'in_ptr21': '*fp32', 'in_ptr22': '*fp32', 'in_ptr23': '*fp32', 'in_ptr24': 'fp32', 'in_ptr25': '*fp32', 'in_ptr26': '*fp32', 'in_ptr27': '*fp32', 'in_ptr28': '*fp32', 'in_ptr29': 'fp32', 'in_ptr30': '*fp32', 'in_ptr31': '*fp32', 'in_ptr32': '*fp32', 'in_ptr33': '*fp32', 'in_ptr34': 'fp32', 'in_ptr35': '*fp32', 'in_ptr36': '*fp32', 'in_ptr37': '*fp32', 'in_ptr38': '*fp32', 'in_ptr39': 'fp32', 'in_ptr40': '*fp32', 'in_ptr41': '*fp32', 'in_ptr42': '*fp32', 'in_ptr43': '*fp32', 'in_ptr44': 'fp32', 'in_ptr45': '*fp32', 'in_ptr46': '*fp32', 'in_ptr47': '*fp32', 'in_ptr48': '*fp32', 'in_ptr49': 'fp32', 'out_ptr3': '*fp32', 'out_ptr4': '*fp32', 'out_ptr5': '*fp32', 'out_ptr9': '*fp32', 'out_ptr10': '*fp32', 'out_ptr11': '*fp32', 'out_ptr15': '*fp32', 'out_ptr16': '*fp32', 'out_ptr17': '*fp32', 'out_ptr21': '*fp32', 'out_ptr22': '*fp32', 'out_ptr23': '*fp32', 'out_ptr27': '*fp32', 'out_ptr28': '*fp32', 'out_ptr29': '*fp32', 'out_ptr33': '*fp32', 'out_ptr34': '*fp32', 'out_ptr35': '*fp32', 'out_ptr39': '*fp32', 'out_ptr40': '*fp32', 'out_ptr41': '*fp32', 'out_ptr45': '*fp32', 'out_ptr46': '*fp32', 'out_ptr47': '*fp32', 'out_ptr51': '*fp32', 'out_ptr52': '*fp32', 'out_ptr53': '*fp32', 'out_ptr57': '*fp32', 'out_ptr58': '*fp32', 'out_ptr59': '*fp32'}, 'device': DeviceProperties(type='cuda', index=0, multi_processor_count=80, cc=86, major=8, regs_per_multiprocessor=65536, max_threads_per_multi_processor=1536, max_threads_per_block=1024, warp_size=32), 'constants': {}, 'enable_fp_fusion': True, 'launch_pdl': False, 'disable_ftz': False, 'configs': [{(0,): [['tt.divisibility', 16]], (1,): [['tt.divisibility', 16]], (2,): [['tt.divisibility', 16]], (3,): [['tt.divisibility', 16]], (5,): [['tt.divisibility', 16]], (6,): [['tt.divisibility', 16]], (7,): [['tt.divisibility', 16]], (8,): [['tt.divisibility', 16]], (10,): [['tt.divisibility', 16]], (11,): [['tt.divisibility', 16]], (12,): [['tt.divisibility', 16]], (13,): [['tt.divisibility', 16]], (15,): [['tt.divisibility', 16]], (16,): [['tt.divisibility', 16]], (17,): [['tt.divisibility', 16]], (18,): [['tt.divisibility', 16]], (20,): [['tt.divisibility', 16]], (21,): [['tt.divisibility', 16]], (22,): [['tt.divisibility', 16]], (23,): [['tt.divisibility', 16]], (25,): [['tt.divisibility', 16]], (26,): [['tt.divisibility', 16]], (27,): [['tt.divisibility', 16]], (28,): [['tt.divisibility', 16]], (30,): [['tt.divisibility', 16]], (31,): [['tt.divisibility', 16]], (32,): [['tt.divisibility', 16]], (33,): [['tt.divisibility', 16]], (35,): [['tt.divisibility', 16]], (36,): [['tt.divisibility', 16]], (37,): [['tt.divisibility', 16]], (38,): [['tt.divisibility', 16]], (40,): [['tt.divisibility', 16]], (41,): [['tt.divisibility', 16]], (42,): [['tt.divisibility', 16]], (43,): [['tt.divisibility', 16]], (45,): [['tt.divisibility', 16]], (46,): [['tt.divisibility', 16]], (47,): [['tt.divisibility', 16]], (48,): [['tt.divisibility', 16]], (50,): [['tt.divisibility', 16]], (51,): [['tt.divisibility', 16]], (52,): [['tt.divisibility', 16]], (53,): [['tt.divisibility', 16]], (54,): [['tt.divisibility', 16]], (55,): [['tt.divisibility', 16]], (56,): [['tt.divisibility', 16]], (57,): [['tt.divisibility', 16]], (58,): [['tt.divisibility', 16]], (59,): [['tt.divisibility', 16]], (60,): [['tt.divisibility', 16]], (61,): [['tt.divisibility', 16]], (62,): [['tt.divisibility', 16]], (63,): [['tt.divisibility', 16]], (64,): [['tt.divisibility', 16]], (65,): [['tt.divisibility', 16]], (66,): [['tt.divisibility', 16]], (67,): [['tt.divisibility', 16]], (68,): [['tt.divisibility', 16]], (69,): [['tt.divisibility', 16]], (70,): [['tt.divisibility', 16]], (71,): [['tt.divisibility', 16]], (72,): [['tt.divisibility', 16]], (73,): [['tt.divisibility', 16]], (74,): [['tt.divisibility', 16]], (75,): [['tt.divisibility', 16]], (76,): [['tt.divisibility', 16]], (77,): [['tt.divisibility', 16]], (78,): [['tt.divisibility', 16]], (79,): [['tt.divisibility', 16]]}]},
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     inductor_meta={'grid_type': 'SequentialComboKernelGrid', 'combo_grid_meta': {'num_kernels': 10, 'min_blocks': 0, 'autotune_grouping': True, 'block_arg_names': ('XBLOCK',), 'default_config': {'XBLOCK': 1024}, 'no_x_dim_0': False, 'xnumel_0': 1048576, 'no_x_dim_1': False, 'xnumel_1': 1048576, 'no_x_dim_2': False, 'xnumel_2': 1048576, 'no_x_dim_3': False, 'xnumel_3': 1048576, 'no_x_dim_4': False, 'xnumel_4': 1048576, 'no_x_dim_5': False, 'xnumel_5': 1048576, 'no_x_dim_6': False, 'xnumel_6': 1048576, 'no_x_dim_7': False, 'xnumel_7': 1048576, 'no_x_dim_8': False, 'xnumel_8': 1048576, 'no_x_dim_9': False, 'xnumel_9': 1048576}, 'kernel_name': 'triton_for_fused_1', 'mutated_arg_names': ['in_ptr0', 'in_ptr10', 'in_ptr12', 'in_ptr13', 'in_ptr15', 'in_ptr17', 'in_ptr18', 'in_ptr2', 'in_ptr20', 'in_ptr22', 'in_ptr23', 'in_ptr25', 'in_ptr27', 'in_ptr28', 'in_ptr3', 'in_ptr30', 'in_ptr32', 'in_ptr33', 'in_ptr35', 'in_ptr37', 'in_ptr38', 'in_ptr40', 'in_ptr42', 'in_ptr43', 'in_ptr45', 'in_ptr47', 'in_ptr48', 'in_ptr5', 'in_ptr7', 'in_ptr8', 'out_ptr10', 'out_ptr11', 'out_ptr15', 'out_ptr16', 'out_ptr17', 'out_ptr21', 'out_ptr22', 'out_ptr23', 'out_ptr27', 'out_ptr28', 'out_ptr29', 'out_ptr3', 'out_ptr33', 'out_ptr34', 'out_ptr35', 'out_ptr39', 'out_ptr4', 'out_ptr40', 'out_ptr41', 'out_ptr45', 'out_ptr46', 'out_ptr47', 'out_ptr5', 'out_ptr51', 'out_ptr52', 'out_ptr53', 'out_ptr57', 'out_ptr58', 'out_ptr59', 'out_ptr9'], 'optimize_mem': True, 'backend_hash': '2D3FC36C68F91221B7F13C045BF32EE89FDA5FED882FDF474D9209EFF5F6DB93', 'assert_indirect_indexing': True, 'autotune_local_cache': True, 'autotune_pointwise': True, 'autotune_remote_cache': None, 'force_disable_caches': False, 'dynamic_scale_rblock': True, 'incremental_autotune': False, 'max_autotune': False, 'max_autotune_pointwise': False, 'min_split_scan_rblock': 256, 'spill_threshold': 16, 'store_cubin': False, 'deterministic': False, 'batch_invariant': False, 'force_filter_reduction_configs': False, 'mix_order_reduction_allow_multi_stages': True, 'dynamic_disable_pipelining': True, 'are_deterministic_algorithms_enabled': False},
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] )
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] @triton.jit
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] def triton_for_fused_1(in_ptr0, in_ptr1, in_ptr2, in_ptr3, in_ptr4, in_ptr5, in_ptr6, in_ptr7, in_ptr8, in_ptr9, in_ptr10, in_ptr11, in_ptr12, in_ptr13, in_ptr14, in_ptr15, in_ptr16, in_ptr17, in_ptr18, in_ptr19, in_ptr20, in_ptr21, in_ptr22, in_ptr23, in_ptr24, in_ptr25, in_ptr26, in_ptr27, in_ptr28, in_ptr29, in_ptr30, in_ptr31, in_ptr32, in_ptr33, in_ptr34, in_ptr35, in_ptr36, in_ptr37, in_ptr38, in_ptr39, in_ptr40, in_ptr41, in_ptr42, in_ptr43, in_ptr44, in_ptr45, in_ptr46, in_ptr47, in_ptr48, in_ptr49, out_ptr3, out_ptr4, out_ptr5, out_ptr9, out_ptr10, out_ptr11, out_ptr15, out_ptr16, out_ptr17, out_ptr21, out_ptr22, out_ptr23, out_ptr27, out_ptr28, out_ptr29, out_ptr33, out_ptr34, out_ptr35, out_ptr39, out_ptr40, out_ptr41, out_ptr45, out_ptr46, out_ptr47, out_ptr51, out_ptr52, out_ptr53, out_ptr57, out_ptr58, out_ptr59):
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     pid = tl.program_id(0)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     XBLOCK: tl.constexpr = 1024
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     num_xblocks_0 = tl.cdiv(1048576, XBLOCK)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     num_xblocks_1 = num_xblocks_0 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     num_xblocks_2 = num_xblocks_1 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     num_xblocks_3 = num_xblocks_2 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     num_xblocks_4 = num_xblocks_3 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     num_xblocks_5 = num_xblocks_4 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     num_xblocks_6 = num_xblocks_5 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     num_xblocks_7 = num_xblocks_6 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     num_xblocks_8 = num_xblocks_7 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     num_xblocks_9 = num_xblocks_8 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     if pid < num_xblocks_0:
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         pid_offset = pid
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         xnumel = 1048576
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         r0_numel = 1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg0 = in_ptr0
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg1 = in_ptr1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg2 = in_ptr2
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg3 = in_ptr3
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg4 = in_ptr4
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg5 = out_ptr3
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg6 = out_ptr4
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg7 = out_ptr5
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     elif pid < num_xblocks_1:
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         pid_offset = pid - num_xblocks_0
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         xnumel = 1048576
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         r0_numel = 1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg0 = in_ptr5
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg1 = in_ptr6
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg2 = in_ptr7
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg3 = in_ptr8
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg4 = in_ptr9
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg5 = out_ptr9
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg6 = out_ptr10
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg7 = out_ptr11
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     elif pid < num_xblocks_2:
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         pid_offset = pid - num_xblocks_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         xnumel = 1048576
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         r0_numel = 1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg0 = in_ptr10
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg1 = in_ptr11
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg2 = in_ptr12
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg3 = in_ptr13
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg4 = in_ptr14
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg5 = out_ptr15
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg6 = out_ptr16
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg7 = out_ptr17
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     elif pid < num_xblocks_3:
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         pid_offset = pid - num_xblocks_2
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         xnumel = 1048576
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         r0_numel = 1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg0 = in_ptr15
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg1 = in_ptr16
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg2 = in_ptr17
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg3 = in_ptr18
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg4 = in_ptr19
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg5 = out_ptr21
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg6 = out_ptr22
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg7 = out_ptr23
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     elif pid < num_xblocks_4:
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         pid_offset = pid - num_xblocks_3
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         xnumel = 1048576
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         r0_numel = 1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg0 = in_ptr20
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg1 = in_ptr21
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg2 = in_ptr22
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg3 = in_ptr23
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg4 = in_ptr24
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg5 = out_ptr27
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg6 = out_ptr28
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg7 = out_ptr29
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     elif pid < num_xblocks_5:
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         pid_offset = pid - num_xblocks_4
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         xnumel = 1048576
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         r0_numel = 1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg0 = in_ptr25
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg1 = in_ptr26
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg2 = in_ptr27
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg3 = in_ptr28
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg4 = in_ptr29
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg5 = out_ptr33
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg6 = out_ptr34
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg7 = out_ptr35
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     elif pid < num_xblocks_6:
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         pid_offset = pid - num_xblocks_5
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         xnumel = 1048576
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         r0_numel = 1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg0 = in_ptr30
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg1 = in_ptr31
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg2 = in_ptr32
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg3 = in_ptr33
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg4 = in_ptr34
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg5 = out_ptr39
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg6 = out_ptr40
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg7 = out_ptr41
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     elif pid < num_xblocks_7:
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         pid_offset = pid - num_xblocks_6
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         xnumel = 1048576
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         r0_numel = 1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg0 = in_ptr35
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg1 = in_ptr36
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg2 = in_ptr37
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg3 = in_ptr38
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg4 = in_ptr39
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg5 = out_ptr45
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg6 = out_ptr46
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg7 = out_ptr47
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     elif pid < num_xblocks_8:
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         pid_offset = pid - num_xblocks_7
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         xnumel = 1048576
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         r0_numel = 1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg0 = in_ptr40
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg1 = in_ptr41
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg2 = in_ptr42
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg3 = in_ptr43
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg4 = in_ptr44
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg5 = out_ptr51
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg6 = out_ptr52
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg7 = out_ptr53
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     elif pid < num_xblocks_9:
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         pid_offset = pid - num_xblocks_8
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         xnumel = 1048576
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         r0_numel = 1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg0 = in_ptr45
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg1 = in_ptr46
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg2 = in_ptr47
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg3 = in_ptr48
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg4 = in_ptr49
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg5 = out_ptr57
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg6 = out_ptr58
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg7 = out_ptr59
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     else:
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         pid_offset = 0
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         xnumel = 0
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         r0_numel = 0
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg0 = in_ptr0
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg1 = in_ptr1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg2 = in_ptr2
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg3 = in_ptr3
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg4 = in_ptr4
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg5 = out_ptr3
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg6 = out_ptr4
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         foreach_arg7 = out_ptr5
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     xoffset = pid_offset * XBLOCK
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     xindex = xoffset + tl.arange(0, XBLOCK)[:]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     xmask = tl.full([XBLOCK], True, tl.int1)[:]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     x0 = xindex
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp0 = tl.load(foreach_arg0 + (x0), None)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp1 = tl.load(foreach_arg1 + (x0), None)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp5 = tl.load(foreach_arg2 + (x0), None)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp11 = tl.load(foreach_arg3 + (x0), None)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp13 = foreach_arg4
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp2 = tmp1 - tmp0
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp3 = tl.full([1], 0.09999999999999998, tl.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp4 = tl.fma(tmp2, tmp3, tmp0)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp6 = tl.full([1], 0.999, tl.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp7 = tmp5 * tmp6
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp8 = libdevice.mul_rn(tmp1, tmp1)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp9 = tl.full([1], 0.0010000000000000009, tl.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp10 = tl.fma(tmp9, tmp8, tmp7)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp12 = tl.sqrt_rn(tmp10)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp14 = libdevice.pow(tmp6, tmp13)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp15 = tl.full([1], 1.0, tl.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp16 = tmp15 - tmp14
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp17 = tl.sqrt_rn(tmp16)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp18 = tl.full([1], 0.9, tl.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp19 = libdevice.pow(tmp18, tmp13)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp20 = tmp15 - tmp19
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp21 = (tmp15 / tmp20)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp22 = tl.full([1], 0.001, tl.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp23 = tmp21 * tmp22
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp24 = -tmp23
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp25 = tmp17 * tmp24
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp26 = (tmp12 / tmp25)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp27 = (tmp15 / tmp24)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp28 = tl.full([1], 1e-08, tl.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp29 = tmp27 * tmp28
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp30 = tmp26 + tmp29
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp31 = (tmp4 / tmp30)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tmp32 = tmp11 + tmp31
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tl.store(foreach_arg5 + (x0), tmp4, None)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tl.store(foreach_arg6 + (x0), tmp10, None)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     tl.store(foreach_arg7 + (x0), tmp32, None)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] ''', device_str='cuda')
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] async_compile.wait(globals())
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] del async_compile
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] class Runner:
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     def __init__(self, partitions):
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         self.partitions = partitions
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     def recursively_apply_fns(self, fns):
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         new_callables = []
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         for fn, c in zip(fns, self.partitions):
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             new_callables.append(fn(c))
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         self.partitions = new_callables
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     def call(self, args):
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         arg0_1, arg1_1, arg2_1, arg3_1, arg4_1, arg5_1, arg6_1, arg7_1, arg8_1, arg9_1, arg10_1, arg11_1, arg12_1, arg13_1, arg14_1, arg15_1, arg16_1, arg17_1, arg18_1, arg19_1, arg20_1, arg21_1, arg22_1, arg23_1, arg24_1, arg25_1, arg26_1, arg27_1, arg28_1, arg29_1, arg30_1, arg31_1, arg32_1, arg33_1, arg34_1, arg35_1, arg36_1, arg37_1, arg38_1, arg39_1, arg40_1, arg41_1, arg42_1, arg43_1, arg44_1, arg45_1, arg46_1, arg47_1, arg48_1, arg49_1 = args
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         args.clear()
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         assert_size_stride(arg10_1, (), (), 'input')
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         assert_size_stride(arg11_1, (), (), 'input')
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         assert_size_stride(arg12_1, (), (), 'input')
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         assert_size_stride(arg13_1, (), (), 'input')
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         assert_size_stride(arg14_1, (), (), 'input')
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         assert_size_stride(arg15_1, (), (), 'input')
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         assert_size_stride(arg16_1, (), (), 'input')
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         assert_size_stride(arg17_1, (), (), 'input')
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         assert_size_stride(arg18_1, (), (), 'input')
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         assert_size_stride(arg19_1, (), (), 'input')
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         assert_size_stride_grouped((arg20_1, arg30_1, arg40_1, arg0_1, arg21_1, arg31_1, arg41_1, arg1_1, arg22_1, arg32_1, arg42_1, arg2_1, arg23_1, arg33_1, arg43_1, arg3_1, arg24_1, arg34_1, arg44_1, arg4_1, arg25_1, arg35_1, arg45_1, arg5_1, arg26_1, arg36_1, arg46_1, arg6_1, arg27_1, arg37_1, arg47_1, arg7_1, arg28_1, arg38_1, arg48_1, arg8_1, arg29_1, arg39_1, arg49_1, arg9_1), ((1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024)), ((1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1)), 'input')
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         cpp_fused__foreach_copy_0(arg10_1, arg11_1, arg12_1, arg13_1, arg14_1, arg15_1, arg16_1, arg17_1, arg18_1, arg19_1, arg10_1, arg11_1, arg12_1, arg13_1, arg14_1, arg15_1, arg16_1, arg17_1, arg18_1, arg19_1)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         with torch.cuda._DeviceGuard(0):
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             torch.cuda.set_device(0)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             arg30_1 = copy_if_misaligned(arg30_1)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             arg31_1 = copy_if_misaligned(arg31_1)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             arg32_1 = copy_if_misaligned(arg32_1)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             arg33_1 = copy_if_misaligned(arg33_1)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             arg34_1 = copy_if_misaligned(arg34_1)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             arg35_1 = copy_if_misaligned(arg35_1)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             arg36_1 = copy_if_misaligned(arg36_1)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             arg37_1 = copy_if_misaligned(arg37_1)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             arg38_1 = copy_if_misaligned(arg38_1)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             arg39_1 = copy_if_misaligned(arg39_1)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             # Unsorted Source Nodes: [], Original ATen: []
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             raw_stream0 = get_raw_stream(0)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             triton_for_fused_1.run(arg20_1, arg30_1, arg40_1, arg0_1, arg10_1.item(), arg21_1, arg31_1, arg41_1, arg1_1, arg11_1.item(), arg22_1, arg32_1, arg42_1, arg2_1, arg12_1.item(), arg23_1, arg33_1, arg43_1, arg3_1, arg13_1.item(), arg24_1, arg34_1, arg44_1, arg4_1, arg14_1.item(), arg25_1, arg35_1, arg45_1, arg5_1, arg15_1.item(), arg26_1, arg36_1, arg46_1, arg6_1, arg16_1.item(), arg27_1, arg37_1, arg47_1, arg7_1, arg17_1.item(), arg28_1, arg38_1, arg48_1, arg8_1, arg18_1.item(), arg29_1, arg39_1, arg49_1, arg9_1, arg19_1.item(), arg20_1, arg40_1, arg0_1, arg21_1, arg41_1, arg1_1, arg22_1, arg42_1, arg2_1, arg23_1, arg43_1, arg3_1, arg24_1, arg44_1, arg4_1, arg25_1, arg45_1, arg5_1, arg26_1, arg46_1, arg6_1, arg27_1, arg47_1, arg7_1, arg28_1, arg48_1, arg8_1, arg29_1, arg49_1, arg9_1, stream=raw_stream0)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg0_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg10_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg11_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg12_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg13_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg14_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg15_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg16_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg17_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg18_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg19_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg1_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg20_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg21_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg22_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg23_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg24_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg25_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg26_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg27_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg28_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg29_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg2_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg30_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg31_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg32_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg33_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg34_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg35_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg36_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg37_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg38_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg39_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg3_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg40_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg41_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg42_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg43_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg44_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg45_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg46_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg47_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg48_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg49_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg4_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg5_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg6_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg7_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg8_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]             del arg9_1
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]         return ()
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] runner = Runner(partitions=[])
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] call = runner.call
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] recursively_apply_fns = runner.recursively_apply_fns
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] def get_args():
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     from torch._dynamo.testing import rand_strided
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg0_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg1_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg2_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg3_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg4_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg5_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg6_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg7_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg8_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg9_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg10_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg11_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg12_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg13_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg14_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg15_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg16_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg17_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg18_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg19_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg20_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg21_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg22_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg23_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg24_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg25_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg26_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg27_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg28_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg29_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg30_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg31_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg32_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg33_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg34_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg35_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg36_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg37_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg38_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg39_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg40_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg41_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg42_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg43_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg44_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg45_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg46_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg47_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg48_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     arg49_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     return [arg0_1, arg1_1, arg2_1, arg3_1, arg4_1, arg5_1, arg6_1, arg7_1, arg8_1, arg9_1, arg10_1, arg11_1, arg12_1, arg13_1, arg14_1, arg15_1, arg16_1, arg17_1, arg18_1, arg19_1, arg20_1, arg21_1, arg22_1, arg23_1, arg24_1, arg25_1, arg26_1, arg27_1, arg28_1, arg29_1, arg30_1, arg31_1, arg32_1, arg33_1, arg34_1, arg35_1, arg36_1, arg37_1, arg38_1, arg39_1, arg40_1, arg41_1, arg42_1, arg43_1, arg44_1, arg45_1, arg46_1, arg47_1, arg48_1, arg49_1]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] def benchmark_compiled_module(args, times=10, repeat=10):
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     from torch._inductor.utils import print_performance
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     fn = lambda: call(list(args))
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     return print_performance(fn, times=times, repeat=repeat, device='cuda')
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code] if __name__ == "__main__":
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     from torch._inductor.wrapper_benchmark import compiled_module_main
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     args = get_args()
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]     compiled_module_main('None', lambda times, repeat: benchmark_compiled_module(args, times=times, repeat=repeat))
V0903 01:21:40.734000 26236 torch/_inductor/graph.py:3114] [0/0] [__output_code]
V0903 01:21:40.767000 26236 torch/_inductor/graph.py:3125] [0/0] [__output_code] Output code written to: /tmp/torchinductor_ci-user/je/cjec6tipekfvsec6jha5phyylzxe757yzas2hx4kb3nkjpii36ii.py
I0903 01:21:41.527000 26236 torch/_inductor/graph.py:3085] [0/0] [__output_code] Output code written to: /tmp/torchinductor_ci-user/je/cjec6tipekfvsec6jha5phyylzxe757yzas2hx4kb3nkjpii36ii.py
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] Output code:
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] # AOT ID: ['1_inference']
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from ctypes import c_void_p, c_long, c_int
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] import torch
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] import math
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] import random
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] import os
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] import tempfile
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from math import inf, nan
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from cmath import nanj
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from torch._inductor.hooks import run_intermediate_hooks
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from torch._inductor.utils import maybe_profile
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from torch._inductor.codegen.memory_planning import _align as align
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from torch import device, empty_strided
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from torch._inductor.async_compile import AsyncCompile
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from torch._inductor.select_algorithm import extern_kernels
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from torch._C._dynamo.guards import copy_if_misaligned
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from torch._C import _cuda_getCurrentRawStream as get_raw_stream
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] import triton
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] import triton.language as tl
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from torch._inductor.runtime.triton_heuristics import start_graph, end_graph
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from torch._C import _cuda_getCurrentRawStream as get_raw_stream
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] aten = torch.ops.aten
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] inductor_ops = torch.ops.inductor
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] _quantized = torch.ops._quantized
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] assert_size_stride = torch._C._dynamo.guards.assert_size_stride
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] assert_size_stride_grouped = torch._C._dynamo.guards.assert_size_stride_grouped
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] assert_alignment = torch._C._dynamo.guards.assert_alignment
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] empty_strided_cpu = torch._C._dynamo.guards._empty_strided_cpu
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] empty_strided_cpu_pinned = torch._C._dynamo.guards._empty_strided_cpu_pinned
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] empty_strided_cuda = torch._C._dynamo.guards._empty_strided_cuda
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] empty_strided_xpu = torch._C._dynamo.guards._empty_strided_xpu
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] empty_strided_mtia = torch._C._dynamo.guards._empty_strided_mtia
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] reinterpret_tensor = torch._C._dynamo.guards._reinterpret_tensor
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] alloc_from_pool = torch.ops.inductor._alloc_from_pool
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] async_compile = AsyncCompile()
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] empty_strided_p2p = torch._C._distributed_c10d._SymmetricMemory.empty_strided_p2p
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] cpp_fused__foreach_copy_0 = async_compile.cpp_pybinding(['const float*', 'const float*', 'const float*', 'const float*', 'const float*', 'const float*', 'const float*', 'const float*', 'const float*', 'const float*', 'float*', 'float*', 'float*', 'float*', 'float*', 'float*', 'float*', 'float*', 'float*', 'float*'], r'''
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] #include <torch/csrc/inductor/cpp_prefix.h>
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] extern "C"  void  kernel(const float* in_ptr0,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        const float* in_ptr1,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        const float* in_ptr2,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        const float* in_ptr3,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        const float* in_ptr4,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        const float* in_ptr5,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        const float* in_ptr6,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        const float* in_ptr7,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        const float* in_ptr8,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        const float* in_ptr9,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        float* out_ptr0,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        float* out_ptr1,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        float* out_ptr2,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        float* out_ptr3,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        float* out_ptr4,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        float* out_ptr5,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        float* out_ptr6,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        float* out_ptr7,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        float* out_ptr8,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                        float* out_ptr9)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     std::atomic<int> inductor_cpu_integer_div_error{0};
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     inductor_cpu_integer_div_error_flag = &inductor_cpu_integer_div_error;
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp0 = in_ptr0[static_cast<int64_t>(0L)];
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 out_ptr0[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp0 = in_ptr1[static_cast<int64_t>(0L)];
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 out_ptr1[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp0 = in_ptr2[static_cast<int64_t>(0L)];
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 out_ptr2[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp0 = in_ptr3[static_cast<int64_t>(0L)];
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 out_ptr3[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp0 = in_ptr4[static_cast<int64_t>(0L)];
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 out_ptr4[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp0 = in_ptr5[static_cast<int64_t>(0L)];
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 out_ptr5[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp0 = in_ptr6[static_cast<int64_t>(0L)];
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 out_ptr6[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp0 = in_ptr7[static_cast<int64_t>(0L)];
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 out_ptr7[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp0 = in_ptr8[static_cast<int64_t>(0L)];
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 out_ptr8[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             {
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp0 = in_ptr9[static_cast<int64_t>(0L)];
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp1 = static_cast<float>(1.0);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 auto tmp2 = float(tmp0 + tmp1);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]                 out_ptr9[static_cast<int64_t>(0L)] = tmp2;
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     inductor_cpu_integer_div_error_flag = nullptr;
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     inductor_cpu_throw_if_integer_div_error(inductor_cpu_integer_div_error);
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] }
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] ''')
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] # kernel path: /tmp/torchinductor_ci-user/dq/cdqpmdn3fm5zhlh3xrxfrnkppgk4c2gpvcboqp7u4dpc3mwaigrd.py
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] # Unsorted Source Nodes: [], Original ATen: []
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] # Source node to ATen node mapping:
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] triton_for_fused_1 = async_compile.triton('triton_for_fused_1', '''
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] import triton
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] import triton.language as tl
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from torch._inductor.runtime import triton_helpers, triton_heuristics
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from torch._inductor.runtime.triton_helpers import libdevice, math as tl_math
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] from torch._inductor.runtime.hints import AutotuneHint, ReductionHint, TileHint, DeviceProperties
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] @triton_heuristics.foreach(
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     filename=__file__,
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     triton_meta={'signature': {'in_ptr0': '*fp32', 'in_ptr1': '*fp32', 'in_ptr2': '*fp32', 'in_ptr3': '*fp32', 'in_ptr4': 'fp32', 'in_ptr5': '*fp32', 'in_ptr6': '*fp32', 'in_ptr7': '*fp32', 'in_ptr8': '*fp32', 'in_ptr9': 'fp32', 'in_ptr10': '*fp32', 'in_ptr11': '*fp32', 'in_ptr12': '*fp32', 'in_ptr13': '*fp32', 'in_ptr14': 'fp32', 'in_ptr15': '*fp32', 'in_ptr16': '*fp32', 'in_ptr17': '*fp32', 'in_ptr18': '*fp32', 'in_ptr19': 'fp32', 'in_ptr20': '*fp32', 'in_ptr21': '*fp32', 'in_ptr22': '*fp32', 'in_ptr23': '*fp32', 'in_ptr24': 'fp32', 'in_ptr25': '*fp32', 'in_ptr26': '*fp32', 'in_ptr27': '*fp32', 'in_ptr28': '*fp32', 'in_ptr29': 'fp32', 'in_ptr30': '*fp32', 'in_ptr31': '*fp32', 'in_ptr32': '*fp32', 'in_ptr33': '*fp32', 'in_ptr34': 'fp32', 'in_ptr35': '*fp32', 'in_ptr36': '*fp32', 'in_ptr37': '*fp32', 'in_ptr38': '*fp32', 'in_ptr39': 'fp32', 'in_ptr40': '*fp32', 'in_ptr41': '*fp32', 'in_ptr42': '*fp32', 'in_ptr43': '*fp32', 'in_ptr44': 'fp32', 'in_ptr45': '*fp32', 'in_ptr46': '*fp32', 'in_ptr47': '*fp32', 'in_ptr48': '*fp32', 'in_ptr49': 'fp32', 'out_ptr3': '*fp32', 'out_ptr4': '*fp32', 'out_ptr5': '*fp32', 'out_ptr9': '*fp32', 'out_ptr10': '*fp32', 'out_ptr11': '*fp32', 'out_ptr15': '*fp32', 'out_ptr16': '*fp32', 'out_ptr17': '*fp32', 'out_ptr21': '*fp32', 'out_ptr22': '*fp32', 'out_ptr23': '*fp32', 'out_ptr27': '*fp32', 'out_ptr28': '*fp32', 'out_ptr29': '*fp32', 'out_ptr33': '*fp32', 'out_ptr34': '*fp32', 'out_ptr35': '*fp32', 'out_ptr39': '*fp32', 'out_ptr40': '*fp32', 'out_ptr41': '*fp32', 'out_ptr45': '*fp32', 'out_ptr46': '*fp32', 'out_ptr47': '*fp32', 'out_ptr51': '*fp32', 'out_ptr52': '*fp32', 'out_ptr53': '*fp32', 'out_ptr57': '*fp32', 'out_ptr58': '*fp32', 'out_ptr59': '*fp32'}, 'device': DeviceProperties(type='cuda', index=0, multi_processor_count=80, cc=86, major=8, regs_per_multiprocessor=65536, max_threads_per_multi_processor=1536, max_threads_per_block=1024, warp_size=32), 'constants': {}, 'enable_fp_fusion': True, 'launch_pdl': False, 'disable_ftz': False, 'configs': [{(0,): [['tt.divisibility', 16]], (1,): [['tt.divisibility', 16]], (2,): [['tt.divisibility', 16]], (3,): [['tt.divisibility', 16]], (5,): [['tt.divisibility', 16]], (6,): [['tt.divisibility', 16]], (7,): [['tt.divisibility', 16]], (8,): [['tt.divisibility', 16]], (10,): [['tt.divisibility', 16]], (11,): [['tt.divisibility', 16]], (12,): [['tt.divisibility', 16]], (13,): [['tt.divisibility', 16]], (15,): [['tt.divisibility', 16]], (16,): [['tt.divisibility', 16]], (17,): [['tt.divisibility', 16]], (18,): [['tt.divisibility', 16]], (20,): [['tt.divisibility', 16]], (21,): [['tt.divisibility', 16]], (22,): [['tt.divisibility', 16]], (23,): [['tt.divisibility', 16]], (25,): [['tt.divisibility', 16]], (26,): [['tt.divisibility', 16]], (27,): [['tt.divisibility', 16]], (28,): [['tt.divisibility', 16]], (30,): [['tt.divisibility', 16]], (31,): [['tt.divisibility', 16]], (32,): [['tt.divisibility', 16]], (33,): [['tt.divisibility', 16]], (35,): [['tt.divisibility', 16]], (36,): [['tt.divisibility', 16]], (37,): [['tt.divisibility', 16]], (38,): [['tt.divisibility', 16]], (40,): [['tt.divisibility', 16]], (41,): [['tt.divisibility', 16]], (42,): [['tt.divisibility', 16]], (43,): [['tt.divisibility', 16]], (45,): [['tt.divisibility', 16]], (46,): [['tt.divisibility', 16]], (47,): [['tt.divisibility', 16]], (48,): [['tt.divisibility', 16]], (50,): [['tt.divisibility', 16]], (51,): [['tt.divisibility', 16]], (52,): [['tt.divisibility', 16]], (53,): [['tt.divisibility', 16]], (54,): [['tt.divisibility', 16]], (55,): [['tt.divisibility', 16]], (56,): [['tt.divisibility', 16]], (57,): [['tt.divisibility', 16]], (58,): [['tt.divisibility', 16]], (59,): [['tt.divisibility', 16]], (60,): [['tt.divisibility', 16]], (61,): [['tt.divisibility', 16]], (62,): [['tt.divisibility', 16]], (63,): [['tt.divisibility', 16]], (64,): [['tt.divisibility', 16]], (65,): [['tt.divisibility', 16]], (66,): [['tt.divisibility', 16]], (67,): [['tt.divisibility', 16]], (68,): [['tt.divisibility', 16]], (69,): [['tt.divisibility', 16]], (70,): [['tt.divisibility', 16]], (71,): [['tt.divisibility', 16]], (72,): [['tt.divisibility', 16]], (73,): [['tt.divisibility', 16]], (74,): [['tt.divisibility', 16]], (75,): [['tt.divisibility', 16]], (76,): [['tt.divisibility', 16]], (77,): [['tt.divisibility', 16]], (78,): [['tt.divisibility', 16]], (79,): [['tt.divisibility', 16]]}]},
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     inductor_meta={'grid_type': 'SequentialComboKernelGrid', 'combo_grid_meta': {'num_kernels': 10, 'min_blocks': 0, 'autotune_grouping': True, 'block_arg_names': ('XBLOCK',), 'default_config': {'XBLOCK': 1024}, 'no_x_dim_0': False, 'xnumel_0': 1048576, 'no_x_dim_1': False, 'xnumel_1': 1048576, 'no_x_dim_2': False, 'xnumel_2': 1048576, 'no_x_dim_3': False, 'xnumel_3': 1048576, 'no_x_dim_4': False, 'xnumel_4': 1048576, 'no_x_dim_5': False, 'xnumel_5': 1048576, 'no_x_dim_6': False, 'xnumel_6': 1048576, 'no_x_dim_7': False, 'xnumel_7': 1048576, 'no_x_dim_8': False, 'xnumel_8': 1048576, 'no_x_dim_9': False, 'xnumel_9': 1048576}, 'kernel_name': 'triton_for_fused_1', 'mutated_arg_names': ['in_ptr0', 'in_ptr10', 'in_ptr12', 'in_ptr13', 'in_ptr15', 'in_ptr17', 'in_ptr18', 'in_ptr2', 'in_ptr20', 'in_ptr22', 'in_ptr23', 'in_ptr25', 'in_ptr27', 'in_ptr28', 'in_ptr3', 'in_ptr30', 'in_ptr32', 'in_ptr33', 'in_ptr35', 'in_ptr37', 'in_ptr38', 'in_ptr40', 'in_ptr42', 'in_ptr43', 'in_ptr45', 'in_ptr47', 'in_ptr48', 'in_ptr5', 'in_ptr7', 'in_ptr8', 'out_ptr10', 'out_ptr11', 'out_ptr15', 'out_ptr16', 'out_ptr17', 'out_ptr21', 'out_ptr22', 'out_ptr23', 'out_ptr27', 'out_ptr28', 'out_ptr29', 'out_ptr3', 'out_ptr33', 'out_ptr34', 'out_ptr35', 'out_ptr39', 'out_ptr4', 'out_ptr40', 'out_ptr41', 'out_ptr45', 'out_ptr46', 'out_ptr47', 'out_ptr5', 'out_ptr51', 'out_ptr52', 'out_ptr53', 'out_ptr57', 'out_ptr58', 'out_ptr59', 'out_ptr9'], 'optimize_mem': True, 'backend_hash': '2D3FC36C68F91221B7F13C045BF32EE89FDA5FED882FDF474D9209EFF5F6DB93', 'assert_indirect_indexing': True, 'autotune_local_cache': True, 'autotune_pointwise': True, 'autotune_remote_cache': None, 'force_disable_caches': False, 'dynamic_scale_rblock': True, 'incremental_autotune': False, 'max_autotune': False, 'max_autotune_pointwise': False, 'min_split_scan_rblock': 256, 'spill_threshold': 16, 'store_cubin': False, 'deterministic': False, 'batch_invariant': False, 'force_filter_reduction_configs': False, 'mix_order_reduction_allow_multi_stages': True, 'dynamic_disable_pipelining': True, 'are_deterministic_algorithms_enabled': False},
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] )
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] @triton.jit
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] def triton_for_fused_1(in_ptr0, in_ptr1, in_ptr2, in_ptr3, in_ptr4, in_ptr5, in_ptr6, in_ptr7, in_ptr8, in_ptr9, in_ptr10, in_ptr11, in_ptr12, in_ptr13, in_ptr14, in_ptr15, in_ptr16, in_ptr17, in_ptr18, in_ptr19, in_ptr20, in_ptr21, in_ptr22, in_ptr23, in_ptr24, in_ptr25, in_ptr26, in_ptr27, in_ptr28, in_ptr29, in_ptr30, in_ptr31, in_ptr32, in_ptr33, in_ptr34, in_ptr35, in_ptr36, in_ptr37, in_ptr38, in_ptr39, in_ptr40, in_ptr41, in_ptr42, in_ptr43, in_ptr44, in_ptr45, in_ptr46, in_ptr47, in_ptr48, in_ptr49, out_ptr3, out_ptr4, out_ptr5, out_ptr9, out_ptr10, out_ptr11, out_ptr15, out_ptr16, out_ptr17, out_ptr21, out_ptr22, out_ptr23, out_ptr27, out_ptr28, out_ptr29, out_ptr33, out_ptr34, out_ptr35, out_ptr39, out_ptr40, out_ptr41, out_ptr45, out_ptr46, out_ptr47, out_ptr51, out_ptr52, out_ptr53, out_ptr57, out_ptr58, out_ptr59):
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     pid = tl.program_id(0)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     XBLOCK: tl.constexpr = 1024
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     num_xblocks_0 = tl.cdiv(1048576, XBLOCK)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     num_xblocks_1 = num_xblocks_0 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     num_xblocks_2 = num_xblocks_1 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     num_xblocks_3 = num_xblocks_2 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     num_xblocks_4 = num_xblocks_3 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     num_xblocks_5 = num_xblocks_4 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     num_xblocks_6 = num_xblocks_5 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     num_xblocks_7 = num_xblocks_6 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     num_xblocks_8 = num_xblocks_7 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     num_xblocks_9 = num_xblocks_8 + tl.cdiv(1048576, XBLOCK)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     if pid < num_xblocks_0:
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         pid_offset = pid
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         xnumel = 1048576
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         r0_numel = 1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg0 = in_ptr0
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg1 = in_ptr1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg2 = in_ptr2
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg3 = in_ptr3
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg4 = in_ptr4
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg5 = out_ptr3
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg6 = out_ptr4
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg7 = out_ptr5
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     elif pid < num_xblocks_1:
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         pid_offset = pid - num_xblocks_0
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         xnumel = 1048576
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         r0_numel = 1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg0 = in_ptr5
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg1 = in_ptr6
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg2 = in_ptr7
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg3 = in_ptr8
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg4 = in_ptr9
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg5 = out_ptr9
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg6 = out_ptr10
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg7 = out_ptr11
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     elif pid < num_xblocks_2:
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         pid_offset = pid - num_xblocks_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         xnumel = 1048576
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         r0_numel = 1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg0 = in_ptr10
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg1 = in_ptr11
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg2 = in_ptr12
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg3 = in_ptr13
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg4 = in_ptr14
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg5 = out_ptr15
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg6 = out_ptr16
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg7 = out_ptr17
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     elif pid < num_xblocks_3:
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         pid_offset = pid - num_xblocks_2
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         xnumel = 1048576
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         r0_numel = 1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg0 = in_ptr15
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg1 = in_ptr16
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg2 = in_ptr17
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg3 = in_ptr18
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg4 = in_ptr19
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg5 = out_ptr21
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg6 = out_ptr22
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg7 = out_ptr23
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     elif pid < num_xblocks_4:
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         pid_offset = pid - num_xblocks_3
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         xnumel = 1048576
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         r0_numel = 1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg0 = in_ptr20
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg1 = in_ptr21
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg2 = in_ptr22
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg3 = in_ptr23
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg4 = in_ptr24
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg5 = out_ptr27
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg6 = out_ptr28
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg7 = out_ptr29
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     elif pid < num_xblocks_5:
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         pid_offset = pid - num_xblocks_4
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         xnumel = 1048576
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         r0_numel = 1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg0 = in_ptr25
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg1 = in_ptr26
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg2 = in_ptr27
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg3 = in_ptr28
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg4 = in_ptr29
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg5 = out_ptr33
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg6 = out_ptr34
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg7 = out_ptr35
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     elif pid < num_xblocks_6:
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         pid_offset = pid - num_xblocks_5
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         xnumel = 1048576
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         r0_numel = 1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg0 = in_ptr30
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg1 = in_ptr31
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg2 = in_ptr32
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg3 = in_ptr33
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg4 = in_ptr34
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg5 = out_ptr39
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg6 = out_ptr40
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg7 = out_ptr41
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     elif pid < num_xblocks_7:
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         pid_offset = pid - num_xblocks_6
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         xnumel = 1048576
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         r0_numel = 1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg0 = in_ptr35
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg1 = in_ptr36
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg2 = in_ptr37
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg3 = in_ptr38
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg4 = in_ptr39
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg5 = out_ptr45
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg6 = out_ptr46
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg7 = out_ptr47
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     elif pid < num_xblocks_8:
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         pid_offset = pid - num_xblocks_7
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         xnumel = 1048576
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         r0_numel = 1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg0 = in_ptr40
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg1 = in_ptr41
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg2 = in_ptr42
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg3 = in_ptr43
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg4 = in_ptr44
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg5 = out_ptr51
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg6 = out_ptr52
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg7 = out_ptr53
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     elif pid < num_xblocks_9:
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         pid_offset = pid - num_xblocks_8
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         xnumel = 1048576
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         r0_numel = 1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg0 = in_ptr45
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg1 = in_ptr46
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg2 = in_ptr47
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg3 = in_ptr48
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg4 = in_ptr49
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg5 = out_ptr57
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg6 = out_ptr58
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg7 = out_ptr59
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     else:
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         pid_offset = 0
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         xnumel = 0
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         r0_numel = 0
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg0 = in_ptr0
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg1 = in_ptr1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg2 = in_ptr2
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg3 = in_ptr3
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg4 = in_ptr4
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg5 = out_ptr3
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg6 = out_ptr4
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         foreach_arg7 = out_ptr5
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     xoffset = pid_offset * XBLOCK
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     xindex = xoffset + tl.arange(0, XBLOCK)[:]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     xmask = tl.full([XBLOCK], True, tl.int1)[:]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     x0 = xindex
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp0 = tl.load(foreach_arg0 + (x0), None)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp1 = tl.load(foreach_arg1 + (x0), None)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp5 = tl.load(foreach_arg2 + (x0), None)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp11 = tl.load(foreach_arg3 + (x0), None)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp13 = foreach_arg4
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp2 = tmp1 - tmp0
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp3 = tl.full([1], 0.09999999999999998, tl.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp4 = tl.fma(tmp2, tmp3, tmp0)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp6 = tl.full([1], 0.999, tl.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp7 = tmp5 * tmp6
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp8 = libdevice.mul_rn(tmp1, tmp1)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp9 = tl.full([1], 0.0010000000000000009, tl.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp10 = tl.fma(tmp9, tmp8, tmp7)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp12 = tl.sqrt_rn(tmp10)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp14 = libdevice.pow(tmp6, tmp13)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp15 = tl.full([1], 1.0, tl.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp16 = tmp15 - tmp14
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp17 = tl.sqrt_rn(tmp16)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp18 = tl.full([1], 0.9, tl.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp19 = libdevice.pow(tmp18, tmp13)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp20 = tmp15 - tmp19
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp21 = (tmp15 / tmp20)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp22 = tl.full([1], 0.001, tl.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp23 = tmp21 * tmp22
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp24 = -tmp23
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp25 = tmp17 * tmp24
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp26 = (tmp12 / tmp25)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp27 = (tmp15 / tmp24)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp28 = tl.full([1], 1e-08, tl.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp29 = tmp27 * tmp28
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp30 = tmp26 + tmp29
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp31 = (tmp4 / tmp30)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tmp32 = tmp11 + tmp31
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tl.store(foreach_arg5 + (x0), tmp4, None)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tl.store(foreach_arg6 + (x0), tmp10, None)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     tl.store(foreach_arg7 + (x0), tmp32, None)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] ''', device_str='cuda')
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] async_compile.wait(globals())
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] del async_compile
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] class Runner:
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     def __init__(self, partitions):
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         self.partitions = partitions
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     def recursively_apply_fns(self, fns):
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         new_callables = []
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         for fn, c in zip(fns, self.partitions):
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             new_callables.append(fn(c))
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         self.partitions = new_callables
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     def call(self, args):
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         arg0_1, arg1_1, arg2_1, arg3_1, arg4_1, arg5_1, arg6_1, arg7_1, arg8_1, arg9_1, arg10_1, arg11_1, arg12_1, arg13_1, arg14_1, arg15_1, arg16_1, arg17_1, arg18_1, arg19_1, arg20_1, arg21_1, arg22_1, arg23_1, arg24_1, arg25_1, arg26_1, arg27_1, arg28_1, arg29_1, arg30_1, arg31_1, arg32_1, arg33_1, arg34_1, arg35_1, arg36_1, arg37_1, arg38_1, arg39_1, arg40_1, arg41_1, arg42_1, arg43_1, arg44_1, arg45_1, arg46_1, arg47_1, arg48_1, arg49_1 = args
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         args.clear()
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         assert_size_stride(arg10_1, (), (), 'input')
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         assert_size_stride(arg11_1, (), (), 'input')
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         assert_size_stride(arg12_1, (), (), 'input')
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         assert_size_stride(arg13_1, (), (), 'input')
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         assert_size_stride(arg14_1, (), (), 'input')
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         assert_size_stride(arg15_1, (), (), 'input')
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         assert_size_stride(arg16_1, (), (), 'input')
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         assert_size_stride(arg17_1, (), (), 'input')
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         assert_size_stride(arg18_1, (), (), 'input')
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         assert_size_stride(arg19_1, (), (), 'input')
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         assert_size_stride_grouped((arg20_1, arg30_1, arg40_1, arg0_1, arg21_1, arg31_1, arg41_1, arg1_1, arg22_1, arg32_1, arg42_1, arg2_1, arg23_1, arg33_1, arg43_1, arg3_1, arg24_1, arg34_1, arg44_1, arg4_1, arg25_1, arg35_1, arg45_1, arg5_1, arg26_1, arg36_1, arg46_1, arg6_1, arg27_1, arg37_1, arg47_1, arg7_1, arg28_1, arg38_1, arg48_1, arg8_1, arg29_1, arg39_1, arg49_1, arg9_1), ((1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024), (1024, 1024)), ((1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 1)), 'input')
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         cpp_fused__foreach_copy_0(arg10_1, arg11_1, arg12_1, arg13_1, arg14_1, arg15_1, arg16_1, arg17_1, arg18_1, arg19_1, arg10_1, arg11_1, arg12_1, arg13_1, arg14_1, arg15_1, arg16_1, arg17_1, arg18_1, arg19_1)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         with torch.cuda._DeviceGuard(0):
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             torch.cuda.set_device(0)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             arg30_1 = copy_if_misaligned(arg30_1)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             arg31_1 = copy_if_misaligned(arg31_1)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             arg32_1 = copy_if_misaligned(arg32_1)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             arg33_1 = copy_if_misaligned(arg33_1)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             arg34_1 = copy_if_misaligned(arg34_1)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             arg35_1 = copy_if_misaligned(arg35_1)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             arg36_1 = copy_if_misaligned(arg36_1)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             arg37_1 = copy_if_misaligned(arg37_1)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             arg38_1 = copy_if_misaligned(arg38_1)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             arg39_1 = copy_if_misaligned(arg39_1)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             # Unsorted Source Nodes: [], Original ATen: []
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             raw_stream0 = get_raw_stream(0)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             triton_for_fused_1.run(arg20_1, arg30_1, arg40_1, arg0_1, arg10_1.item(), arg21_1, arg31_1, arg41_1, arg1_1, arg11_1.item(), arg22_1, arg32_1, arg42_1, arg2_1, arg12_1.item(), arg23_1, arg33_1, arg43_1, arg3_1, arg13_1.item(), arg24_1, arg34_1, arg44_1, arg4_1, arg14_1.item(), arg25_1, arg35_1, arg45_1, arg5_1, arg15_1.item(), arg26_1, arg36_1, arg46_1, arg6_1, arg16_1.item(), arg27_1, arg37_1, arg47_1, arg7_1, arg17_1.item(), arg28_1, arg38_1, arg48_1, arg8_1, arg18_1.item(), arg29_1, arg39_1, arg49_1, arg9_1, arg19_1.item(), arg20_1, arg40_1, arg0_1, arg21_1, arg41_1, arg1_1, arg22_1, arg42_1, arg2_1, arg23_1, arg43_1, arg3_1, arg24_1, arg44_1, arg4_1, arg25_1, arg45_1, arg5_1, arg26_1, arg46_1, arg6_1, arg27_1, arg47_1, arg7_1, arg28_1, arg48_1, arg8_1, arg29_1, arg49_1, arg9_1, stream=raw_stream0)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg0_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg10_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg11_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg12_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg13_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg14_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg15_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg16_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg17_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg18_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg19_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg1_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg20_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg21_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg22_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg23_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg24_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg25_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg26_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg27_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg28_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg29_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg2_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg30_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg31_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg32_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg33_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg34_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg35_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg36_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg37_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg38_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg39_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg3_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg40_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg41_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg42_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg43_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg44_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg45_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg46_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg47_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg48_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg49_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg4_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg5_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg6_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg7_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg8_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]             del arg9_1
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]         return ()
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] runner = Runner(partitions=[])
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] call = runner.call
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] recursively_apply_fns = runner.recursively_apply_fns
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] def get_args():
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     from torch._dynamo.testing import rand_strided
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg0_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg1_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg2_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg3_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg4_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg5_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg6_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg7_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg8_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg9_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg10_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg11_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg12_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg13_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg14_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg15_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg16_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg17_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg18_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg19_1 = rand_strided((), (), device='cpu', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg20_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg21_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg22_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg23_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg24_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg25_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg26_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg27_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg28_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg29_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg30_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg31_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg32_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg33_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg34_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg35_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg36_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg37_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg38_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg39_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg40_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg41_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg42_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg43_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg44_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg45_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg46_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg47_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg48_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     arg49_1 = rand_strided((1024, 1024), (1024, 1), device='cuda:0', dtype=torch.float32)
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     return [arg0_1, arg1_1, arg2_1, arg3_1, arg4_1, arg5_1, arg6_1, arg7_1, arg8_1, arg9_1, arg10_1, arg11_1, arg12_1, arg13_1, arg14_1, arg15_1, arg16_1, arg17_1, arg18_1, arg19_1, arg20_1, arg21_1, arg22_1, arg23_1, arg24_1, arg25_1, arg26_1, arg27_1, arg28_1, arg29_1, arg30_1, arg31_1, arg32_1, arg33_1, arg34_1, arg35_1, arg36_1, arg37_1, arg38_1, arg39_1, arg40_1, arg41_1, arg42_1, arg43_1, arg44_1, arg45_1, arg46_1, arg47_1, arg48_1, arg49_1]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] def benchmark_compiled_module(args, times=10, repeat=10):
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     from torch._inductor.utils import print_performance
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     fn = lambda: call(list(args))
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     return print_performance(fn, times=times, repeat=repeat, device='cuda')
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code] if __name__ == "__main__":
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     from torch._inductor.wrapper_benchmark import compiled_module_main
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     args = get_args()
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]     compiled_module_main('None', lambda times, repeat: benchmark_compiled_module(args, times=times, repeat=repeat))
V0903 01:21:44.280000 26236 torch/_inductor/graph.py:3114] [0/1] [__output_code]
V0903 01:21:44.311000 26236 torch/_inductor/graph.py:3125] [0/1] [__output_code] Output code written to: /tmp/torchinductor_ci-user/5e/c5eetsctxdjbmnv6hn6aq7dinlvta34727bj7k4zq5e6vlelv3be.py
I0903 01:21:44.342000 26236 torch/_inductor/graph.py:3085] [0/1] [__output_code] Output code written to: /tmp/torchinductor_ci-user/5e/c5eetsctxdjbmnv6hn6aq7dinlvta34727bj7k4zq5e6vlelv3be.py
eager runtime: 1222.7033249973829us
compiled runtime: 724.5177869692717us

Conclusion

In this tutorial, we successfully implemented a custom fully-fused Adam optimizer using foreach_map. By leveraging the power of foreach_map and torch.compile, we were able to create an optimized version of the Adam optimizer that can be used in various machine learning applications. This tutorial provides a comprehensive guide on how to use foreach_map and torch.compile to optimize machine learning models, and serves as a valuable resource for developers looking to improve the performance of their models with horizontal fusion.

See also:

• Compiled optimizer tutorial - an intro into the compiled optimizer.

• Compiling the optimizer with PT2 - deeper technical details on the compiled optimizer.