-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  --------------------------------------------------------------------------------  ------------  
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg       CPU Mem  Self CPU Mem      CUDA Mem  Self CUDA Mem    # of Calls                                                                      Input Shapes   Total FLOPs  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  --------------------------------------------------------------------------------  ------------  
                                          ProfilerStep*         0.00%       0.000us         0.00%       0.000us       0.000us     866.996ms        70.08%     866.996ms      43.350ms           0 B           0 B           0 B           0 B            20                                                                                []            --  
autograd::engine::evaluate_function: CudnnRnnBackwar...         0.02%     229.976us         4.43%      43.010ms       4.301ms       0.000us         0.00%     598.506ms      59.851ms           0 B           0 B     -35.08 GB     -37.17 GB            10                                                                                []            --  
                                      CudnnRnnBackward0         0.07%     656.993us         4.41%      42.780ms       4.278ms       0.000us         0.00%     598.506ms      59.851ms           0 B           0 B       2.09 GB     -70.00 MB            10                                                         [[8192, 68, 224], [], []]            --  
                              aten::_cudnn_rnn_backward         2.51%      24.336ms         4.34%      42.123ms       4.212ms     581.305ms        46.99%     598.506ms      59.851ms           0 B           0 B       2.15 GB     -34.64 GB            10  [[8192, 68, 200], [], [], [588672], [2, 8192, 224], [], [8192, 68, 224], [8192,             --  
                        DistributedDataParallel.forward         0.00%       0.000us         0.00%       0.000us       0.000us     487.938ms        39.44%     487.938ms       9.759ms           0 B           0 B           0 B           0 B            50                                                                                []            --  
                                          ProfilerStep*        -8.75%  -84947.346us        76.03%     738.146ms      73.815ms       0.000us         0.00%     380.179ms      38.018ms           0 B          -8 B     747.00 KB     -39.24 GB            10                                                                                []            --  
                                       aten::_cudnn_rnn         1.46%      14.173ms         5.06%      49.107ms       2.455ms     148.281ms        11.99%     314.995ms      15.750ms           0 B           0 B      67.66 GB     -16.12 GB            20  [[8192, 68, 200], [], [], [588672], [2, 8192, 224], [], [], [], [], [], [], [],             --  
ampere_fp16_s16816gemm_fp16_128x64_ldg8_f2f_stages_6...         0.00%       0.000us         0.00%       0.000us       0.000us     255.999ms        20.69%     255.999ms       4.267ms           0 B           0 B           0 B           0 B            60                                                                                []            --  
autograd::engine::evaluate_function: torch::autograd...         0.22%       2.135ms         0.93%       9.045ms      45.226us       0.000us         0.00%     225.508ms       1.128ms           0 B           8 B           0 B           0 B           200                                                                                []            --  
                                     record_param_comms         0.17%       1.605ms         0.22%       2.107ms      95.779us     224.862ms        18.18%     224.862ms      10.221ms           0 B           0 B           0 B           0 B            22                                          [[], [], [], [], [], [], [], [], [], []]            --  
                                       c10d::allreduce_         0.06%     606.439us         0.24%       2.309ms     115.460us       0.000us         0.00%     224.845ms      11.242ms           0 B           0 B           0 B           0 B            20                                                          [[], [], [], [], [], []]            --  
ncclDevKernel_AllReduce_Sum_f32_RING_LL(ncclDevKerne...         0.00%       0.000us         0.00%       0.000us       0.000us     224.845ms        18.17%     224.845ms      11.242ms           0 B           0 B           0 B           0 B            20                                                                                []            --  
                                        nccl:all_reduce         0.00%       0.000us         0.00%       0.000us       0.000us     224.845ms        18.17%     224.845ms      11.242ms           0 B           0 B           0 B           0 B            20                                                                                []            --  
                                         aten::_to_copy         0.03%     272.862us         0.15%       1.449ms      48.310us       0.000us         0.00%     186.864ms       6.229ms           0 B           0 B      10.40 GB           0 B            30                                         [[8192, 68, 200], [], [], [], [], [], []]            --  
                                            aten::copy_         0.02%     169.657us         0.04%     404.794us      13.493us     186.864ms        15.10%     186.864ms       6.229ms           0 B           0 B           0 B           0 B            30                                            [[8192, 68, 200], [8192, 68, 200], []]            --  
                                               aten::to         0.26%       2.548ms         0.40%       3.882ms     129.392us       0.000us         0.00%     179.841ms       5.995ms           0 B           0 B       8.32 GB           0 B            30                                     [[8192, 68, 200], [], [], [], [], [], [], []]            --  
                        DistributedDataParallel.forward         0.90%       8.767ms        42.78%     415.292ms      41.529ms       0.000us         0.00%     176.372ms      17.637ms           0 B         -92 B      35.10 GB      -4.55 GB            10                                                                                []            --  
                         Memcpy HtoD (Pinned -> Device)         0.00%       0.000us         0.00%       0.000us       0.000us     173.029ms        13.99%     173.029ms       5.768ms           0 B           0 B           0 B           0 B            30                                                                                []            --  
                                              aten::gru         0.08%     792.709us         2.87%      27.818ms       2.782ms       0.000us         0.00%     161.123ms      16.112ms           0 B           0 B      34.91 GB           0 B            10                     [[8192, 68, 200], [2, 8192, 224], [], [], [], [], [], [], []]            --  
void GRU_elementWise_bp2<__half, __half, float, 3, (...         0.00%       0.000us         0.00%       0.000us       0.000us     130.687ms        10.56%     130.687ms       6.534ms           0 B           0 B           0 B           0 B            20                                                                                []            --  
ampere_fp16_s1688gemm_fp16_128x128_ldg8_f2f_stages_3...         0.00%       0.000us         0.00%       0.000us       0.000us     106.878ms         8.64%     106.878ms      52.391us           0 B           0 B           0 B           0 B          2040                                                                                []            --  
void GRU_elementWise_bp1<__half, __half, float>(int,...         0.00%       0.000us         0.00%       0.000us       0.000us      72.384ms         5.85%      72.384ms      53.223us           0 B           0 B           0 B           0 B          1360                                                                                []            --  
void elemWiseRNNcell<__half, __half, float, (cudnnRN...         0.00%       0.000us         0.00%       0.000us       0.000us      66.704ms         5.39%      66.704ms      49.047us           0 B           0 B           0 B           0 B          1360                                                                                []            --  
ampere_fp16_s1688gemm_fp16_128x128_ldg8_f2f_stages_3...         0.00%       0.000us         0.00%       0.000us       0.000us      39.131ms         3.16%      39.131ms      28.773us           0 B           0 B           0 B           0 B          1360                                                                                []            --  
ampere_fp16_s1688gemm_fp16_256x64_ldg8_f2f_stages_32...         0.00%       0.000us         0.00%       0.000us       0.000us      32.504ms         2.63%      32.504ms      95.599us           0 B           0 B           0 B           0 B           340                                                                                []            --  
void at::native::elementwise_kernel<128, 2, at::nati...         0.00%       0.000us         0.00%       0.000us       0.000us      24.204ms         1.96%      24.204ms     806.791us           0 B           0 B           0 B           0 B            30                                                                                []            --  
void at::native::elementwise_kernel<128, 4, at::nati...         0.00%       0.000us         0.00%       0.000us       0.000us      22.913ms         1.85%      22.913ms     458.251us           0 B           0 B           0 B           0 B            50                                                                                []            --  
void cudnn::dropout_apply_mask<__half, float, 4>(flo...         0.00%       0.000us         0.00%       0.000us       0.000us      21.343ms         1.73%      21.343ms      41.044us           0 B           0 B           0 B           0 B           520                                                                                []            --  
      autograd::engine::evaluate_function: MulBackward0         0.07%     678.124us         0.18%       1.735ms      86.753us       0.000us         0.00%      20.089ms       1.004ms           0 B           0 B      -4.13 GB      -8.37 GB            20                                                                                []            --  
                                       aten::contiguous         0.01%      92.658us         0.08%     742.293us      24.743us       0.000us         0.00%      16.846ms     561.524us           0 B           0 B       6.23 GB           0 B            30                                                             [[68, 8192, 200], []]            --  
                                            aten::clone         0.01%      84.726us         0.07%     649.635us      21.655us       0.000us         0.00%      16.846ms     561.524us           0 B           0 B       6.23 GB           0 B            30                                                             [[68, 8192, 200], []]            --  
                                            aten::copy_         0.02%     217.568us         0.04%     392.378us      13.079us      16.846ms         1.36%      16.846ms     561.524us           0 B           0 B           0 B           0 B            30                                            [[68, 8192, 200], [68, 8192, 200], []]            --  
                                           MulBackward0         0.00%      37.016us         0.02%     185.028us      18.503us       0.000us         0.00%      14.732ms       1.473ms           0 B           0 B       4.15 GB           0 B            10                                                                 [[8192, 68, 200]]            --  
                                              aten::mul         0.01%      97.537us         0.02%     148.012us      14.801us      14.732ms         1.19%      14.732ms       1.473ms           0 B           0 B       4.15 GB       4.15 GB            10                                                [[8192, 68, 200], [8192, 68, 200]]  1114112000.000  
                                             aten::mul_         0.01%      99.791us         0.02%     162.704us      16.270us       9.458ms         0.76%       9.458ms     945.827us           0 B           0 B           0 B           0 B            10                                                    [[8192, 68, 200], [1, 1, 200]]            --  
                                             aten::sub_         0.02%     177.307us         0.03%     257.487us      25.749us       9.436ms         0.76%       9.436ms     943.632us           0 B           0 B           0 B           0 B            10                                                [[8192, 68, 200], [1, 1, 200], []]            --  
void at::native::elementwise_kernel<128, 2, at::nati...         0.00%       0.000us         0.00%       0.000us       0.000us       9.436ms         0.76%       9.436ms     943.632us           0 B           0 B           0 B           0 B            10                                                                                []            --  
                                      aten::nan_to_num_         0.01%      51.860us         0.03%     318.239us      31.824us       0.000us         0.00%       9.194ms     919.385us           0 B           0 B           0 B           0 B            10                                                     [[8192, 68, 200], [], [], []]            --  
                                       aten::nan_to_num         0.01%     115.642us         0.03%     266.379us      26.638us       9.194ms         0.74%       9.194ms     919.385us           0 B           0 B           0 B           0 B            10                                    [[8192, 68, 200], [], [], [], [8192, 68, 200]]            --  
void at::native::vectorized_elementwise_kernel<4, at...         0.00%       0.000us         0.00%       0.000us       0.000us       9.194ms         0.74%       9.194ms     919.385us           0 B           0 B           0 B           0 B            10                                                                                []            --  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  --------------------------------------------------------------------------------  ------------  
Self CPU time total: 970.868ms
Self CUDA time total: 1.237s
