-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  --------------------------------------------------------------------------------  ------------  
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg       CPU Mem  Self CPU Mem      CUDA Mem  Self CUDA Mem    # of Calls                                                                      Input Shapes   Total FLOPs  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  --------------------------------------------------------------------------------  ------------  
                                          ProfilerStep*         0.00%       0.000us         0.00%       0.000us       0.000us     803.025ms        68.11%     803.025ms      40.151ms           0 B           0 B           0 B           0 B            20                                                                                []            --  
autograd::engine::evaluate_function: CudnnRnnBackwar...         0.03%     200.740us         4.92%      32.925ms       3.292ms       0.000us         0.00%     596.043ms      59.604ms           0 B           0 B     -35.08 GB     -37.17 GB            10                                                                                []            --  
                                      CudnnRnnBackward0         0.02%     120.932us         4.89%      32.724ms       3.272ms       0.000us         0.00%     596.043ms      59.604ms           0 B           0 B       2.09 GB     -70.00 MB            10                                                         [[8192, 68, 224], [], []]            --  
                              aten::_cudnn_rnn_backward         2.76%      18.426ms         4.88%      32.603ms       3.260ms     578.815ms        49.09%     596.043ms      59.604ms           0 B           0 B       2.15 GB     -34.64 GB            10  [[8192, 68, 200], [], [], [588672], [2, 8192, 224], [], [8192, 68, 224], [8192,             --  
                        DistributedDataParallel.forward         0.00%       0.000us         0.00%       0.000us       0.000us     484.775ms        41.12%     484.775ms       9.696ms           0 B           0 B           0 B           0 B            50                                                                                []            --  
                                          ProfilerStep*        33.58%     224.519ms        42.42%     283.599ms      28.360ms       0.000us         0.00%     413.413ms      41.341ms           0 B           0 B           0 B     -39.24 GB            10                                                                                []            --  
                                       aten::_cudnn_rnn         2.45%      16.374ms         8.95%      59.826ms       2.991ms     148.925ms        12.63%     346.226ms      17.311ms           0 B           0 B      67.66 GB     -16.12 GB            20  [[8192, 68, 200], [], [], [588672], [2, 8192, 224], [], [], [], [], [], [], [],             --  
ampere_fp16_s16816gemm_fp16_128x64_ldg8_f2f_stages_6...         0.00%       0.000us         0.00%       0.000us       0.000us     256.211ms        21.73%     256.211ms       4.270ms           0 B           0 B           0 B           0 B            60                                                                                []            --  
                                         aten::_to_copy         0.39%       2.601ms         0.55%       3.666ms     122.203us       0.000us         0.00%     204.091ms       6.803ms           0 B           0 B      10.40 GB           0 B            30                                         [[8192, 68, 200], [], [], [], [], [], []]            --  
                                            aten::copy_         0.03%     187.472us         0.06%     411.720us      13.724us     204.091ms        17.31%     204.091ms       6.803ms           0 B           0 B           0 B           0 B            30                                            [[8192, 68, 200], [8192, 68, 200], []]            --  
                                               aten::to         0.00%      26.878us         0.53%       3.546ms     118.195us       0.000us         0.00%     197.008ms       6.567ms           0 B           0 B       8.32 GB           0 B            30                                     [[8192, 68, 200], [], [], [], [], [], [], []]            --  
                        DistributedDataParallel.forward         0.51%       3.384ms         6.59%      44.041ms       4.404ms       0.000us         0.00%     192.040ms      19.204ms           0 B           0 B      35.10 GB      -4.56 GB            10                                                                                []            --  
                         Memcpy HtoD (Pinned -> Device)         0.00%       0.000us         0.00%       0.000us       0.000us     190.179ms        16.13%     190.179ms       6.339ms           0 B           0 B           0 B           0 B            30                                                                                []            --  
                                              aten::gru         0.06%     416.035us         4.92%      32.892ms       3.289ms       0.000us         0.00%     176.772ms      17.677ms           0 B           0 B      34.91 GB           0 B            10                     [[8192, 68, 200], [2, 8192, 224], [], [], [], [], [], [], []]            --  
autograd::engine::evaluate_function: torch::autograd...         0.20%       1.344ms         0.82%       5.471ms      27.357us       0.000us         0.00%     151.216ms     756.080us           0 B           0 B           0 B           0 B           200                                                                                []            --  
                                       c10d::allreduce_         0.05%     328.445us         0.21%       1.383ms      69.171us       0.000us         0.00%     150.593ms       7.530ms           0 B           0 B           0 B           0 B            20                                                          [[], [], [], [], [], []]            --  
                                     record_param_comms         0.12%     776.737us         0.16%       1.055ms      52.749us     150.593ms        12.77%     150.593ms       7.530ms           0 B           0 B           0 B           0 B            20                                          [[], [], [], [], [], [], [], [], [], []]            --  
ncclDevKernel_AllReduce_Sum_f32_RING_LL(ncclDevKerne...         0.00%       0.000us         0.00%       0.000us       0.000us     150.593ms        12.77%     150.593ms       7.530ms           0 B           0 B           0 B           0 B            20                                                                                []            --  
                                        nccl:all_reduce         0.00%       0.000us         0.00%       0.000us       0.000us     150.593ms        12.77%     150.593ms       7.530ms           0 B           0 B           0 B           0 B            20                                                                                []            --  
void GRU_elementWise_bp2<__half, __half, float, 3, (...         0.00%       0.000us         0.00%       0.000us       0.000us     127.055ms        10.78%     127.055ms       6.353ms           0 B           0 B           0 B           0 B            20                                                                                []            --  
ampere_fp16_s1688gemm_fp16_128x128_ldg8_f2f_stages_3...         0.00%       0.000us         0.00%       0.000us       0.000us     107.403ms         9.11%     107.403ms      52.649us           0 B           0 B           0 B           0 B          2040                                                                                []            --  
void GRU_elementWise_bp1<__half, __half, float>(int,...         0.00%       0.000us         0.00%       0.000us       0.000us      73.392ms         6.22%      73.392ms      53.965us           0 B           0 B           0 B           0 B          1360                                                                                []            --  
void elemWiseRNNcell<__half, __half, float, (cudnnRN...         0.00%       0.000us         0.00%       0.000us       0.000us      66.896ms         5.67%      66.896ms      49.188us           0 B           0 B           0 B           0 B          1360                                                                                []            --  
ampere_fp16_s1688gemm_fp16_128x128_ldg8_f2f_stages_3...         0.00%       0.000us         0.00%       0.000us       0.000us      39.211ms         3.33%      39.211ms      28.831us           0 B           0 B           0 B           0 B          1360                                                                                []            --  
ampere_fp16_s1688gemm_fp16_256x64_ldg8_f2f_stages_32...         0.00%       0.000us         0.00%       0.000us       0.000us      32.743ms         2.78%      32.743ms      96.304us           0 B           0 B           0 B           0 B           340                                                                                []            --  
void at::native::elementwise_kernel<128, 2, at::nati...         0.00%       0.000us         0.00%       0.000us       0.000us      24.226ms         2.05%      24.226ms     807.543us           0 B           0 B           0 B           0 B            30                                                                                []            --  
void at::native::elementwise_kernel<128, 4, at::nati...         0.00%       0.000us         0.00%       0.000us       0.000us      23.009ms         1.95%      23.009ms     460.187us           0 B           0 B           0 B           0 B            50                                                                                []            --  
void cudnn::dropout_apply_mask<__half, float, 4>(flo...         0.00%       0.000us         0.00%       0.000us       0.000us      20.826ms         1.77%      20.826ms      40.049us           0 B           0 B           0 B           0 B           520                                                                                []            --  
      autograd::engine::evaluate_function: MulBackward0         0.06%     376.385us         0.16%       1.048ms      52.378us       0.000us         0.00%      20.096ms       1.005ms           0 B           0 B      -4.13 GB      -8.37 GB            20                                                                                []            --  
                                       aten::contiguous         0.01%      40.006us         0.09%     593.647us      19.788us       0.000us         0.00%      16.927ms     564.226us           0 B           0 B       6.23 GB           0 B            30                                                             [[68, 8192, 200], []]            --  
                                            aten::clone         0.01%      67.609us         0.08%     553.641us      18.455us       0.000us         0.00%      16.927ms     564.226us           0 B           0 B       6.23 GB           0 B            30                                                             [[68, 8192, 200], []]            --  
                                            aten::copy_         0.03%     169.915us         0.05%     333.683us      11.123us      16.927ms         1.44%      16.927ms     564.226us           0 B           0 B           0 B           0 B            30                                            [[68, 8192, 200], [68, 8192, 200], []]            --  
                                Activity Buffer Request         0.21%       1.388ms         0.21%       1.388ms       1.388ms      14.868ms         1.26%      14.868ms      14.868ms           0 B           0 B           0 B           0 B             1                                                                                []            --  
                                           MulBackward0         0.01%      33.593us         0.02%     140.953us      14.095us       0.000us         0.00%      14.729ms       1.473ms           0 B           0 B       4.15 GB           0 B            10                                                                 [[8192, 68, 200]]            --  
                                              aten::mul         0.01%      72.557us         0.02%     107.360us      10.736us      14.729ms         1.25%      14.729ms       1.473ms           0 B           0 B       4.15 GB       4.15 GB            10                                                [[8192, 68, 200], [8192, 68, 200]]  1114112000.000  
                                             aten::sub_         0.02%     149.992us         0.03%     227.069us      22.707us       9.638ms         0.82%       9.638ms     963.808us           0 B           0 B           0 B           0 B            10                                                [[8192, 68, 200], [1, 1, 200], []]            --  
void at::native::elementwise_kernel<128, 2, at::nati...         0.00%       0.000us         0.00%       0.000us       0.000us       9.638ms         0.82%       9.638ms     963.808us           0 B           0 B           0 B           0 B            10                                                                                []            --  
                                             aten::mul_         0.01%      75.814us         0.02%     142.083us      14.208us       9.484ms         0.80%       9.484ms     948.394us           0 B           0 B           0 B           0 B            10                                                    [[8192, 68, 200], [1, 1, 200]]            --  
                                      aten::nan_to_num_         0.01%      43.518us         0.04%     299.641us      29.964us       0.000us         0.00%       9.382ms     938.157us           0 B           0 B           0 B           0 B            10                                                     [[8192, 68, 200], [], [], []]            --  
                                       aten::nan_to_num         0.01%      94.869us         0.04%     256.123us      25.612us       9.382ms         0.80%       9.382ms     938.157us           0 B           0 B           0 B           0 B            10                                    [[8192, 68, 200], [], [], [], [8192, 68, 200]]            --  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  --------------------------------------------------------------------------------  ------------  
Self CPU time total: 668.564ms
Self CUDA time total: 1.179s
