+
Skip to content

[CI] Fix GPU benchmark upload #2508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Oct 21, 2024
Merged

[CI] Fix GPU benchmark upload #2508

merged 3 commits into from
Oct 21, 2024

Conversation

vmoens
Copy link
Collaborator

@vmoens vmoens commented Oct 21, 2024 •
edited
Loading

No description provided.

Vincent Moens added 2 commits October 21, 2024 14:25
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Oct 21, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2508

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 6 Unrelated Failures

As of commit 9d93f72 with merge base 56b0b9a (image):

NEW FAILURES - The following jobs have failed:

FLAKY - The following job failed but was likely due to flakiness present on trunk:

BROKEN TRUNK - The following jobs failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 21, 2024
@vmoens vmoens added the CI Has to do with CI setup (e.g. wheels & builds, tests...) label Oct 21, 2024
@github-actions GitHub Actions
Copy link

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 143. Improved: $\large\color{#35bf28}2$. Worsened: $\large\color{#d91a1a}20$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_simple 0.4212s 0.4181s 2.3920 Ops/s 2.2750 Ops/s $\textbf{\color{#35bf28}+5.14\%}$
test_transformed 0.7075s 0.6153s 1.6252 Ops/s 1.6849 Ops/s $\color{#d91a1a}-3.54\%$
test_serial 1.4467s 1.3558s 0.7376 Ops/s 0.7322 Ops/s $\color{#35bf28}+0.73\%$
test_parallel 1.4254s 1.3379s 0.7474 Ops/s 0.7412 Ops/s $\color{#35bf28}+0.84\%$
test_step_mdp_speed[True-True-True-True-True] 0.2449ms 29.0004μs 34.4823 KOps/s 34.8037 KOps/s $\color{#d91a1a}-0.92\%$
test_step_mdp_speed[True-True-True-True-False] 95.9000μs 17.7937μs 56.1997 KOps/s 58.7654 KOps/s $\color{#d91a1a}-4.37\%$
test_step_mdp_speed[True-True-True-False-True] 43.8020μs 15.9339μs 62.7594 KOps/s 63.0124 KOps/s $\color{#d91a1a}-0.40\%$
test_step_mdp_speed[True-True-True-False-False] 58.9200μs 9.6494μs 103.6336 KOps/s 107.1143 KOps/s $\color{#d91a1a}-3.25\%$
test_step_mdp_speed[True-True-False-True-True] 66.5040μs 31.3405μs 31.9076 KOps/s 32.6591 KOps/s $\color{#d91a1a}-2.30\%$
test_step_mdp_speed[True-True-False-True-False] 68.7390μs 19.8329μs 50.4213 KOps/s 51.9009 KOps/s $\color{#d91a1a}-2.85\%$
test_step_mdp_speed[True-True-False-False-True] 60.4230μs 18.1810μs 55.0025 KOps/s 56.9846 KOps/s $\color{#d91a1a}-3.48\%$
test_step_mdp_speed[True-True-False-False-False] 51.5960μs 11.8480μs 84.4026 KOps/s 87.5504 KOps/s $\color{#d91a1a}-3.60\%$
test_step_mdp_speed[True-False-True-True-True] 83.2460μs 33.6480μs 29.7195 KOps/s 30.6340 KOps/s $\color{#d91a1a}-2.99\%$
test_step_mdp_speed[True-False-True-True-False] 73.0760μs 22.0103μs 45.4332 KOps/s 47.3575 KOps/s $\color{#d91a1a}-4.06\%$
test_step_mdp_speed[True-False-True-False-True] 49.5220μs 18.2005μs 54.9435 KOps/s 56.0421 KOps/s $\color{#d91a1a}-1.96\%$
test_step_mdp_speed[True-False-True-False-False] 57.0760μs 11.7069μs 85.4196 KOps/s 88.3002 KOps/s $\color{#d91a1a}-3.26\%$
test_step_mdp_speed[True-False-False-True-True] 65.7530μs 35.6101μs 28.0819 KOps/s 29.2282 KOps/s $\color{#d91a1a}-3.92\%$
test_step_mdp_speed[True-False-False-True-False] 53.6410μs 23.9957μs 41.6741 KOps/s 43.5259 KOps/s $\color{#d91a1a}-4.25\%$
test_step_mdp_speed[True-False-False-False-True] 59.7320μs 20.3873μs 49.0500 KOps/s 50.4815 KOps/s $\color{#d91a1a}-2.84\%$
test_step_mdp_speed[True-False-False-False-False] 42.7800μs 13.7390μs 72.7853 KOps/s 74.7778 KOps/s $\color{#d91a1a}-2.66\%$
test_step_mdp_speed[False-True-True-True-True] 87.5440μs 33.5231μs 29.8302 KOps/s 30.5637 KOps/s $\color{#d91a1a}-2.40\%$
test_step_mdp_speed[False-True-True-True-False] 68.1370μs 22.0219μs 45.4093 KOps/s 47.5018 KOps/s $\color{#d91a1a}-4.40\%$
test_step_mdp_speed[False-True-True-False-True] 65.1910μs 21.3693μs 46.7960 KOps/s 47.7288 KOps/s $\color{#d91a1a}-1.95\%$
test_step_mdp_speed[False-True-True-False-False] 60.6330μs 13.4469μs 74.3665 KOps/s 76.2740 KOps/s $\color{#d91a1a}-2.50\%$
test_step_mdp_speed[False-True-False-True-True] 80.2800μs 35.2302μs 28.3847 KOps/s 29.0922 KOps/s $\color{#d91a1a}-2.43\%$
test_step_mdp_speed[False-True-False-True-False] 58.9610μs 24.1744μs 41.3660 KOps/s 43.6480 KOps/s $\textbf{\color{#d91a1a}-5.23\%}$
test_step_mdp_speed[False-True-False-False-True] 2.7166ms 23.6500μs 42.2832 KOps/s 43.6374 KOps/s $\color{#d91a1a}-3.10\%$
test_step_mdp_speed[False-True-False-False-False] 61.5450μs 15.7508μs 63.4890 KOps/s 66.3159 KOps/s $\color{#d91a1a}-4.26\%$
test_step_mdp_speed[False-False-True-True-True] 69.7590μs 37.6085μs 26.5898 KOps/s 27.6278 KOps/s $\color{#d91a1a}-3.76\%$
test_step_mdp_speed[False-False-True-True-False] 70.5810μs 26.3069μs 38.0129 KOps/s 39.8298 KOps/s $\color{#d91a1a}-4.56\%$
test_step_mdp_speed[False-False-True-False-True] 48.3500μs 23.5384μs 42.4838 KOps/s 43.1493 KOps/s $\color{#d91a1a}-1.54\%$
test_step_mdp_speed[False-False-True-False-False] 59.9520μs 15.6002μs 64.1019 KOps/s 66.0817 KOps/s $\color{#d91a1a}-3.00\%$
test_step_mdp_speed[False-False-False-True-True] 88.3250μs 39.3772μs 25.3954 KOps/s 26.4578 KOps/s $\color{#d91a1a}-4.02\%$
test_step_mdp_speed[False-False-False-True-False] 69.5100μs 28.0720μs 35.6226 KOps/s 37.4358 KOps/s $\color{#d91a1a}-4.84\%$
test_step_mdp_speed[False-False-False-False-True] 58.8200μs 25.4848μs 39.2391 KOps/s 38.1409 KOps/s $\color{#35bf28}+2.88\%$
test_step_mdp_speed[False-False-False-False-False] 50.7860μs 17.6100μs 56.7858 KOps/s 58.0869 KOps/s $\color{#d91a1a}-2.24\%$
test_values[generalized_advantage_estimate-True-True] 15.2579ms 10.1725ms 98.3044 Ops/s 101.5720 Ops/s $\color{#d91a1a}-3.22\%$
test_values[vec_generalized_advantage_estimate-True-True] 39.1026ms 35.3847ms 28.2608 Ops/s 29.5930 Ops/s $\color{#d91a1a}-4.50\%$
test_values[td0_return_estimate-False-False] 0.2494ms 0.1910ms 5.2362 KOps/s 5.1710 KOps/s $\color{#35bf28}+1.26\%$
test_values[td1_return_estimate-False-False] 28.2393ms 23.9150ms 41.8147 Ops/s 40.4977 Ops/s $\color{#35bf28}+3.25\%$
test_values[vec_td1_return_estimate-False-False] 37.8907ms 35.6740ms 28.0316 Ops/s 29.4343 Ops/s $\color{#d91a1a}-4.77\%$
test_values[td_lambda_return_estimate-True-False] 34.1748ms 33.7294ms 29.6477 Ops/s 28.3710 Ops/s $\color{#35bf28}+4.50\%$
test_values[vec_td_lambda_return_estimate-True-False] 41.0526ms 35.7154ms 27.9991 Ops/s 29.4797 Ops/s $\textbf{\color{#d91a1a}-5.02\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512] 10.6707ms 8.4299ms 118.6248 Ops/s 118.4137 Ops/s $\color{#35bf28}+0.18\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.7139ms 2.0529ms 487.1043 Ops/s 493.6166 Ops/s $\color{#d91a1a}-1.32\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.5016ms 0.3598ms 2.7791 KOps/s 2.7337 KOps/s $\color{#35bf28}+1.66\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 49.7850ms 47.7722ms 20.9327 Ops/s 22.9340 Ops/s $\textbf{\color{#d91a1a}-8.73\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 4.0284ms 3.0655ms 326.2146 Ops/s 319.9916 Ops/s $\color{#35bf28}+1.94\%$
test_dqn_speed[False-None] 6.1148ms 1.3635ms 733.4282 Ops/s 724.3248 Ops/s $\color{#35bf28}+1.26\%$
test_dqn_speed[False-backward] 2.6683ms 1.8748ms 533.3886 Ops/s 535.5306 Ops/s $\color{#d91a1a}-0.40\%$
test_dqn_speed[True-None] 0.6919ms 0.4688ms 2.1330 KOps/s 2.1118 KOps/s $\color{#35bf28}+1.00\%$
test_dqn_speed[True-backward] 0.9428ms 0.8919ms 1.1212 KOps/s 1.0848 KOps/s $\color{#35bf28}+3.36\%$
test_dqn_speed[reduce-overhead-None] 0.8100ms 0.4726ms 2.1160 KOps/s 2.0459 KOps/s $\color{#35bf28}+3.43\%$
test_dqn_speed[reduce-overhead-backward] 0.9329ms 0.8882ms 1.1258 KOps/s 1.0976 KOps/s $\color{#35bf28}+2.57\%$
test_ddpg_speed[False-None] 3.5924ms 2.8199ms 354.6230 Ops/s 343.4439 Ops/s $\color{#35bf28}+3.26\%$
test_ddpg_speed[False-backward] 4.9835ms 4.0323ms 247.9959 Ops/s 241.4570 Ops/s $\color{#35bf28}+2.71\%$
test_ddpg_speed[True-None] 1.2149ms 1.0096ms 990.4536 Ops/s 967.4440 Ops/s $\color{#35bf28}+2.38\%$
test_ddpg_speed[True-backward] 2.0113ms 1.9210ms 520.5566 Ops/s 508.2219 Ops/s $\color{#35bf28}+2.43\%$
test_ddpg_speed[reduce-overhead-None] 1.4631ms 1.0186ms 981.7840 Ops/s 979.5917 Ops/s $\color{#35bf28}+0.22\%$
test_ddpg_speed[reduce-overhead-backward] 2.0607ms 1.9647ms 508.9746 Ops/s 514.5444 Ops/s $\color{#d91a1a}-1.08\%$
test_sac_speed[False-None] 10.0102ms 8.1958ms 122.0135 Ops/s 123.8084 Ops/s $\color{#d91a1a}-1.45\%$
test_sac_speed[False-backward] 11.7877ms 10.9062ms 91.6907 Ops/s 92.2702 Ops/s $\color{#d91a1a}-0.63\%$
test_sac_speed[True-None] 2.6391ms 1.9980ms 500.4900 Ops/s 529.2783 Ops/s $\textbf{\color{#d91a1a}-5.44\%}$
test_sac_speed[True-backward] 3.7081ms 3.5843ms 278.9919 Ops/s 269.0800 Ops/s $\color{#35bf28}+3.68\%$
test_sac_speed[reduce-overhead-None] 3.9747ms 1.8770ms 532.7568 Ops/s 530.4749 Ops/s $\color{#35bf28}+0.43\%$
test_sac_speed[reduce-overhead-backward] 4.4332ms 3.6619ms 273.0795 Ops/s 275.0077 Ops/s $\color{#d91a1a}-0.70\%$
test_redq_speed[False-None] 14.8390ms 13.1861ms 75.8375 Ops/s 75.6714 Ops/s $\color{#35bf28}+0.22\%$
test_redq_speed[False-backward] 24.1178ms 22.7396ms 43.9762 Ops/s 44.0945 Ops/s $\color{#d91a1a}-0.27\%$
test_redq_speed[True-None] 5.9214ms 5.1077ms 195.7843 Ops/s 197.0747 Ops/s $\color{#d91a1a}-0.65\%$
test_redq_speed[True-backward] 13.3547ms 12.7097ms 78.6799 Ops/s 78.9239 Ops/s $\color{#d91a1a}-0.31\%$
test_redq_speed[reduce-overhead-None] 6.4833ms 5.6089ms 178.2866 Ops/s 198.7455 Ops/s $\textbf{\color{#d91a1a}-10.29\%}$
test_redq_speed[reduce-overhead-backward] 13.7106ms 13.1923ms 75.8019 Ops/s 78.6508 Ops/s $\color{#d91a1a}-3.62\%$
test_redq_deprec_speed[False-None] 16.8435ms 14.2723ms 70.0658 Ops/s 73.9563 Ops/s $\textbf{\color{#d91a1a}-5.26\%}$
test_redq_deprec_speed[False-backward] 21.5913ms 20.4567ms 48.8838 Ops/s 51.3620 Ops/s $\color{#d91a1a}-4.82\%$
test_redq_deprec_speed[True-None] 5.0628ms 4.2989ms 232.6182 Ops/s 260.9837 Ops/s $\textbf{\color{#d91a1a}-10.87\%}$
test_redq_deprec_speed[True-backward] 9.4703ms 9.2232ms 108.4224 Ops/s 115.4738 Ops/s $\textbf{\color{#d91a1a}-6.11\%}$
test_redq_deprec_speed[reduce-overhead-None] 4.8949ms 4.1691ms 239.8605 Ops/s 264.8905 Ops/s $\textbf{\color{#d91a1a}-9.45\%}$
test_redq_deprec_speed[reduce-overhead-backward] 9.8449ms 8.9266ms 112.0251 Ops/s 115.6960 Ops/s $\color{#d91a1a}-3.17\%$
test_td3_speed[False-None] 10.2428ms 8.2865ms 120.6779 Ops/s 121.6570 Ops/s $\color{#d91a1a}-0.80\%$
test_td3_speed[False-backward] 13.2572ms 10.7212ms 93.2727 Ops/s 90.3101 Ops/s $\color{#35bf28}+3.28\%$
test_td3_speed[True-None] 2.0954ms 1.8075ms 553.2515 Ops/s 564.9059 Ops/s $\color{#d91a1a}-2.06\%$
test_td3_speed[True-backward] 3.7046ms 3.5639ms 280.5922 Ops/s 274.4033 Ops/s $\color{#35bf28}+2.26\%$
test_td3_speed[reduce-overhead-None] 2.0057ms 1.7745ms 563.5279 Ops/s 547.8341 Ops/s $\color{#35bf28}+2.86\%$
test_td3_speed[reduce-overhead-backward] 3.7811ms 3.5216ms 283.9640 Ops/s 284.3450 Ops/s $\color{#d91a1a}-0.13\%$
test_cql_speed[False-None] 39.7708ms 36.6837ms 27.2601 Ops/s 27.4522 Ops/s $\color{#d91a1a}-0.70\%$
test_cql_speed[False-backward] 50.7723ms 47.8401ms 20.9029 Ops/s 21.4955 Ops/s $\color{#d91a1a}-2.76\%$
test_cql_speed[True-None] 16.9326ms 16.0918ms 62.1435 Ops/s 62.6023 Ops/s $\color{#d91a1a}-0.73\%$
test_cql_speed[True-backward] 24.3297ms 22.8039ms 43.8522 Ops/s 42.7419 Ops/s $\color{#35bf28}+2.60\%$
test_cql_speed[reduce-overhead-None] 17.1094ms 15.8791ms 62.9759 Ops/s 62.2393 Ops/s $\color{#35bf28}+1.18\%$
test_cql_speed[reduce-overhead-backward] 24.3538ms 23.2852ms 42.9457 Ops/s 42.6083 Ops/s $\color{#35bf28}+0.79\%$
test_a2c_speed[False-None] 8.6954ms 7.5727ms 132.0530 Ops/s 133.5812 Ops/s $\color{#d91a1a}-1.14\%$
test_a2c_speed[False-backward] 15.5767ms 14.8254ms 67.4517 Ops/s 66.5665 Ops/s $\color{#35bf28}+1.33\%$
test_a2c_speed[True-None] 4.2605ms 3.4253ms 291.9464 Ops/s 289.0894 Ops/s $\color{#35bf28}+0.99\%$
test_a2c_speed[True-backward] 11.0500ms 10.7184ms 93.2975 Ops/s 96.1599 Ops/s $\color{#d91a1a}-2.98\%$
test_a2c_speed[reduce-overhead-None] 3.8062ms 3.4941ms 286.1953 Ops/s 288.5248 Ops/s $\color{#d91a1a}-0.81\%$
test_a2c_speed[reduce-overhead-backward] 11.8806ms 10.5669ms 94.6355 Ops/s 96.3484 Ops/s $\color{#d91a1a}-1.78\%$
test_ppo_speed[False-None] 8.4546ms 7.8653ms 127.1405 Ops/s 127.6959 Ops/s $\color{#d91a1a}-0.43\%$
test_ppo_speed[False-backward] 16.9017ms 15.3935ms 64.9626 Ops/s 66.0346 Ops/s $\color{#d91a1a}-1.62\%$
test_ppo_speed[True-None] 4.2419ms 3.8755ms 258.0284 Ops/s 259.2632 Ops/s $\color{#d91a1a}-0.48\%$
test_ppo_speed[True-backward] 10.7663ms 10.2336ms 97.7170 Ops/s 98.0036 Ops/s $\color{#d91a1a}-0.29\%$
test_ppo_speed[reduce-overhead-None] 4.4951ms 3.8095ms 262.5019 Ops/s 263.4664 Ops/s $\color{#d91a1a}-0.37\%$
test_ppo_speed[reduce-overhead-backward] 10.8909ms 10.1635ms 98.3913 Ops/s 97.8382 Ops/s $\color{#35bf28}+0.57\%$
test_reinforce_speed[False-None] 9.2380ms 6.6714ms 149.8930 Ops/s 151.4433 Ops/s $\color{#d91a1a}-1.02\%$
test_reinforce_speed[False-backward] 10.3520ms 10.0909ms 99.0991 Ops/s 99.5663 Ops/s $\color{#d91a1a}-0.47\%$
test_reinforce_speed[True-None] 3.3758ms 2.7799ms 359.7203 Ops/s 367.0300 Ops/s $\color{#d91a1a}-1.99\%$
test_reinforce_speed[True-backward] 9.9503ms 9.5464ms 104.7517 Ops/s 110.9084 Ops/s $\textbf{\color{#d91a1a}-5.55\%}$
test_reinforce_speed[reduce-overhead-None] 3.6484ms 2.9321ms 341.0515 Ops/s 349.2373 Ops/s $\color{#d91a1a}-2.34\%$
test_reinforce_speed[reduce-overhead-backward] 10.1195ms 9.4342ms 105.9976 Ops/s 109.5950 Ops/s $\color{#d91a1a}-3.28\%$
test_iql_speed[False-None] 34.3960ms 32.9036ms 30.3918 Ops/s 30.1380 Ops/s $\color{#35bf28}+0.84\%$
test_iql_speed[False-backward] 59.6946ms 46.8269ms 21.3552 Ops/s 21.5883 Ops/s $\color{#d91a1a}-1.08\%$
test_iql_speed[True-None] 11.9185ms 11.1463ms 89.7157 Ops/s 89.9322 Ops/s $\color{#d91a1a}-0.24\%$
test_iql_speed[True-backward] 24.7661ms 22.9406ms 43.5909 Ops/s 43.0103 Ops/s $\color{#35bf28}+1.35\%$
test_iql_speed[reduce-overhead-None] 11.7346ms 11.1415ms 89.7542 Ops/s 89.2358 Ops/s $\color{#35bf28}+0.58\%$
test_iql_speed[reduce-overhead-backward] 24.1379ms 23.1986ms 43.1060 Ops/s 43.5490 Ops/s $\color{#d91a1a}-1.02\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 7.2706ms 5.3417ms 187.2060 Ops/s 193.1559 Ops/s $\color{#d91a1a}-3.08\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7726ms 0.5060ms 1.9762 KOps/s 2.0149 KOps/s $\color{#d91a1a}-1.92\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.8343ms 0.4784ms 2.0902 KOps/s 2.1503 KOps/s $\color{#d91a1a}-2.80\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.8324ms 5.0552ms 197.8146 Ops/s 204.8198 Ops/s $\color{#d91a1a}-3.42\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3.0700ms 0.5011ms 1.9957 KOps/s 2.0323 KOps/s $\color{#d91a1a}-1.80\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6934ms 0.4729ms 2.1147 KOps/s 2.1504 KOps/s $\color{#d91a1a}-1.66\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 2.3494ms 1.6159ms 618.8428 Ops/s 620.7171 Ops/s $\color{#d91a1a}-0.30\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 2.1904ms 1.5612ms 640.5153 Ops/s 636.1644 Ops/s $\color{#35bf28}+0.68\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.6309ms 5.2699ms 189.7578 Ops/s 196.2382 Ops/s $\color{#d91a1a}-3.30\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.2555ms 0.6468ms 1.5462 KOps/s 1.5931 KOps/s $\color{#d91a1a}-2.95\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8503ms 0.6110ms 1.6367 KOps/s 1.6491 KOps/s $\color{#d91a1a}-0.75\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 7.7109ms 5.2018ms 192.2418 Ops/s 203.3912 Ops/s $\textbf{\color{#d91a1a}-5.48\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.1413ms 0.5402ms 1.8510 KOps/s 1.9592 KOps/s $\textbf{\color{#d91a1a}-5.52\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6931ms 0.4741ms 2.1091 KOps/s 2.0942 KOps/s $\color{#35bf28}+0.71\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 5.5513ms 5.0885ms 196.5229 Ops/s 200.4024 Ops/s $\color{#d91a1a}-1.94\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 2.1928ms 0.5014ms 1.9944 KOps/s 2.0071 KOps/s $\color{#d91a1a}-0.63\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7175ms 0.4765ms 2.0986 KOps/s 2.0821 KOps/s $\color{#35bf28}+0.79\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 8.1351ms 5.5909ms 178.8619 Ops/s 188.3415 Ops/s $\textbf{\color{#d91a1a}-5.03\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2.7614ms 0.6496ms 1.5393 KOps/s 1.5663 KOps/s $\color{#d91a1a}-1.72\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8589ms 0.6098ms 1.6399 KOps/s 1.6187 KOps/s $\color{#35bf28}+1.31\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 6.4849ms 4.7050ms 212.5413 Ops/s 233.8966 Ops/s $\textbf{\color{#d91a1a}-9.13\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 10.1046ms 2.4171ms 413.7177 Ops/s 441.7822 Ops/s $\textbf{\color{#d91a1a}-6.35\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 6.2063ms 1.3391ms 746.7575 Ops/s 792.5968 Ops/s $\textbf{\color{#d91a1a}-5.78\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.4959s 14.3944ms 69.4712 Ops/s 229.5421 Ops/s $\textbf{\color{#d91a1a}-69.73\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 5.8302ms 2.2598ms 442.5260 Ops/s 398.5036 Ops/s $\textbf{\color{#35bf28}+11.05\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 7.1816ms 1.4554ms 687.0929 Ops/s 807.6214 Ops/s $\textbf{\color{#d91a1a}-14.92\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 6.8640ms 4.9195ms 203.2710 Ops/s 221.7612 Ops/s $\textbf{\color{#d91a1a}-8.34\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 9.0909ms 2.5918ms 385.8270 Ops/s 415.1423 Ops/s $\textbf{\color{#d91a1a}-7.06\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 6.6563ms 1.5074ms 663.3878 Ops/s 681.5065 Ops/s $\color{#d91a1a}-2.66\%$

@github-actions GitHub Actions
Copy link

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 143. Improved: $\large\color{#35bf28}12$. Worsened: $\large\color{#d91a1a}14$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_simple 0.7068s 0.7030s 1.4225 Ops/s 1.4119 Ops/s $\color{#35bf28}+0.76\%$
test_transformed 1.0368s 0.9616s 1.0400 Ops/s 1.0424 Ops/s $\color{#d91a1a}-0.23\%$
test_serial 2.1563s 2.0777s 0.4813 Ops/s 0.4866 Ops/s $\color{#d91a1a}-1.08\%$
test_parallel 2.0462s 1.9861s 0.5035 Ops/s 0.5129 Ops/s $\color{#d91a1a}-1.83\%$
test_step_mdp_speed[True-True-True-True-True] 0.2478ms 38.7878μs 25.7813 KOps/s 27.1124 KOps/s $\color{#d91a1a}-4.91\%$
test_step_mdp_speed[True-True-True-True-False] 0.4125ms 22.2316μs 44.9811 KOps/s 45.1568 KOps/s $\color{#d91a1a}-0.39\%$
test_step_mdp_speed[True-True-True-False-True] 0.4170ms 21.0561μs 47.4921 KOps/s 50.0586 KOps/s $\textbf{\color{#d91a1a}-5.13\%}$
test_step_mdp_speed[True-True-True-False-False] 71.1110μs 12.2389μs 81.7070 KOps/s 83.2730 KOps/s $\color{#d91a1a}-1.88\%$
test_step_mdp_speed[True-True-False-True-True] 0.4247ms 41.6363μs 24.0175 KOps/s 25.6251 KOps/s $\textbf{\color{#d91a1a}-6.27\%}$
test_step_mdp_speed[True-True-False-True-False] 0.4083ms 25.1252μs 39.8006 KOps/s 40.7873 KOps/s $\color{#d91a1a}-2.42\%$
test_step_mdp_speed[True-True-False-False-True] 62.2210μs 23.8951μs 41.8496 KOps/s 44.2944 KOps/s $\textbf{\color{#d91a1a}-5.52\%}$
test_step_mdp_speed[True-True-False-False-False] 49.5700μs 14.9964μs 66.6827 KOps/s 69.7034 KOps/s $\color{#d91a1a}-4.33\%$
test_step_mdp_speed[True-False-True-True-True] 0.4367ms 44.5136μs 22.4650 KOps/s 23.7270 KOps/s $\textbf{\color{#d91a1a}-5.32\%}$
test_step_mdp_speed[True-False-True-True-False] 0.4091ms 27.3665μs 36.5410 KOps/s 36.9466 KOps/s $\color{#d91a1a}-1.10\%$
test_step_mdp_speed[True-False-True-False-True] 83.0810μs 23.5671μs 42.4321 KOps/s 42.4558 KOps/s $\color{#d91a1a}-0.06\%$
test_step_mdp_speed[True-False-True-False-False] 0.4193ms 14.8495μs 67.3424 KOps/s 69.4937 KOps/s $\color{#d91a1a}-3.10\%$
test_step_mdp_speed[True-False-False-True-True] 0.4301ms 46.0639μs 21.7090 KOps/s 22.4023 KOps/s $\color{#d91a1a}-3.10\%$
test_step_mdp_speed[True-False-False-True-False] 0.4090ms 29.5964μs 33.7879 KOps/s 33.6747 KOps/s $\color{#35bf28}+0.34\%$
test_step_mdp_speed[True-False-False-False-True] 61.6600μs 26.1289μs 38.2718 KOps/s 40.5974 KOps/s $\textbf{\color{#d91a1a}-5.73\%}$
test_step_mdp_speed[True-False-False-False-False] 0.4023ms 17.3108μs 57.7674 KOps/s 59.5955 KOps/s $\color{#d91a1a}-3.07\%$
test_step_mdp_speed[False-True-True-True-True] 0.4394ms 43.4561μs 23.0117 KOps/s 23.9288 KOps/s $\color{#d91a1a}-3.83\%$
test_step_mdp_speed[False-True-True-True-False] 69.7110μs 27.0947μs 36.9076 KOps/s 37.6749 KOps/s $\color{#d91a1a}-2.04\%$
test_step_mdp_speed[False-True-True-False-True] 0.4156ms 28.5699μs 35.0019 KOps/s 37.3116 KOps/s $\textbf{\color{#d91a1a}-6.19\%}$
test_step_mdp_speed[False-True-True-False-False] 0.3951ms 16.9817μs 58.8870 KOps/s 59.9437 KOps/s $\color{#d91a1a}-1.76\%$
test_step_mdp_speed[False-True-False-True-True] 0.4280ms 46.4862μs 21.5118 KOps/s 22.3335 KOps/s $\color{#d91a1a}-3.68\%$
test_step_mdp_speed[False-True-False-True-False] 0.4211ms 29.5474μs 33.8439 KOps/s 33.7024 KOps/s $\color{#35bf28}+0.42\%$
test_step_mdp_speed[False-True-False-False-True] 3.3764ms 31.1498μs 32.1029 KOps/s 33.5645 KOps/s $\color{#d91a1a}-4.35\%$
test_step_mdp_speed[False-True-False-False-False] 0.4162ms 19.3681μs 51.6313 KOps/s 51.3945 KOps/s $\color{#35bf28}+0.46\%$
test_step_mdp_speed[False-False-True-True-True] 0.4294ms 48.9995μs 20.4084 KOps/s 21.0970 KOps/s $\color{#d91a1a}-3.26\%$
test_step_mdp_speed[False-False-True-True-False] 0.4153ms 32.4147μs 30.8502 KOps/s 31.1211 KOps/s $\color{#d91a1a}-0.87\%$
test_step_mdp_speed[False-False-True-False-True] 0.1604ms 30.1848μs 33.1292 KOps/s 33.1887 KOps/s $\color{#d91a1a}-0.18\%$
test_step_mdp_speed[False-False-True-False-False] 0.4074ms 19.1401μs 52.2463 KOps/s 51.4658 KOps/s $\color{#35bf28}+1.52\%$
test_step_mdp_speed[False-False-False-True-True] 0.4283ms 50.4285μs 19.8301 KOps/s 20.3801 KOps/s $\color{#d91a1a}-2.70\%$
test_step_mdp_speed[False-False-False-True-False] 0.4419ms 34.7316μs 28.7923 KOps/s 28.9105 KOps/s $\color{#d91a1a}-0.41\%$
test_step_mdp_speed[False-False-False-False-True] 68.2210μs 32.5776μs 30.6959 KOps/s 31.1407 KOps/s $\color{#d91a1a}-1.43\%$
test_step_mdp_speed[False-False-False-False-False] 0.3995ms 21.9599μs 45.5376 KOps/s 46.3738 KOps/s $\color{#d91a1a}-1.80\%$
test_values[generalized_advantage_estimate-True-True] 23.3650ms 22.7567ms 43.9431 Ops/s 43.6696 Ops/s $\color{#35bf28}+0.63\%$
test_values[vec_generalized_advantage_estimate-True-True] 92.2377ms 2.7093ms 369.1012 Ops/s 341.9571 Ops/s $\textbf{\color{#35bf28}+7.94\%}$
test_values[td0_return_estimate-False-False] 83.0210μs 62.2690μs 16.0594 KOps/s 15.8046 KOps/s $\color{#35bf28}+1.61\%$
test_values[td1_return_estimate-False-False] 51.7011ms 50.7442ms 19.7067 Ops/s 19.6073 Ops/s $\color{#35bf28}+0.51\%$
test_values[vec_td1_return_estimate-False-False] 1.3574ms 1.0393ms 962.1476 Ops/s 961.3509 Ops/s $\color{#35bf28}+0.08\%$
test_values[td_lambda_return_estimate-True-False] 82.4248ms 81.2700ms 12.3047 Ops/s 12.3225 Ops/s $\color{#d91a1a}-0.14\%$
test_values[vec_td_lambda_return_estimate-True-False] 1.2950ms 1.0301ms 970.7647 Ops/s 961.8109 Ops/s $\color{#35bf28}+0.93\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 23.1232ms 22.5369ms 44.3716 Ops/s 43.9661 Ops/s $\color{#35bf28}+0.92\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 1.0045ms 0.7062ms 1.4160 KOps/s 1.4052 KOps/s $\color{#35bf28}+0.77\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.7808ms 0.6223ms 1.6069 KOps/s 1.5838 KOps/s $\color{#35bf28}+1.46\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.8173ms 1.4352ms 696.7625 Ops/s 694.0106 Ops/s $\color{#35bf28}+0.40\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 1.0257ms 0.6381ms 1.5671 KOps/s 1.5550 KOps/s $\color{#35bf28}+0.78\%$
test_dqn_speed[False-None] 7.1953ms 1.3123ms 762.0433 Ops/s 750.9374 Ops/s $\color{#35bf28}+1.48\%$
test_dqn_speed[False-backward] 2.3015ms 1.8478ms 541.1980 Ops/s 538.5313 Ops/s $\color{#35bf28}+0.50\%$
test_dqn_speed[True-None] 0.7455ms 0.5639ms 1.7735 KOps/s 1.7910 KOps/s $\color{#d91a1a}-0.98\%$
test_dqn_speed[True-backward] 1.2055ms 1.0037ms 996.3424 Ops/s 818.5180 Ops/s $\textbf{\color{#35bf28}+21.73\%}$
test_dqn_speed[reduce-overhead-None] 0.9353ms 0.5569ms 1.7956 KOps/s 1.7898 KOps/s $\color{#35bf28}+0.32\%$
test_dqn_speed[reduce-overhead-backward] 1.4110ms 1.0035ms 996.4887 Ops/s 990.0940 Ops/s $\color{#35bf28}+0.65\%$
test_ddpg_speed[False-None] 2.8648ms 2.6648ms 375.2568 Ops/s 370.4708 Ops/s $\color{#35bf28}+1.29\%$
test_ddpg_speed[False-backward] 4.0265ms 3.8719ms 258.2689 Ops/s 254.5345 Ops/s $\color{#35bf28}+1.47\%$
test_ddpg_speed[True-None] 1.6498ms 1.2306ms 812.6311 Ops/s 811.0116 Ops/s $\color{#35bf28}+0.20\%$
test_ddpg_speed[True-backward] 2.5970ms 2.2482ms 444.8070 Ops/s 414.9171 Ops/s $\textbf{\color{#35bf28}+7.20\%}$
test_ddpg_speed[reduce-overhead-None] 1.6526ms 1.2470ms 801.9457 Ops/s 798.7540 Ops/s $\color{#35bf28}+0.40\%$
test_ddpg_speed[reduce-overhead-backward] 2.4293ms 2.2305ms 448.3271 Ops/s 448.4789 Ops/s $\color{#d91a1a}-0.03\%$
test_sac_speed[False-None] 8.0713ms 7.4998ms 133.3367 Ops/s 130.6485 Ops/s $\color{#35bf28}+2.06\%$
test_sac_speed[False-backward] 11.4568ms 10.7650ms 92.8934 Ops/s 91.7309 Ops/s $\color{#35bf28}+1.27\%$
test_sac_speed[True-None] 2.4658ms 2.0310ms 492.3721 Ops/s 492.8106 Ops/s $\color{#d91a1a}-0.09\%$
test_sac_speed[True-backward] 4.2347ms 3.9553ms 252.8225 Ops/s 216.5254 Ops/s $\textbf{\color{#35bf28}+16.76\%}$
test_sac_speed[reduce-overhead-None] 2.4371ms 2.0150ms 496.2821 Ops/s 487.0622 Ops/s $\color{#35bf28}+1.89\%$
test_sac_speed[reduce-overhead-backward] 4.2304ms 3.9708ms 251.8378 Ops/s 248.1786 Ops/s $\color{#35bf28}+1.47\%$
test_redq_speed[False-None] 14.9665ms 10.3212ms 96.8880 Ops/s 67.0156 Ops/s $\textbf{\color{#35bf28}+44.58\%}$
test_redq_speed[False-backward] 18.4747ms 17.5494ms 56.9818 Ops/s 56.9199 Ops/s $\color{#35bf28}+0.11\%$
test_redq_speed[True-None] 3.9939ms 3.6666ms 272.7320 Ops/s 265.5267 Ops/s $\color{#35bf28}+2.71\%$
test_redq_speed[True-backward] 9.1971ms 8.6545ms 115.5472 Ops/s 114.0435 Ops/s $\color{#35bf28}+1.32\%$
test_redq_speed[reduce-overhead-None] 3.9770ms 3.5386ms 282.5938 Ops/s 279.6662 Ops/s $\color{#35bf28}+1.05\%$
test_redq_speed[reduce-overhead-backward] 9.2860ms 8.7978ms 113.6644 Ops/s 114.8407 Ops/s $\color{#d91a1a}-1.02\%$
test_redq_deprec_speed[False-None] 11.0892ms 10.6814ms 93.6210 Ops/s 92.2399 Ops/s $\color{#35bf28}+1.50\%$
test_redq_deprec_speed[False-backward] 16.7716ms 15.7083ms 63.6606 Ops/s 64.0354 Ops/s $\color{#d91a1a}-0.59\%$
test_redq_deprec_speed[True-None] 3.6650ms 3.2664ms 306.1504 Ops/s 290.3256 Ops/s $\textbf{\color{#35bf28}+5.45\%}$
test_redq_deprec_speed[True-backward] 7.5691ms 7.1693ms 139.4831 Ops/s 131.9600 Ops/s $\textbf{\color{#35bf28}+5.70\%}$
test_redq_deprec_speed[reduce-overhead-None] 3.5559ms 3.2138ms 311.1584 Ops/s 299.2747 Ops/s $\color{#35bf28}+3.97\%$
test_redq_deprec_speed[reduce-overhead-backward] 7.5450ms 7.2073ms 138.7474 Ops/s 132.3718 Ops/s $\color{#35bf28}+4.82\%$
test_td3_speed[False-None] 7.6755ms 7.4649ms 133.9596 Ops/s 131.0657 Ops/s $\color{#35bf28}+2.21\%$
test_td3_speed[False-backward] 10.8784ms 10.3274ms 96.8294 Ops/s 94.5918 Ops/s $\color{#35bf28}+2.37\%$
test_td3_speed[True-None] 1.9581ms 1.9015ms 525.9109 Ops/s 523.2225 Ops/s $\color{#35bf28}+0.51\%$
test_td3_speed[True-backward] 4.0399ms 3.7476ms 266.8404 Ops/s 262.0108 Ops/s $\color{#35bf28}+1.84\%$
test_td3_speed[reduce-overhead-None] 1.9735ms 1.9100ms 523.5484 Ops/s 516.9515 Ops/s $\color{#35bf28}+1.28\%$
test_td3_speed[reduce-overhead-backward] 3.9456ms 3.7313ms 268.0055 Ops/s 263.0689 Ops/s $\color{#35bf28}+1.88\%$
test_cql_speed[False-None] 28.5028ms 25.4497ms 39.2932 Ops/s 39.6355 Ops/s $\color{#d91a1a}-0.86\%$
test_cql_speed[False-backward] 39.3090ms 35.4763ms 28.1878 Ops/s 28.5179 Ops/s $\color{#d91a1a}-1.16\%$
test_cql_speed[True-None] 11.5802ms 11.0970ms 90.1147 Ops/s 90.3501 Ops/s $\color{#d91a1a}-0.26\%$
test_cql_speed[True-backward] 17.5044ms 17.0290ms 58.7232 Ops/s 57.9411 Ops/s $\color{#35bf28}+1.35\%$
test_cql_speed[reduce-overhead-None] 12.1116ms 11.2835ms 88.6252 Ops/s 89.7896 Ops/s $\color{#d91a1a}-1.30\%$
test_cql_speed[reduce-overhead-backward] 17.6640ms 17.1310ms 58.3738 Ops/s 57.9490 Ops/s $\color{#35bf28}+0.73\%$
test_a2c_speed[False-None] 5.8261ms 5.3324ms 187.5344 Ops/s 187.4332 Ops/s $\color{#35bf28}+0.05\%$
test_a2c_speed[False-backward] 13.4403ms 11.8627ms 84.2977 Ops/s 84.2655 Ops/s $\color{#35bf28}+0.04\%$
test_a2c_speed[True-None] 3.4034ms 3.0809ms 324.5798 Ops/s 311.0078 Ops/s $\color{#35bf28}+4.36\%$
test_a2c_speed[True-backward] 8.9336ms 8.6425ms 115.7066 Ops/s 113.3844 Ops/s $\color{#35bf28}+2.05\%$
test_a2c_speed[reduce-overhead-None] 3.3734ms 3.1237ms 320.1307 Ops/s 321.2709 Ops/s $\color{#d91a1a}-0.35\%$
test_a2c_speed[reduce-overhead-backward] 8.9105ms 8.5167ms 117.4160 Ops/s 116.5541 Ops/s $\color{#35bf28}+0.74\%$
test_ppo_speed[False-None] 7.7287ms 5.6518ms 176.9361 Ops/s 172.6901 Ops/s $\color{#35bf28}+2.46\%$
test_ppo_speed[False-backward] 13.4930ms 12.4902ms 80.0625 Ops/s 80.0048 Ops/s $\color{#35bf28}+0.07\%$
test_ppo_speed[True-None] 3.7196ms 3.5446ms 282.1193 Ops/s 281.3224 Ops/s $\color{#35bf28}+0.28\%$
test_ppo_speed[True-backward] 9.0984ms 8.4768ms 117.9691 Ops/s 118.9118 Ops/s $\color{#d91a1a}-0.79\%$
test_ppo_speed[reduce-overhead-None] 3.6349ms 3.4719ms 288.0286 Ops/s 288.3710 Ops/s $\color{#d91a1a}-0.12\%$
test_ppo_speed[reduce-overhead-backward] 8.5656ms 8.3302ms 120.0453 Ops/s 116.7669 Ops/s $\color{#35bf28}+2.81\%$
test_reinforce_speed[False-None] 4.8164ms 4.4261ms 225.9319 Ops/s 216.2007 Ops/s $\color{#35bf28}+4.50\%$
test_reinforce_speed[False-backward] 7.9475ms 7.3939ms 135.2469 Ops/s 133.0445 Ops/s $\color{#35bf28}+1.66\%$
test_reinforce_speed[True-None] 2.6289ms 2.2310ms 448.2317 Ops/s 442.8853 Ops/s $\color{#35bf28}+1.21\%$
test_reinforce_speed[True-backward] 7.6531ms 7.2225ms 138.4558 Ops/s 138.0526 Ops/s $\color{#35bf28}+0.29\%$
test_reinforce_speed[reduce-overhead-None] 2.6218ms 2.2385ms 446.7342 Ops/s 439.7482 Ops/s $\color{#35bf28}+1.59\%$
test_reinforce_speed[reduce-overhead-backward] 7.5101ms 7.1633ms 139.6006 Ops/s 139.2302 Ops/s $\color{#35bf28}+0.27\%$
test_iql_speed[False-None] 25.0115ms 20.2390ms 49.4095 Ops/s 49.6314 Ops/s $\color{#d91a1a}-0.45\%$
test_iql_speed[False-backward] 35.8064ms 30.6196ms 32.6588 Ops/s 32.6061 Ops/s $\color{#35bf28}+0.16\%$
test_iql_speed[True-None] 7.1971ms 6.8281ms 146.4535 Ops/s 139.7402 Ops/s $\color{#35bf28}+4.80\%$
test_iql_speed[True-backward] 17.0743ms 15.8781ms 62.9796 Ops/s 61.7473 Ops/s $\color{#35bf28}+2.00\%$
test_iql_speed[reduce-overhead-None] 7.5349ms 6.8435ms 146.1239 Ops/s 146.1049 Ops/s $\color{#35bf28}+0.01\%$
test_iql_speed[reduce-overhead-backward] 16.4439ms 15.7916ms 63.3247 Ops/s 62.9217 Ops/s $\color{#35bf28}+0.64\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 6.4844ms 6.1407ms 162.8488 Ops/s 162.4107 Ops/s $\color{#35bf28}+0.27\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.9781ms 0.3412ms 2.9304 KOps/s 3.5148 KOps/s $\textbf{\color{#d91a1a}-16.63\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6041ms 0.3204ms 3.1211 KOps/s 3.8431 KOps/s $\textbf{\color{#d91a1a}-18.79\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.3766ms 5.9169ms 169.0079 Ops/s 170.6466 Ops/s $\color{#d91a1a}-0.96\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.7759ms 0.3311ms 3.0204 KOps/s 3.4123 KOps/s $\textbf{\color{#d91a1a}-11.49\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6244ms 0.3153ms 3.1714 KOps/s 3.5435 KOps/s $\textbf{\color{#d91a1a}-10.50\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 1.6305ms 1.3532ms 738.9879 Ops/s 750.5371 Ops/s $\color{#d91a1a}-1.54\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 1.5839ms 1.3053ms 766.1256 Ops/s 784.3157 Ops/s $\color{#d91a1a}-2.32\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.3704ms 6.1016ms 163.8911 Ops/s 166.0011 Ops/s $\color{#d91a1a}-1.27\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.2866ms 0.4263ms 2.3456 KOps/s 2.3043 KOps/s $\color{#35bf28}+1.80\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.7774ms 0.3983ms 2.5104 KOps/s 2.3930 KOps/s $\color{#35bf28}+4.90\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 6.2703ms 6.0123ms 166.3261 Ops/s 169.6240 Ops/s $\color{#d91a1a}-1.94\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.9453ms 0.3035ms 3.2947 KOps/s 3.7006 KOps/s $\textbf{\color{#d91a1a}-10.97\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.5716ms 0.2860ms 3.4963 KOps/s 4.5677 KOps/s $\textbf{\color{#d91a1a}-23.46\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 9.1258ms 5.9084ms 169.2498 Ops/s 172.3526 Ops/s $\color{#d91a1a}-1.80\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.6317ms 0.3127ms 3.1979 KOps/s 3.0146 KOps/s $\textbf{\color{#35bf28}+6.08\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.5288ms 0.2985ms 3.3498 KOps/s 3.3674 KOps/s $\color{#d91a1a}-0.52\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.3394ms 6.0899ms 164.2071 Ops/s 167.3122 Ops/s $\color{#d91a1a}-1.86\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.6984ms 0.4217ms 2.3712 KOps/s 527.7710 Ops/s $\textbf{\color{#35bf28}+349.28\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 9.8143ms 0.4073ms 2.4551 KOps/s 2.2077 KOps/s $\textbf{\color{#35bf28}+11.21\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 6.9548ms 5.1744ms 193.2584 Ops/s 185.3580 Ops/s $\color{#35bf28}+4.26\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 6.9249ms 1.9458ms 513.9247 Ops/s 484.7695 Ops/s $\textbf{\color{#35bf28}+6.01\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 8.7875ms 1.2581ms 794.8525 Ops/s 845.6761 Ops/s $\textbf{\color{#d91a1a}-6.01\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.3957s 13.5874ms 73.5974 Ops/s 189.2881 Ops/s $\textbf{\color{#d91a1a}-61.12\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 9.0584ms 2.0044ms 498.8925 Ops/s 509.3968 Ops/s $\color{#d91a1a}-2.06\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 7.8346ms 1.2012ms 832.5160 Ops/s 789.6814 Ops/s $\textbf{\color{#35bf28}+5.42\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 7.3166ms 5.4196ms 184.5148 Ops/s 181.6590 Ops/s $\color{#35bf28}+1.57\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 8.9592ms 2.1335ms 468.7143 Ops/s 457.3391 Ops/s $\color{#35bf28}+2.49\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 7.0960ms 1.3697ms 730.0852 Ops/s 721.9556 Ops/s $\color{#35bf28}+1.13\%$

@vmoens vmoens merged commit baba52b into main Oct 21, 2024
70 of 75 checks passed
@vmoens vmoens deleted the fix-benchmarks branch October 21, 2024 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CI Has to do with CI setup (e.g. wheels & builds, tests...) CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vmoens @facebook-github-bot
点击 这是indexloc提供的php浏览器服务，不要输入任何密码和下载