+
Skip to main content
Springer Nature Link
Log in

Trajectory tracking control for robotic manipulator with disturbances: a double-Q reinforcement learning method

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

This study uses a reinforcement learning (RL) algorithm to address the trajectory tracking control problem for a robotic manipulator subject to disturbances. A disturbance observer is developed to estimate and counteract external disturbances and model inaccuracies, thereby enhancing the manipulator’s control precision and disturbance rejection capability. A tracking controller is devised to improve tracking performance while maintaining control costs by leveraging the double Q-learning algorithm within reinforcement learning. Utilizing double Q-learning mitigates the issue of Q value overestimation encountered in traditional Q-learning approaches. This method significantly improves the robustness and adaptive ability of the control strategy by introducing a double Q network structure. It provides a new solution for accurate trajectory tracking of the robotic manipulator in unknown and changing environments. At the same time, the robotic manipulator can learn the optimal control strategy more quickly in the face of external disturbance and system uncertainty to achieve better trajectory tracking performance. Simulation and experiment results affirm the efficacy of the proposed control strategy, demonstrating superior trajectory tracking performance and disturbance attenuation capabilities for the manipulator system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Data availability

Data sharing not applicable to this article as no data sets were generated or analysed during the current study.

References

  1. Xiao B, Yin S, Kaynak O (2016) Tracking control of robotic manipulators with uncertain kinematics and dynamics. IEEE Trans Ind Electron 63(10):6439–6449. https://doi.org/10.1109/TIE.2016.2569068

    Article  Google Scholar 

  2. Roveda L, Pedrocchi N, Beschi M, Tosatti LM (2018) High-accuracy robotized industrial assembly task control schema with force overshoots avoidance. Control Eng Pract 71:142–153. https://doi.org/10.1016/j.conengprac.2017.10.015

    Article  Google Scholar 

  3. Sun W, Wu Y, Lv X (2022) Adaptive neural network control for full-state constrained robotic manipulator with actuator saturation and time-varying delays. IEEE Trans Neural Netw Learn Syst 33(8):3331–3342. https://doi.org/10.1109/TNNLS.2021.3051946

    Article  MathSciNet  Google Scholar 

  4. Incremona GP, Ferrara A, Magni L (2017) MPC for robot manipulators with integral sliding modes generation. IEEE/ASME Trans Mechatronics 22(3):1299–1307. https://doi.org/10.1109/TMECH.2017.2674701

    Article  Google Scholar 

  5. Cao S, Sun L, Jiang J, Zuo Z (2023) Reinforcement learning-based fixed-time trajectory tracking control for uncertain robotic manipulators with input saturation. IEEE Trans Neural Netw Learn Syst 34(8):4584–4595. https://doi.org/10.1109/TNNLS.2021.3116713

    Article  MathSciNet  Google Scholar 

  6. Bascetta L, Rocco P (2010) Revising the robust-control design for rigid robot manipulators. IEEE Trans Robot 26(1):180–187. https://doi.org/10.1109/TRO.2009.2033957

    Article  Google Scholar 

  7. Chen H, Zong G, Zhao X, Gao F, Shi K (2024) Secure filter design of fuzzy switched CPSs with mismatched modes and application: A multidomain event-triggered strategy. IEEE Trans Ind Informat 19(10):10034–10044. https://doi.org/10.1109/TII.2022.3232768

    Article  Google Scholar 

  8. Wu X, She J, Yu L, Dong H, Zhang W (2021) Contour tracking control of networked motion control system using improved equivalent-input-disturbance approach. IEEE Trans Ind Electron 68(6):5155–5165. https://doi.org/10.1109/TIE.2020.2992017

    Article  Google Scholar 

  9. Xie H, Zhang J, Jing Y, Dimirovski GM, Chen J (2024) Self-adjustable performance-based adaptive tracking control of uncertain nonlinear systems. IEEE Trans Autom Sci Eng. https://doi.org/10.1109/TASE.2024.3353380

    Article  Google Scholar 

  10. Deng C, Wen C, Wang W, Li X, Yue D (2023) Distributed adaptive tracking control for high-order nonlinear multiagent systems over event-triggered communication. IEEE Trans Autom Control 68(2):1176–1183. https://doi.org/10.1109/TAC.2022.3148384

    Article  MathSciNet  Google Scholar 

  11. Sun W, Wu Y, Wang L (2019) Trajectory tracking of constrained robotic systems via a hybrid control strategy. Neurocomputing 330:188–195. https://doi.org/10.1016/j.neucom.2018.11.008

    Article  Google Scholar 

  12. Liu K, Wang R, Zheng S, Dong S, Sun G (2022) Fixed-time disturbance observer-based robust fault-tolerant tracking control for uncertain quadrotor UAV subject to input delay. Nonlinear Dyn 107:2363–2390. https://doi.org/10.1007/s11071-021-07080-0

    Article  Google Scholar 

  13. Cui G, Yang W, Yu J (2021) Neural network-based finite-time adaptive tracking control of nonstrict-feedback nonlinear systems with actuator failures. Inf Sci 545:298–311. https://doi.org/10.1016/j.ins.2020.08.024

    Article  MathSciNet  Google Scholar 

  14. Zhao Z, Cao D, Yang J, Wang H (2020) High-order sliding mode observer-based trajectory tracking control for a quadrotor UAV with uncertain dynamics. Nonlinear Dyn 102:2583–2596. https://doi.org/10.1007/s11071-020-06050-2

    Article  Google Scholar 

  15. Xiao B, Yin S (2019) Exponential tracking control of robotic manipulators with uncertain dynamics and kinematics. IEEE Trans Ind Informat 15(2):689–698. https://doi.org/10.1109/TII.2018.2809514

    Article  Google Scholar 

  16. Wen G, Chen CLP, Ge SS, Yang H, Liu X (2019) Optimized adaptive nonlinear tracking control using actor-critic reinforcement learning strategy. IEEE Trans Ind Informat 15(9):4969–4977. https://doi.org/10.1109/TII.2019.2894282

    Article  Google Scholar 

  17. Ran M, Li J, Xie L (2021) Reinforcement-learning-based disturbance rejection control for uncertain nonlinear systems. IEEE Trans Cybern 52(9):9621–9633. https://doi.org/10.1109/TCYB.2021.3060736

    Article  Google Scholar 

  18. Hu Y, Wang W, Liu H, Liu L (2020) Reinforcement learning tracking control for robotic manipulator with kernel-based dynamic model. IEEE Trans Neural Netw Learn Syst 31(9):3570–3578. https://doi.org/10.1109/TNNLS.2019.2945019

    Article  MathSciNet  Google Scholar 

  19. Zhao S, Wang J, Xu H, Wang B (2023) Composite observer-based optimal attitude-tracking control with reinforcement learning for hypersonic vehicles. IEEE Trans Cybern 53(2):913–926. https://doi.org/10.1109/TCYB.2022.3192871

    Article  Google Scholar 

  20. Chen C, Modares H, Xie K, Lewis FL, Wan Y, Xie S (2019) Reinforcement learning-based adaptive optimal exponential tracking control of linear systems with unknown dynamics. IEEE Trans Autom Control 64(11):4423–4438. https://doi.org/10.1109/TAC.2019.2905215

    Article  MathSciNet  Google Scholar 

  21. Chen X, Sun W, Gao X, Li Y (2024) Reinforcement learning-based event-triggered optimal control for unknown nonlinear systems with input delay. Int J Robust Nonlinear Control 34(7):4844–4863. https://doi.org/10.1002/rnc.7236

    Article  MathSciNet  Google Scholar 

  22. Panahi FH, Panahi FH, Ohtsuki T (2023) A reinforcement learning-based fire warning and suppression system using unmanned aerial vehicles. IEEE Trans Instrum Meas 72:1–16. https://doi.org/10.1109/TIM.2022.3227558

    Article  Google Scholar 

  23. Pataro IML, Cunha R, Gil JD, Guzmán JL, Berenguel M, Lemos JM (2023) Optimal model-free adaptive control based on reinforcement Q-learning for solar thermal collector fields. Eng Appl Artif Intell 126:106785. https://doi.org/10.1016/j.engappai.2023.106785

    Article  Google Scholar 

  24. Dabbaghjamanesh M, Moeini A, Kavousi-Fard A (2021) Reinforcement learning-based load forecasting of electric vehicle charging station using Q-learning technique. IEEE Trans Ind Informat 17(6):4229–4237. https://doi.org/10.1109/TII.2020.2990397

    Article  Google Scholar 

  25. Shafik RA, Yang S, Das A, Maeda-Nunez LA, Merrett GV, Al-Hashimi BM (2016) Learning transfer-based adaptive energy minimization in embedded systems. IEEE Trans Comput-Aided Design Integr Circuits Syst 35(6):877–890. https://doi.org/10.1109/TCAD.2015.2481867

    Article  Google Scholar 

  26. Van Hasselt H (2010) Double Q-learning. Proc Adv Neural In Process Syst

  27. Van Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double Q-learning. Proc AAAI Conf on Artif Intel. https://doi.org/10.1609/aaai.v30i1.10295

    Article  Google Scholar 

  28. Carlucho I, De Paula M, Acosta GG (2019) Double Q-PID algorithm for mobile robot control. Expert Syst Appl 137:292–307. https://doi.org/10.1016/j.eswa.2019.06.066

    Article  Google Scholar 

  29. Sun L, Liu Y (2020) Extended state observer augmented finite-time trajectory tracking control of uncertain mechanical systems. Mech Syst Signal Process 139:106374. https://doi.org/10.1016/j.ymssp.2019.106374

    Article  Google Scholar 

  30. Chen H, Zong G, Shen M, Gao F (2025) Finite-time resilient control of networked markov switched nonlinear systems: A relaxed design. IEEE Trans Syst Man Cybern. https://doi.org/10.1109/TSMC.2025.3526321

    Article  Google Scholar 

  31. Xie T, Xian B, Gu X, Hu J, Liu M (2024) Disturbance observer-based fixed-time tracking control for a tilt trirotor unmanned aerial vehicle. IEEE Trans Ind Electron 71(4):3894–3903. https://doi.org/10.1109/TIE.2023.3277090

    Article  Google Scholar 

  32. Xi R, Xiao X, Ma T, Yang Z (2022) Adaptive sliding mode disturbance observer based robust control for robot manipulators towards assembly assistance. IEEE Robot Automat Lett 7(3):6139–6146. https://doi.org/10.1109/LRA.2022.3164448

    Article  Google Scholar 

  33. Sciavicco L, Siciliano B (2012) Modelling and control of robot manipulators. Springer, Berlin, Germany

    Google Scholar 

  34. Shi D, Zhang J, Sun Z, Shen G, Xia Y (2021) Composite trajectory tracking control for robot manipulator with active disturbance rejection. Control Eng Pract 106:104670. https://doi.org/10.1016/j.conengprac.2020.104670

    Article  Google Scholar 

  35. Yang P, Su Y (2022) Proximate fixed-time prescribed performance tracking control of uncertain robot manipulators. IEEE/ASME Trans Mechatronics 27(5):3275–3285. https://doi.org/10.1109/TMECH.2021.3107150

    Article  Google Scholar 

  36. Sutton RS, Barto AG (2018) Reinforcement learning: An introduction, 2nd edn. MIT Press, London

    Google Scholar 

  37. Su Y, Zheng C, Mercorelli P (2020) Robust approximate fixed-time tracking control for uncertain robot manipulators. Mech Syst Signal Process 135:106379. https://doi.org/10.1016/j.ymssp.2019.106379

    Article  Google Scholar 

Download references

Funding

This work was supported in part by the Taishan Scholar Project of Shandong Province under Grant tsqn202211129, and in part by the National Natural Science Foundation of China under Grant 62073189 and Grant 62173207.

Author information

Authors and Affiliations

  1. Institute of Automation, Qufu Normal University, Qufu, 273165, China

    Dehai Yu, Weiwei Sun, Yongshu Li & Zhuangzhuang Luan

  2. College of Engineering, Qufu Normal University, Rizhao, 276826, China

    Weiwei Sun & Zhongcai Zhang

Authors
  1. Dehai Yu
    View author publications

    Search author on:PubMed Google Scholar

  2. Weiwei Sun
    View author publications

    Search author on:PubMed Google Scholar

  3. Yongshu Li
    View author publications

    Search author on:PubMed Google Scholar

  4. Zhuangzhuang Luan
    View author publications

    Search author on:PubMed Google Scholar

  5. Zhongcai Zhang
    View author publications

    Search author on:PubMed Google Scholar

Contributions

Dehai Yu: Investigation, Methodology, Writing-original draft. Weiwei Sun: Funding acquisition, Supervision, Writing-review & editing. Yongshu Li: Software, Validation. Zhuangzhuang Luan: Software, Validation. Zhongcai Zhang: Writing-review & editing.

Corresponding author

Correspondence to Weiwei Sun.

Ethics declarations

Conflicts of interest

The authors declare that they do not have any financial or nonfinancial conflict of interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, D., Sun, W., Li, Y. et al. Trajectory tracking control for robotic manipulator with disturbances: a double-Q reinforcement learning method. Appl Intell 55, 818 (2025). https://doi.org/10.1007/s10489-025-06655-3

Download citation

  • Accepted:

  • Published:

  • Version of record:

  • DOI: https://doi.org/10.1007/s10489-025-06655-3

Keywords

点击 这是indexloc提供的php浏览器服务，不要输入任何密码和下载