Skip to main content
SpringerLink
Log in

HCDQN-ORA: a novel hybrid clustering and deep Q-network technique for dynamic user location-based optimal resource allocation in a fog environment

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

With the proliferation of the Internet of Things and smart devices, there exists an urge to address the critical computation demands of end users for several real-time applications. Fog computing (FC) targets to address the computation requirements nearer to the end user in almost real-time. Mainly, the fog environments (FE) exhibit characteristics of resource constraint and wide heterogeneity. In addition, the complex fog environment enforces the fog resources (FR) deployment either within fixed or distributed over random geographical locations. In the literature, there exist several solutions for the optimization of FR allocation. However, the location-dependent FR allocation has not been addressed thoroughly. Hence, this paper proposes a novel hybrid clustering-based deep Q-network for location-based optimal resource allocation (HCDQN-ORA) model. Here, the first objective is to minimize the search operation for identifying suitable FR and to meet the quality of service (QoS) demand using an enhanced fog resource clustering (E-FRC) algorithm. The second objective aims to achieve optimal FR allocation among the set of location-dependent FRs using reinforcement learning techniques. Problem formulation has been modelled using Markov decision process. A deep Q-network algorithm with two variants, namely enhanced optimal resource allocation using deep Q-network (EORA-DQN) and enhanced optimal resource allocation using deep Q-network with experience replay (EORA-DQN-ER), has been incorporated. To analyse the statically located and randomly distributed FR environment, the experimental results exhibit that, in the case of clustering of FRs, E-FRC is more efficient than conventional clustering algorithms. Next, the location-dependent optimal FR allocation problem incorporates the performance analysis based on the convergence of loss function and the parametric analysis based on batch size, respectively. Results exhibit that EORA-DQN performs better only for statically allotted resources. However, the proposed EORA-DQN-ER outperforms both types of FR with a success rate of 89.6% of optimal allocation with the batch size of 200.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Algorithm 2
Algorithm 3
Algorithm 4
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28

Similar content being viewed by others

Data availability

Not applicable.

References

  1. Liu Y, Yu H, Xie S, Zhang Y (2019) Deep reinforcement learning for offloading and resource allocation in vehicle edge computing and networks. IEEE Trans Veh Technol 68(11):11158–11168. https://doi.org/10.1109/TVT.2019.2935450

    Article  Google Scholar 

  2. Parwekar P (2011) From Internet of Things towards cloud of things. In: 2011 2nd International Conference on Computer and CommunicationTtechnology (ICCCT-2011), IEEE, pp 329–333. https://doi.org/10.1109/ICCCT.2011.6075156

  3. Wang S, Zhao T, Pang S (2020) Task scheduling algorithm based on improved firework algorithm in fog computing. IEEE Access 8:32385–32394. https://doi.org/10.1109/ACCESS.2020.2973758

    Article  Google Scholar 

  4. Shukla P, Pandey S, Agarwal D (2023) An efficient offloading technique using DQN for MEC-IoT Networks. In 2023 6th International Conference on Information Systems and Computer Networks (ISCON), IEEE, pp 1–7. https://doi.org/10.1109/ISCON57294.2023.10112053

  5. Ahlawat C, Krishnamurthi R (2023) Towards smart technologies with integration of the internet of things, cloud computing, and fog computing. Int J Netw Virtual Organ 29(1):73–124. https://doi.org/10.1504/IJNVO.2023.134304

    Article  Google Scholar 

  6. Parwekar P, Kumar P, Saxena M, Saxena S (2014) Public auditing: cloud data storage. In: 2014 5th International Conference—Confluence the Next Generation Information Technology Summit (Confluence), pp 169–173, IEEE. doi: https://doi.org/10.1109/CONFLUENCE.2014.6949366

  7. Tang Z, Zhou X, Zhang F, Jia W, Zhao W (2019) Migration modeling and learning algorithms for containers in fog computing. IEEE Trans Serv Comput 12(5):712–725. https://doi.org/10.1109/TSC.2018.2827070

    Article  Google Scholar 

  8. Yuan H, Bi J, Tan W, Zhou M, Li BH, Li J (2017) TTSA: an effective scheduling approach for delay bounded tasks in hybrid clouds. IEEE Trans Cybern 47(11):3658–3668. https://doi.org/10.1109/TCYB.2016.2574766

    Article  Google Scholar 

  9. Li G, Liu Y, Wu J, Lin D, Zhao S (2019) Methods of resource scheduling based on optimized fuzzy clustering in fog computing. Sensors 19(9):2122. https://doi.org/10.3390/s19092122

    Article  Google Scholar 

  10. Gazori P, Rahbari D, Nickray M (2020) Saving time and cost on the scheduling of fog-based IoT applications using deep reinforcement learning approach. Futur Gener Comput Syst 110:1098–1115. https://doi.org/10.1016/j.future.2019.09.060

    Article  Google Scholar 

  11. Samie F, Bauer L, Henkel J (2019) From cloud down to things: an overview of machine learning in Internet of Things. IEEE Internet Things J 6(3):4921–4934. https://doi.org/10.1109/JIOT.2019.2893866

    Article  Google Scholar 

  12. Mishra MK, Ray NK, Swain AR, Mund GB, Mishra BSP (2019) An adaptive model for resource selection and allocation in fog computing environment. Comput Electr Eng 77:217–229. https://doi.org/10.1016/j.compeleceng.2019.05.010

    Article  Google Scholar 

  13. Sowjanya K, Porwal A, Pandey S, Mishra PK (2022) TLBO-based resource allocation scheme in 5G H-CRAN In: 2022 14th International Conference on COMmunication Systems & NETworkS (COMSNETS), pp. 646–651 (IEEE). https://doi.org/10.1109/COMSNETS53615.2022.9668491

  14. Tran-Dang H, Bhardwaj S, Rahim T, Musaddiq A, Kim D-S (2022) Reinforcement learning based resource management for fog computing environment: literature review, challenges, and open issues. J Commun Netw 24(1):83–98. https://doi.org/10.23919/JCN.2021.000041

    Article  Google Scholar 

  15. Yin L, Luo J, Luo H (2018) Tasks scheduling and resource allocation in fog computing based on containers for smart manufacturing. IEEE Trans Industr Inform 14(10):4712–4721. https://doi.org/10.1109/TII.2018.2851241

    Article  Google Scholar 

  16. Vimal S, Khari M, Dey N, Crespo RG, Harold Robinson Y (2020) Enhanced resource allocation in mobile edge computing using reinforcement learning based MOACO algorithm for IIOT. Comput Commun 151:355–364. https://doi.org/10.1016/j.comcom.2020.01.018

    Article  Google Scholar 

  17. Liu X, Yu J, Wang J, Gao Y (2020) Resource allocation with edge computing in IoT Networks via machine learning. IEEE Internet Things J 7(4):3415–3426. https://doi.org/10.1109/JIOT.2020.2970110

    Article  Google Scholar 

  18. Chollet F (2021) Deep learning with python. Simon and Schuster

    Google Scholar 

  19. Sun Y, Peng M, Mao S (2019) Deep reinforcement learning-based mode selection and resource management for green fog radio access networks. IEEE Internet Things J 6(2):1960–1971. https://doi.org/10.1109/JIOT.2018.2871020

    Article  Google Scholar 

  20. Lakhan A, Mohammed MA, Obaid OI, Chakraborty C, Abdulkareem KH, Kadry S (2022) Efficient deep-reinforcement learning aware resource allocation in SDN-enabled fog paradigm. Autom Softw Eng 29(1):20. https://doi.org/10.1007/s10515-021-00318-6

    Article  Google Scholar 

  21. Singh J, Singh P, Hedabou M, Kumar N (2023) An efficient machine learning-based resource allocation scheme for SDN-enabled fog computing environment. IEEE Trans Veh Technol 72(6):8004–8017. https://doi.org/10.1109/TVT.2023.3242585

    Article  Google Scholar 

  22. Khumalo NN, Oyerinde OO, Mfupe L (2021) Reinforcement learning-based resource management model for fog radio access network architectures in 5G. IEEE Access 9:12706–12716. https://doi.org/10.1109/ACCESS.2021.3051695

    Article  Google Scholar 

  23. Qiu Y, Zhang H, Long K (2021) Computation offloading and wireless resource management for healthcare monitoring in fog-computing-based internet of medical things. IEEE Internet Things J 8(21):15875–15883. https://doi.org/10.1109/JIOT.2021.3066604

    Article  Google Scholar 

  24. Chen X, Zhou Y, Yang L, Lv L (2021) Hybrid fog/cloud computing resource allocation: joint consideration of limited communication resources and user credibility. Comput Commun 169:48–58. https://doi.org/10.1016/j.comcom.2021.01.026

    Article  Google Scholar 

  25. Baburao D, Pavankumar T, Prabhu CSR (2021) Load balancing in the fog nodes using particle swarm optimization-based enhanced dynamic resource allocation method. Appl Nanosci. https://doi.org/10.1007/s13204-021-01970-w

    Article  Google Scholar 

  26. Deng S et al (2020) Dynamical resource allocation in edge for trustable Internet-of-Things systems: a reinforcement learning method. IEEE Trans Industr Inform 16(9):6103–6113. https://doi.org/10.1109/TII.2020.2974875

    Article  Google Scholar 

  27. Liang H et al (2021) Reinforcement learning enabled dynamic resource allocation in the internet of vehicles. IEEE Trans Industr Inform 17(7):4957–4967. https://doi.org/10.1109/TII.2020.3019386

    Article  Google Scholar 

  28. Rahman GMS, Dang T, Ahmed M (2020) Deep reinforcement learning based computation offloading and resource allocation for low-latency fog radio access networks. Intell Converg Netw 1(3):243–257. https://doi.org/10.23919/ICN.2020.0020

    Article  Google Scholar 

  29. Talaat FM, Saraya MS, Saleh AI, Ali HA, Ali SH (2020) A load balancing and optimization strategy (LBOS) using reinforcement learning in fog computing environment. J Ambient Intell Humaniz Comput 11(11):4951–4966. https://doi.org/10.1007/s12652-020-01768-8

    Article  Google Scholar 

  30. Subbaraj S, Thiyagarajan R, Rengaraj M (2021) A smart fog computing based real-time secure resource allocation and scheduling strategy using multi-objective crow search algorithm. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-021-03354-y

    Article  Google Scholar 

  31. Shukla P, Pandey S, Hatwar P, Pant A (2023) FAT-ETO: fuzzy-AHP-TOPSIS-based efficient task offloading algorithm for scientific workflows in heterogeneous fog-cloud environment. Proc Natl Acad Sci India Sect A Phys Sci 93(2):339–353. https://doi.org/10.1007/s40010-023-00809-z

    Article  MathSciNet  Google Scholar 

  32. Shukla P, Pandey S (2023) MAA: multi-objective artificial algae algorithm for workflow scheduling in heterogeneous fog-cloud environment. J Supercomput 79(10):11218–11260. https://doi.org/10.1007/s11227-023-05110-9

    Article  Google Scholar 

  33. Naha RK, Garg S (2021) Multi-criteria–based dynamic user behaviour–aware resource allocation in fog computing. ACM Trans Internet Things 2(1):1–31. https://doi.org/10.1145/3423332

    Article  Google Scholar 

  34. Zhao J, Kong M, Li Q, Sun X (2020) Contract-based computing resource management via deep reinforcement learning in vehicular fog computing. IEEE Access 8:3319–3329. https://doi.org/10.1109/ACCESS.2019.2963051

    Article  Google Scholar 

  35. Wang M, Cui Y, Wang X, Xiao S, Jiang J (2018) Machine learning for networking: workflow, advances and opportunities. IEEE Netw 32(2):92–99. https://doi.org/10.1109/MNET.2017.1700200

    Article  Google Scholar 

  36. Golalipour K, Akbari E, Hamidi SS, Lee M, Enayatifar R (2021) From clustering to clustering ensemble selection: a review. Eng Appl Artif Intell 104:104388. https://doi.org/10.1016/j.engappai.2021.104388

    Article  Google Scholar 

  37. Sharma KK, Seal A (2020) Clustering analysis using an adaptive fused distance. Eng Appl Artif Intell 96:103928. https://doi.org/10.1016/j.engappai.2020.103928

    Article  Google Scholar 

  38. Sun Z et al (2019) An energy-efficient cross-layer-sensing clustering method based on intelligent fog computing in WSNs. IEEE Access 7:144165–144177. https://doi.org/10.1109/ACCESS.2019.2944858

    Article  Google Scholar 

  39. Bashir H, Lee S, Kim KH (2022) Resource allocation through logistic regression and multicriteria decision making method in IoT fog computing. Trans Emerg Telecommun Technol 33(2):e3824. https://doi.org/10.1002/ett.3824

    Article  Google Scholar 

  40. Erdogmus P, Kayaalp F (2020) Introductory chapter: clustering with nature-inspired optimization algorithms. Introduction to data science and machine learning. IntechOpen. https://doi.org/10.5772/intechopen.91678

    Chapter  Google Scholar 

  41. Saini G, Kaur HA (2014) A novel approach towards K-mean clustering algorithm with PSO. Int J Comput Sci Inf Technol 5(4):5978–5986

    Google Scholar 

  42. Shutaywi M, Kachouie NN (2021) Silhouette analysis for performance evaluation in machine learning with applications to clustering. Entropy 23(6):759. https://doi.org/10.3390/e23060759

    Article  MathSciNet  Google Scholar 

  43. Ahlawat C, Krishnamurthi R (2022) Q-learning with function Approximator for clustering based Optimal resource Allocation in fog environment. In: Proceedings of the 2022 14th International Conference on Contemporary Computing, New York, ACM. pp 127–135, https://doi.org/10.1145/3549206.3549230

  44. Borowska B (2017) Exponential inertia weight in particle swarm optimization. Springer, pp 265–275

    Google Scholar 

Download references

Funding

Not received any funding.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Jaypee Institute of Information Technology, Noida, 201309, India

    Chanchal Ahlawat & Rajalakshmi Krishnamurthi

Authors
  1. Chanchal Ahlawat
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rajalakshmi Krishnamurthi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

CA helped in design, acquisition of data, analysis, and interpretation of data and drafting. R helped in conceptualization of idea, revising the article, supervision, and project administration.

Corresponding author

Correspondence to Chanchal Ahlawat.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ahlawat, C., Krishnamurthi, R. HCDQN-ORA: a novel hybrid clustering and deep Q-network technique for dynamic user location-based optimal resource allocation in a fog environment. J Supercomput (2024). https://doi.org/10.1007/s11227-023-05832-w

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11227-023-05832-w

Keywords

Search

Navigation