Optimizing Network Routing and Resource Allocation using Deep Reinforcement Learning in Next-Generation Networks
Keywords:
Deep Reinforcement Learning, Network Routing, Resource Allocation, Next-Generation Networks, 5G, Software-Defined Networking, Network Optimization.Abstract
Next-Generation Networks (NGNs), including 5G and beyond, are characterized by unprecedented
scale, dynamism, and heterogeneity, posing significant challenges for traditional network management
paradigms. Static routing protocols and rule-based resource allocation mechanisms are increasingly
inadequate for handling the complex and time-varying traffic patterns inherent in these environments.
This paper proposes a novel framework based on Deep Reinforcement Learning (DRL) to jointly
optimize network routing and resource allocation. We formulate the problem as a Markov Decision
Process (MDP), where a centralized DRL agent learns optimal control policies by observing the global
network state, which includes link utilization, buffer occupancy, and quality of service (QoS) metrics.
The proposed model utilizes a Deep Q-Network (DQN) with experience replay and a target network
to ensure stable and efficient learning. The objective of the agent is to maximize a composite reward
function that balances network throughput, minimizes end-to-end delay, and ensures equitable resource
distribution. We conduct extensive simulations in a software-defined networking (SDN) environment
to evaluate the performance of our DRL-based approach. The results demonstrate that our framework
significantly outperforms conventional algorithms like Open Shortest Path First (OSPF) and a standard
Q-learning-based approach, achieving up to a 25% reduction in average network delay and a 15%
increase in overall throughput under high traffic loads. The findings confirm the potential of DRL to
enable intelligent, autonomous, and adaptive control in future communication networks.
References
A. Gupta and R. K. Jha, ”A Survey of 5G Network: Architecture and Emerging Technologies,” IEEE Access, vol. 3, pp.
-1232, 2015, doi: 10.1109/ACCESS.2015.2461602.
W. Saad, M. Bennis, and M. Chen, ”A Vision of 6G Wireless Systems: Applications, Trends, Technologies, and Open
Research Problems,” IEEE Network, vol. 34, no. 3, pp. 134-142, May/June 2020, doi: 10.1109/MNET.001.1900287.
N. Chiotellis, D. D. Tsiouranaki, and S. Papavassiliou, ”Dynamic Resource Management in 5G Networks: A Survey
of Emerging Trends and Techniques,” Journal of Network and Computer Applications, vol. 158, p. 102577, 2020, doi:
1016/j.jnca.2020.102577.
J. Moy, ”OSPF Version 2,” RFC 2328, April 1998, doi: 10.17487/RFC2328.
D. Awduche et al., ”RSVP-TE: Extensions to RSVP for LSP Tunnels,” RFC 3209, December 2001, doi: 10.17487/RFC3209.
F. A. Akyildiz, W. Lee, M. Vuran, and S. Mohanty, ”NeXt Generation/Dynamic Spectrum Access/Cognitive Radio Wireless
Networks: A Survey,” Computer Networks, vol. 50, no. 13, pp. 2127-2159, 2006, doi: 10.1016/j.comnet.2006.05.001.
B. Fortz and M. Thorup, ”Internet traffic engineering by optimizing OSPF weights,” in Proc. IEEE INFOCOM 2000, Tel
Aviv, Israel, 2000, pp. 519-528, doi: 10.1109/INFCOM.2000.832225.
R. Boutaba et al., ”A comprehensive survey on machine learning for networking: evolution, applications and research
opportunities,” Journal of Internet Services and Applications, vol. 9, no. 1, p. 16, 2018, doi: 10.1186/s13174-018-0087-2.
J. Yan, S. Liu, Z. Wang, and G. Li, ”A Survey of Deep Reinforcement Learning for Networking,” IEEE Communications
Surveys & Tutorials, vol. 22, no. 4, pp. 2234-2283, Fourthquarter 2020, doi: 10.1109/COMST.2020.3013444.
V. Mnih et al., ”Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529-533, 2015,
doi: 10.1038/nature14236.
Y. Lu, C. Huang, and Z. Zhang, ”Deep Reinforcement Learning for Intelligent Network Control: A Comprehensive Survey,”
IEEE Access, vol. 8, pp. 136931-136959, 2020, doi: 10.1109/ACCESS.2020.3011382.
”Software-Defined Networking: The New Norm for Networks,” ONF White Paper, 2012. [Online]. Available:
https://opennetworking.org/wp-content/uploads/2013/02/wp-sdn-newnorm.pdf
A. L. Buczak and E. Guven, ”A Survey of Data Mining and Machine Learning Methods for Cyber Security,” IEEE Com
munications Surveys & Tutorials, vol. 18, no. 2, pp. 1153-1176, Secondquarter 2016, doi: 10.1109/COMST.2015.2494502.
T. O’Shea and J. Hoydis, ”An Introduction to Deep Learning for the Physical Layer,” IEEE Transactions on Cognitive
Communications and Networking, vol. 3, no. 4, pp. 563-575, Dec. 2017, doi: 10.1109/TCCN.2017.2758370.
C. Fachkha, E. Bou-Harb, and C. Assi, ”A Data-Driven Approach for Profiling and Detecting BGP Anomalies,” in Proc.
IEEE/IFIP NOMS 2020, Budapest, Hungary, 2020, pp. 1-6, doi: 10.1109/NOMS47738.2020.9110410.
J. A. Boyan and M. L. Littman, ”Packet routing in dynamically changing networks: A reinforcement learning approach,”
in Advances in Neural Information Processing Systems, vol. 6, 1994, pp. 671-678.
S. P. Singh and M. P. G. Sutton, R. S. and Barto, ”Reinforcement Learning: An Introduction,” MIT Press, 2018.
Z. Mao, M. Alizadeh, and M. Abolhasan, ”A Survey of Learning-based Approaches for Network Performance Optimiza
tion,” ACM SIGCOMM Computer Communication Review, vol. 49, no. 2, pp. 52-65, 2019, doi: 10.1145/3331003.3331009.
G. Stampa et al., ”A Deep Reinforcement Learning Approach for Routing in SDN,” in Proc. 2017 IEEE International
Conference on Communications (ICC), Paris, France, 2017, pp. 1-6, doi: 10.1109/ICC.2017.7996582.
M. Al-Fares, A. Loukissas, and A. Vahdat, ”A scalable, commodity data center network architecture,” in Proc. ACM
SIGCOMM 2008, Seattle, WA, USA, 2008, pp. 63-74, doi: 10.1145/1402958.1402967.
F. B. Zafari, A. Gkelias, and K. K. Leung, ”A Survey of Intrusion Detection Systems in Wireless Sensor Networks,” IEEE
Communications Surveys & Tutorials, vol. 21, no. 3, pp. 2661-2713, thirdquarter 2019, doi: 10.1109/COMST.2019.2917088.
Y. Sun, M. Peng, and S. Mao, ”Deep Reinforcement Learning for Resource Allocation in V2X Communications,” IEEE
Transactions on Vehicular Technology, vol. 68, no. 8, pp. 7480-7492, Aug. 2019, doi: 10.1109/TVT.2019.2921963.
H. R. G. de Oliveira, A. L. S. dos Santos, and E. M. G. de F. Neto, ”Deep Reinforcement Learning for Dynamic Resource
Allocation in 5G Network Slicing,” in Proc. IEEE LATINCOM 2021, Santo Domingo, Dominican Republic, 2021, pp. 1-6,
doi: 10.1109/LATINCOM53176.2021.9622540.
A. A. Al-Hashedi, M. F. A. Rasid, and S. A. G. Othman, ”Deep reinforcement learning for resource allocation in network
slicing: A survey,” Computer Networks, vol. 201, p. 108544, 2021, doi: 10.1016/j.comnet.2021.108544.
R. Li, Z. Zhao, X. Chen, J. Palicot, and H. Zhang, ”Deep Reinforcement Learning for Resource Management in Network
Slicing,” IEEE Access, vol. 6, pp. 74429-74441, 2018, doi: 10.1109/ACCESS.2018.2881249.
Z. Liu, Y. Zhang, C. Yuen, and Z. Han, ”Joint Routing and Scheduling for Industrial IoT with Deep Rein
forcement Learning,” IEEE Transactions on Industrial Informatics, vol. 17, no. 8, pp. 5693-5702, Aug. 2021, doi:
1109/TII.2020.3023812.
X. Wang, X. Chen, and S. Cui, ”Deep Reinforcement Learning-based Joint Power and Channel Allocation for D2D Commu
nications,” IEEE Wireless Communications Letters, vol. 9, no. 4, pp. 524-528, April 2020, doi: 10.1109/LWC.2019.2961732.
Z. Zhou, Z. Tan, and T. Q. S. Quek, ”Deep Reinforcement Learning for Joint Optimization of Radio and Computational
Resources in 5G and Beyond,” IEEE Transactions on Wireless Communications, vol. 20, no. 2, pp. 1060-1074, Feb. 2021,
doi: 10.1109/TWC.2020.3031024.
S. O’Malley, C. L. Sterling, J. M. Rodriguez-Vidal, and A. C. Weaver, ”Graph-based Deep Reinforcement Learning for
Network Routing,” in Proc. 2023 IEEE International Conference on Network Protocols (ICNP), Reykjavik, Iceland, 2023,
pp. 1-12, doi: 10.1109/ICNP59223.2023.10299999.
L. Chen, Y. Liu, and B. Shou, ”A Multi-Agent Deep Reinforcement Learning Framework for Cooperative Traffic
Engineering in SDN,” IEEE Journal on Selected Areas in Communications, vol. 38, no. 10, pp. 2319-2331, Oct. 2020,
doi: 10.1109/JSAC.2020.3000407.
