International Journal of Science, Engineering and TechnologyAuthors: Abhirup Guha
Abstract: The rapid deployment of 5G networks has introduced complex security challenges that span both physical-layer signal integrity and higher-layer protocol semantics, necessitating unified anomaly detection frameworks capable of addressing multi-vector attacks. We propose a cross-layer anomaly detection system that integrates radio-frequency (RF) signal distortions with protocol-level behavioral patterns to identify threats such as downgrade exploits, data sniffing, and device fingerprinting. The proposed method employs a hybrid 1D-CNN-LSTM architecture, where the 1D-CNN processes cyclostationary features extracted from in-phase/quadrature (I/Q) samples, while the LSTM analyzes graph-based embeddings of 5G protocol messages generated by a Graph Attention Network (GAT). These heterogeneous features are fused through a cross-attention mechanism, enabling real-time anomaly classification without specialized hardware. Moreover, the system interfaces with the 5G Core’s Security Edge Protection Proxy (SEPP) and Access and Mobility Management Function (AMF) to trigger conditional re-authentication upon detecting anomalies, thereby preserving session continuity for legitimate devices. The architecture achieves sub-millisecond latency by leveraging dilated residual blocks in the 1D-CNN and peephole connections in the LSTM, trained with a contrastive loss function to improve discriminative power. Experimental results demonstrate significant improvements over standalone SDR-based or protocol-centric detectors, particularly in scenarios involving coordinated RF and protocol-level attacks. This work bridges the gap between physical-layer signal analysis and network intrusion detection, offering a scalable solution for securing next-generation wireless systems.
DOI: https://doi.org/10.5281/zenodo.16910463
