Coupled Transceiver-Fiber Nonlinearity Compensation Based on Machine Learning for Probabilistic Shaping System

In this article, we experimentally demonstrate the combined benefit of artificial neural network-based nonlinearity compensation and probabilistic shaping for the first time. We demonstrate that the scheme not only compensates for transceiver's nonlinearity, enabling the full benefits of shaping to be achieved, but also the combined effects of transceiver and fiber propagation nonlinearities. The performance of the proposed artificial neural network is demonstrated at 28 Gbaud for both 64-QAM and 256-QAM probabilistically shaped systems and compared to that of uniformly distributed constellations. Our experimental results demonstrate: the expected performance gains for shaping alone; an additional SNR performance gain up to 1 dB in the linear region; an additional mutual information gain of 0.2 bits per channel use in the constellation-entropy limited region. In the presence of coupled transceiver and fiber-induced nonlinearities, an additional mutual information enhancement of ~0.13 bits/symbol is experimentally observed for a fiber link of up to 500 km with the aid of the proposed artificial neural network.