Abstract: A finite-length impulse response (FIR) filter whose tap coefficients are trained using a neural network for compensation of chromatic dispersion in digital coherent optical receivers is proposed. The frequency response is optimal when the square error is minimized. The pulse shaping filter limits the effective bandwidth of the signal. Therefore, the filter can be designed in a narrow frequency band. The simulation system is designed with MATLAB, and the BER performance of the filters in QPSK, 16 QAM and 64 QAM systems is analyzed through numerical simulation. Results show that compared with the time-domain equalization algorithm under the same conditions, when the number of taps is the same, the filter has a better compensation effect, and the increase in the number of taps will not cause the deterioration of the compensation effect. By designing the filter in a narrow frequency band, the number of taps can be reduced by more than 37.5% under the premise of the same compensation effect, which reduces the complexity of the filter hardware implementation and the filter delay.

Key words: optical communication, coherent reception, dispersion compensation, FIR filter, neural network