Abstract: Based on the Optisystem platform, a 2.88Tb/s (40-Gbaud/ch) DWDM system with 150GHz channel spacing is built for high spectral efficiency fiber optical communication systems. The 8-ary modulation formats adopted here include two types. One of them is 8APSK (8-level amplitude- and differential phase-shift keying), the other is D8PSK (differential 8-level phase-shift keying). The simulation results show that when the probe channel is 8APSK modulated, larger amplitude modulation index corresponds to larger optimal input average optical power. When the input average optical power is optimal, the impact of four-wave mixing (FWM) noise on system performance is not strong whether the modulation format is 8APSK or D8PSK, but when it exceeds the optimal value, the impact increases abruptly with its value. 8APSK with optimized amplitude modulation index apparently outperforms D8PSK for our discussed DWDM system, as is mainly attributed to better tolerance of total nonlinearities for the former.

Key words: D8PSK