Optimum pulse amplitude modulation--I: Transmitter-receiver design and bounds from information theory

Intersymbol interference and additive noise are two common sources of distortion in data transmission systems. For pulse amplitude modulation (PAM) communication links, the combination of transmitter waveform and linear receiver that minimizes the mean-squared error arising from these sources is determined. An extension to include the effects of timing jitter is performed in a companion paper. Performance characteristics of the optimal PAM systems, showing the mean-squared error versus the signal-to-noise ratio, are determined explicitly for several examples. These characteristics are compared both with those of certain suboptimal systems and with the optimal performance theoretically attainable (OPTA), derived by combining Shannon's concepts of the capacity of a channel and the rate distortion function of a source. The optimal PAM systems are seen to perform very close to the OPTA for low signal-to-noise ratios. For high signal-to-noise ratios, however, the mean-squared error of optimal PAM systems decreases as the reciprocal of the signal-to-noise ratio, but the OPTA decreases more rapidly, except for band-limited channels. The performance of PAM systems can be improved at high signal-to-noise ratios by coding techniques. One such technique, called Shannon-Cantor coding, is discussed briefly.