Wyner-Ziv Coding for Compression of Large Multimodal Models

The compression of encoded sources in a large multimodal model (LMM) can be theoretically analyzed within Wyner-Ziv coding to limit communication overhead, as the incorporation of multimodalities at the decoder is highly analogous to combining side information with a source. In this paper, after dividing the coding framework of LMM into physical and semantic levels, we study fundamental information-combining approaches within Wyner-Ziv scheme in terms of performance-complexity tradeoff, namely incorporation of sources (i) at the beginning (for best performance) and (ii) at the later layers (for fast inference) of a decoder and examine them in terms of rate-distortion function and semantic efficiency. Precisely, the theoretical closed-form rate-distortion functions of these two methods are derived to understand them better. Then, a novel compression algorithm based on indexing is developed to investigate the resultant semantic similarity of these combining approaches. The results indicate that the compression rate for the fast inference is disturbed less for bad (noisy/low throughput) channels with respect to the best performance case, and the semantic similarity can be moderately preserved under certain circumstances. Additionally, the performance drop is negligible after some compression ratios for both approaches.