In the Geneva meeting held in Feb. 2009, a proposal with the title “Video Coding Using Extended Block Sizes” was adopted by KTA, where the MB size is extended up to 64×64 and the motion partitions are scaled accordingly. At the same time, a 2D order-16 transform was also proposed for transforming the residual blocks with the size larger than or equal to 16×16. The transformation matrix of the proposed 2D order-16 transform is given as below, which is obtained by scaling the transformation matrix of 2D order-16 DCT by the factor 128 and rounding, and is non-orthogonal.

  Non-orthogonality will inevitably introduce transform error. Before analyzing the transform error quantitatively, let’s recall two properties of orthogonal transforms. Firstly, signals can be reconstructed perfectly if no quantization is performed in the transform domain. Secondly, if quantization is performed in the transform domain, the average variance (or energy) of the reconstruction error equals that of the quantization error.

  When it comes to non-orthogonal transform, the two properties are not held any more. First of all, we discuss the average variance of the reconstruction error without quantization in the transform domain. Suppose the vector input to the transform is a sample of a 1D zero-mean uni-variance source with length 16. Let σ_r0² denote the average variance of the reconstruction error. It can be proved that the upper bound of σ_r0^2  is expressed as the following formula,

where N is equal to 16, σ_x² is the variance of the input, and M is calculated by (T₁₆^TT₁₆-I)^T(T₁₆^TT₁₆-I). Then, we take the quantization into consideration and it can be proved that the relationship of the average variance of the reconstruction error, σ_r², and the quantization error, σ_q², can be expressed as below. All the detailed deductions can be found here.

 σ_r²=σ_q²+σ_r0²

  Clearly, with non-orthogonal transform, the average variance of the reconstruction error σ_r² is always σ_r0² larger than the average variance of the quantization error σ_q², no matter the transform coefficients are quantized or not, whereas with an orthogonal transform, σ_q² is equal to σ_r².

  For the case of the proposed non-orthogonal transform, the upper bound of σ_r0² is equal to 0.00024σ_x². Note that this figure represents the transform error for 1D input and when extended to 2D, σ_r0² will significantly increase, because the 1st dimension transform increases the magnitudes of the input and σ_x²Â for the 2nd dimension transform is 256 times larger. Furthermore, in case the quantization stepsize is very small, e.g., QP<6, and the energy of the prediction error is very large, the transform error may dominate the reconstruction error.

Permanent Link: Transform Error Introduced by Non-orthogonality