A method of cross-component residual prediction for video data comprising two or more components is disclosed. First prediction data and second prediction data for a first component and a second component of a current block are received respectively. One or more parameters of a cross-component function are derived based on the first prediction data and the second prediction data. The cross-component function is related to the first component and the second component with the first component as an input of the cross-component function and the second component as an output of the cross-component function. A residual predictor is derived for second residuals of the second component using the cross-component function with first reconstructed residuals of the first component as the input of the cross-component function. The predicted difference between the second residuals of the second component and the residual predictor is encoded or decoded.
