In order to speed up video encoding

a new fast algorithm is proposed

on P-frame mode decision which has high complexity in H.264/AVC. In the proposed algorithm

the check range of candidate inter modes are reduced gradually through the following two steps: the monotonicity check of Jmode and the selection of horizontal or vertical orientation based on sub-blocks’ motion vector (MV). Meanwhile

an intra mode checking method based on the best inter mode is proposed

which check intra modes selectively. Thus

it is avoided effectively that checking unnecessary modes. The experiment results show that

compared with exhaustive mode decision algorithm in the JM

the proposed algorithm reduced the encoding time about by 63.80% on average

while brought negligible performance degradation.