Referring to FIG. 9b, if a size of a macroblock is 32×32, it is able to use one prediction mode for each 16×16 block. In particular, one most suitable prediction mode is selected from the 9 kinds of prediction modes shown in FIG. 8 for each 16×16 block and the selected prediction mode can be coded by 4×4 block unit. For instance, if an encoder selects a vertical prediction mode for a 1st 16×16 block, the encoder encodes the selected prediction mode and then transmits the encoded prediction mode in FIG. 9b, a decoder decodes the 0th block using the received vertical prediction mode. Subsequently, a reconstructed residual is added to a prediction sample. A 1st block is then decoded using a vertical prediction mode equal to a prediction mode of the 0th block and the reconstructed 0th block. In this manner, all blocks in the 1st 16×16 block are decoded. Subsequently, other 16×16 blocks are decoded in order of second, third and fourth.