Devices, systems and methods for sub-block based prediction are described. In a representative aspect, a method for video processing includes partitioning a first component of a current video block into a first set of sub-blocks and partitioning a second component of the current video block into a second set of sub-blocks. A sub-block of the second component corresponds to one or more sub-blocks of the first component. The method also includes deriving, based on a color format of the current video block, motion vectors for a sub-block of the second component based on motion vectors for one or more corresponding sub-blocks of the first color component.