Referring to FIG. 27, alternative to completely removing the sacrificial layers 205 of the fins 204 at operation 522, the method 500 may partially remove the sacrificial layers 205 of the fins 204 at operation 722 as depicted in FIGS. 27, 28A and 28B. The remaining portions of the sacrificial layers 205 are disposed over the top and bottom surfaces of the channel layers 207 (except the top surface of the topmost channel layer 207) and the top surfaces of the base fins 204B. Thereafter, the method 500 forms the capping layer 216 over the channel layers 207 and the base fins 204B as shown in FIGS. 29A and 29B. The remaining portions of the sacrificial layers 205 and the capping layer 216 are collectively referred to as the Vt modulation layer 218. In other words, the Vt modulation layer 218 includes the remaining portions of the sacrificial layer 205 and the capping layer 216 wrapping around the sacrificial layer 205. The method 500 may then proceeds to operation 528 to form gate stacks as discussed in detail above.