In the embodiments depicted in FIGS. 26A and 26B, the Vt modulation layer 218 covers the top surfaces, the bottom surfaces, and the sidewalls of the channel layers 207 and the base fins 204B. The channel layers 207 directly contact the Vt modulation layer 118 without any sacrificial layers 205 disposed therebetween. The Vt modulation layer 118 has a thickness t5, while the thickness of the channel layers 207 together with the Vt modulation layer 118 wrapped thereover is referred to as a2. In some embodiments, the thickness t5 is different from (e.g., less than) the thickness t4 and the thickness a2 is different from (e.g., less than) the thickness a. In the present embodiments, the thickness t5 equals to or substantially equals to the thickness t4, and the thickness a2 equals or substantially equals to a. In the present embodiments, the thickness t5 is about 10% to about 50% of the thickness a1. In the present embodiments, the thickness t5 is selected to effectively fine tune (or modulate) the Vt. If the thickness t5 is too thin (e.g., less than 10% of the thickness a1), the Vt modulation layer 218 may not be able to modulate the Vt effectively. On the other hand, if the thickness t5 is too thick (e.g., greater than 50% of the thickness a1), the opening 211 may be too narrow to form the various layers of the gate stack 620 in the subsequent processes.