switchport trunk encapsulation selects the protocol a Cisco IOS trunk port uses: 802.1Q (dot1q) or ISL. On switches supporting both, manually choosing encapsulation is required before manually setting trunk mode. section 12.3.2
Configure a trunk with 802.1Q
Attempting switchport mode trunk while encapsulation is Auto is rejected. Configure the protocol first, then set trunk mode: switchport trunk encapsulation dot1q followed by switchport mode trunk. Verify the result with show interfaces trunk, which should display 802.1q and trunking.
SW1(config-if)# switchport trunk encapsulation dot1q #1 SW1(config-if)# switchport mode trunk #2 SW1(config-if)# #1 Manually configures 802.1Q encapsulation #2 The command works this time.
Review the command sequence showing why encapsulation must be selected before trunk mode on switches supporting both protocols. section 12.3.2
