In order to configure the TGW attachment in appliance mode you must perform this from the AWS CLI. Go to the IAM role and create an access key and record your secret key. Then configure your AWS CLI client and use these keys to be able to access the AWS CLI. When in the AWS CLI, enter the following:
aws ec2 modify-transit-gateway-vpc-attachment --options "ApplianceModeSupport=enable" --transit-gateway-attachment-id <YOUR TGW ATTACHMENT HERE> --region <YOUR REGION HERE>
Replace <xxx> with actuals.