If you disable this option, FontAwesome won't be loaded. This issue could be related to caching. Please try clearing your caches and check again.
Additionally, please check and make sure you have not made any customizations.
In case you want to say thank you! 🙂 We'd really appreciate if you leave a good review on the plugin page. This is the best way to say thank you to this project and the support team.