UK Supplier not registered for VAT in SAP ByDesign

We have a supplier not registered for VAT so he doesn't charge us VAT on his invoice and there is no references to VAT registration in their master data. However, when we raise a purchase order or a new invoice, SAP ByDesign automatically adds VAT, which has lead to us overpaying them.

I raised a ticket with SAP to point out this is an incorrect system behaviour and I was asked to prove it is a legal requirement in the UK not to charge VAT if you are not VAT registered - or to claim back VAT you should not have been charged in the first place!

Has anyone else come across the issue and how did you solve the issue? I was looking at some tax configuration in External Tax Determination for Goods and Services but could not see how to add a specific supplier to exclude them from VAT calculations