You can use the company search API to check either a company with that VAT exist or not and if it exist you can do the next step accordingly by preventing user to submit the form.
here is the curl for that just replace the vat_value with value you want to check :
You can use ajax in jquery to hit api on the form submit and if in response you get the result greater then 0 then there is records with that vat and you will not alow user to submit form and if these are 0 then user can submit that form