To make the the check on the vendor balance mandatory for making a payment, you can write the validation as below:-
Create a message ID via transaction code SE91 for message class /EACC/MSG/ACC_COMP as you want to display to user.
Go to transaction GGB0 and create the validation step at line item level in Financial Accounting as below:-
BKPF-BSCHL = '25'.
Create a exit code using user exit and check that the vendor (LIFNR) is having at least balance >= the payment being made. There are function modules to fetch the vendor balance on a key date which will be posting date of the payment being made.
For this take the help of ABAPer and create a copy of the program RGGBR000 and code your check here and assign to the application area GBLR in view V_T80D via transaction code SM30.
Message created in step 1 as error message.
Please also check that this validation has active status (Status=1) at line complete document level (level 3) for your company code in transaction OB28.