When user tries to merge two Leads, the message appears
Deduplication customizing not maintained for transaction type *

First look into below customizing and see if the Lead transaction type has been defined here and also active .
Customer Relationship Management
->Transactions
->Settings for Leads
->Assign Classes for Deduplication

Please also enable Active checkbox in the customizing.
Component/View:
BT108S_LEA/Result
See Also:
SAP KBA 1874573 - Deduplication customizing not maintained for transaction type <transaction type name>