Performace Impact due to unreferenced Supplier Master Data

Hi

We have uploaded approx 200 thousand distinct Supplier master records. Only 30 thousand Suppliers are referenced in PO and Invoice,

Is there a performance impact by retaining the unreferenced Suppliers in the master data?

Regards

Neel