Can I limit which taxonomies display in the Taxonomy Query builder iview?

I’ve built a bunch of (query based) taxonomies successfully. I have also placed a taxonomy query builder iview on two portal content pages (one page is called OHS and the other Finance): In the taxonomy query builder iview, is there a way to ONLY show related taxonomies for OHS users on the OHS page (and only show Finance taxonomies on the Finance pages)?

In the Taxonomy Query builder it appears that All taxonomies appear all the time. I’ve tried setting the permissions on the individual indexes and also on the taxonomy folders in KM.