I am having an index with over 150000 documents which contains documents from various repositories. We have indexed some files which we dont want the users to see. All these documents are external documents(internet webpages etc). I was thinking to add a filter which will filter the resources by checking the file name or the file extension. I know that we can add filer in the crawler to not to pick up certain docs but i need these pages because these pages contains links which i need to index.
Is there anyway we can do this?