Improving search efficiency for a large resource website

Hi ​@stabes, welcome to the Community! 😁

Zapier isn't really built for improving website search functionality, but you could try adding Zapier Chatbots to your site. It's similar to the agent idea you mentioned, so rather than users having to search your website themselves, they could ask the chatbot a question directly and it would surface relevant answers or point them to the right resources on your site.

If you give that a try and run into any trouble, just let us know!

the chatbot direction sam mentioned is the right instinct, the piece that actually makes or breaks it with 10k files is how you index them, not the bot on top.

what i'd do is pull the text out of every html page and pdf, split each into smaller chunks, and embed those into a vector store. then when someone asks a question the agent searches by meaning instead of keywords, pulls back the handful of most relevant chunks, and answers with a link straight to the source file it used. for practitioners and academics that citation part matters, they get pointed to the exact document rather than a guess.

the built-in zapier chatbot can do a lightweight version, but at 10k mixed html and pdf files you usually want a proper retrieval setup behind it so answers stay accurate and don't invent a file that isn't there. the pdfs especially need real text extraction first or search quality falls apart.

happy to walk through how i'd handle the pdf extraction and wire up the retrieval if it's useful, that's the part that trips most people up at this scale.