Improving search efficiency for a large resource website

Hi ​@stabes, welcome to the Community! 😁

Zapier isn't really built for improving website search functionality, but you could try adding Zapier Chatbots to your site. It's similar to the agent idea you mentioned, so rather than users having to search your website themselves, they could ask the chatbot a question directly and it would surface relevant answers or point them to the right resources on your site.

If you give that a try and run into any trouble, just let us know!

the chatbot direction sam mentioned is the right instinct, the piece that actually makes or breaks it with 10k files is how you index them, not the bot on top.

what i'd do is pull the text out of every html page and pdf, split each into smaller chunks, and embed those into a vector store. then when someone asks a question the agent searches by meaning instead of keywords, pulls back the handful of most relevant chunks, and answers with a link straight to the source file it used. for practitioners and academics that citation part matters, they get pointed to the exact document rather than a guess.

the built-in zapier chatbot can do a lightweight version, but at 10k mixed html and pdf files you usually want a proper retrieval setup behind it so answers stay accurate and don't invent a file that isn't there. the pdfs especially need real text extraction first or search quality falls apart.

happy to walk through how i'd handle the pdf extraction and wire up the retrieval if it's useful, that's the part that trips most people up at this scale.

One thing worth adding to the retrieval advice above, because it is the part that surprises people at 10k files: pure vector search underperforms on exactly this kind of library.

Practitioner and academic content is full of jargon, acronyms, statute numbers and citation codes. Embeddings are good at meaning and bad at rare exact tokens, so a search for a specific form number or abbreviation will often return something topically close and completely wrong. The fix is hybrid search, run keyword (BM25) and vector retrieval side by side and merge the results. It is not much more work and it removes most of the embarrassing misses.

Two other things that matter more than which model you pick:

Chunk on structure, not character count. Split on headings and section boundaries so a chunk is a complete thought, and keep the document title and section heading inside the chunk text. A chunk that reads "see Table 3 above" with no context is noise no matter how good your embeddings are.

Store metadata alongside every chunk (document type, year, author, category) and let the search filter on it. Half of what feels like a search problem at this scale is really a filtering problem, people want the current guidance, not the 2011 version.

On the PDFs, agreed that extraction is where quality is won or lost. Anything scanned needs OCR, and multi column layouts need a parser that respects reading order or the text comes out interleaved and the chunks become gibberish. Worth spot checking 20 extracted files by hand before you embed the whole set, it tells you fast whether the pipeline is sound.