Semantic Search
Semantic Search
What is semantic search?
- Search by meaning, not exact characters
- Finds results by meaning, not exact wording
- Built on embeddings and a vector database
What changes is who gets blamed when a search fails. In the keyword era, the fault lay with the 'employee who typed the wrong search term.' Type 'half-day off' into the intranet portal and get zero results, and the employee tries 'vacation,' 'annual leave usage,' and 'attendance policy' in turn, then ends up calling HR. With semantic search, the responsibility for not asking the question precisely shifts to the system. What the owner manages also shifts, from 'how many synonyms have we added to the dictionary' to how documents are chunked and which model vectorizes them.
How it works is simple. Documents are split into chunks of 300–800 tokens, and an embedding model converts each chunk into an array of 384 to 3,072 numbers, which is stored. The question is vectorized the same way, and the top 5–10 chunks by cosine similarity are retrieved. However, it is not the better method for every search. For queries where the exact characters are the information, such as a part number like 'KX-2200B', 'Article 60 of the Labor Standards Act', or a contract number, BM25-style keyword search is still far more accurate. That is why the standard setup in the field is now a hybrid search that merges both result sets, with a reranker step that re-orders the rankings added on top.
The most common mistake is expecting that 'if there is no answer, it will say so'. Vector search always returns the closest match, even when no answer exists. If five irrelevant documents with a similarity of 0.55 go straight into the RAG prompt, the result is an answer that cites an internal document but gets the content wrong. You need a similarity threshold, and anything below it should be cut off with 'no material found'. Also, 'the business trip expense policy before the revision' and 'the policy after the revision' are nearly identical in meaning, so vectors cannot tell them apart, and a negation such as 'travel expenses are not paid' is still pulled in with a high score for the question 'travel expense payment'. Revision date, department, and validity should be filtered with metadata, and the process of removing retired documents from the index must go with it.
John KimInstructor John Kim, author of B2B Marketing Design and CEO of Nanum Consulting
Seolyeong ParkGenerative AI and AX training specialist with 17 years of hands-on experience and 200+ lectures, helping non-technical staff put AI to work right awayIt became practical with advances in word and sentence embeddings (since word2vec in 2013), and became the standard search method as large language models and RAG spread.
Consider a hypothetical case. Company A, an industrial valve manufacturer with 620 employees, had posted 12,000 policy and technical documents on its internal portal, yet the HR and technical sales teams still received more than 40 'where is that?' calls a day. The search logs showed that 34% of searches returned zero results. They launched a three-month project to move to semantic search.
- When Kim, a manager on the IT systems team, pulled 500 zero-result search queries, 62% were colloquial phrases that do not appear in policy documents, such as 'half-day off', 'condolence money', and 'business trip meal allowance'. The meeting concluded: 'Employees can't be expected to know legal terminology.'
- Policies and equipment manuals were split into clause-level chunks of about 600 tokens each, producing 34,000 chunks, which were vectorized with a Korean embedding model. Each chunk was tagged with department, revision date, and validity as metadata.
- In the first test, a question about the 'business travel expense limit' returned an old policy abolished in 2021 as the top result. A valid=Y filter was forced in front of the search, and the 1,800 abolished documents were removed from the index entirely.
- After the technical sales team complained that part-number search had been better before, the company merged keyword (BM25) and vector results and re-ranked them with a reranker. Search accuracy for 'KX-2200B' recovered to its previous level.
- Eight weeks later, the zero-result rate had fallen from 34% to 6%, and calls to the help desk had dropped from 40 a day to 13. After the system was set to answer 'no material found' below a similarity score of 0.6, incorrect-answer reports fell from 9 a week to 1.