Knowledge Base
A knowledge base is an advanced system that organizes information based on semantic meaning rather than simple keyword matching. It integrates embedding models, reranking models, and vector stores to enable context-aware data retrieval.
By performing semantic reasoning across multiple data points, a knowledge base delivers deeper insights and more accurate responses, making it a powerful tool for intelligent data access.
CREATE KNOWLEDGE_BASE
Syntax
Here is the syntax for creating a knowledge base:
Upon execution, it registers my_kb
and associates the specified models and storage.
my_kb
is a unique identifier of the knowledge base within MindsDB.
As MindsDB stores objects, such as models or knowledge bases, inside projects, you can create a knowledge base inside a custom project.
embedding_model
The embedding model is a required component of the knowledge base. It stores specifications of the embedding model to be used.
Users can define the embedding model choosing one of the following options.
Option 1. Use the embedding_model
parameter to define the specification.
Option 2. Define the default embedding model in the MindsDB configuration file.
Note that if you define default_embedding_model
in the configuration file, you do not need to provide the embedding_model
parameter when creating a knowledge base. If provide both, then the values from the embedding_model
parameter are used.
The embedding model specification includes:
-
provider
It is a required parameter. It defines the model provider. Currently, the supported providers include OpenAI (openai
) and OpenAI via Azure (azure_openai
). -
model_name
It is a required parameter. It defines the embedding model name as specified by the provider. Users can choose one of the OpenAI embedding models. -
api_key
The API key is required to access the embedding model assigned to a knowledge base. Users can provide it either in thisapi_key
parameter, or in theOPENAI_API_KEY
environment variable. -
base_url
It is an optional parameter, which defaults tohttps://api.openai.com/v1/
. It is a required parameter when using theazure_openai
provider. It is the root URL used to send API requests. -
api_version
It is an optional parameter. It is a required parameter when using theazure_openai
provider. It defines the API version.
reranking_model
The reranking model is an optional component of the knowledge base. It stores specifications of the reranking model to be used.
Users can define the reranking model choosing one of the following options.
Option 1. Use the reranking_model
parameter to define the specification.
Option 2. Define the default reranking model in the MindsDB configuration file.
Note that if you define default_llm
in the configuration file, you do not need to provide the reranking_model
parameter when creating a knowledge base. If provide both, then the values from the reranking_model
parameter are used.
The reranking model specification includes:
-
provider
It is a required parameter. It defines the model provider. Currently, the supported providers include OpenAI (openai
) and OpenAI via Azure (azure_openai
). -
model_name
It is a required parameter. It defines the embedding model name as specified by the provider. Users can choose one of the OpenAI chat models. -
api_key
The API key is required to access the reranking model assigned to a knowledge base. Users can provide it either in thisapi_key
parameter, or in theOPENAI_API_KEY
environment variable. -
base_url
It is an optional parameter, which defaults tohttps://api.openai.com/v1/
. It is a required parameter when using theazure_openai
provider. It is the root URL used to send API requests. -
api_version
It is an optional parameter. It is a required parameter when using theazure_openai
provider. It defines the API version. -
method
It is an optional parameter. It defines the method used to calculate the relevance of the output rows. The available options includemulti-class
andbinary
. It defaults tomulti-class
.
Reranking Method
The multi-class
reranking method classifies each document chunk (that meets any specified metadata filtering conditions) into one of four relevance classes:
- Not relevant with class weight of 0.25.
- Slightly relevant with class weight of 0.5.
- Moderately relevant with class weight of 0.75.
- Highly relevant with class weight of 1.
The overall relevance_score
of a document is calculated as the sum of each chunk’s class weight multiplied by its class probability (from model logprob output).
The binary
reranking method simplifies classification by determining whether a document is relevant or not, without intermediate relevance levels. With this method, the overall relevance_score
of a document is calculated based on the model log probability.
storage
The vector store is a required component of the knowledge base. It stores data in the form of embeddings.
It is optional for users to provide the storage
parameter. If not provided, the default ChromaDB is created when creating a knowledge base.
In order to provide the storage vector database, it is required to connect it to MindsDB beforehand.
Here is an example for PGVector.
Note that you do not need to have the storage_table
created as it is created when creating a knowledge base.
The available options include either PGVector or ChromaDB.
If the storage
parameter is not provided, the system creates the default ChromaDB vector database called <kb_name>_chromadb
with the default table called default_collection
that stores the embedded data.
This default ChromaDB vector database is stored in MindsDB’s storage. In order to define a different storage directory, create your own ChromaDB referring to this documentation page.
metadata_columns
The data inserted into the knowledge base can be classified as metadata, which enables users to filter the search results using defined data fields.
Note that source data column(s) included in metadata_columns
cannot be used in content_columns
, and vice versa.
This parameter is an array of strings that lists column names from the source data to be used as metadata. If not provided, the metadata is not used.
Here is an example of usage. A user wants to store the following data in a knowledge base.
Go to the Complete Example section below to find out how to access this sample data.
The product
column can be used as metadata to enable metadata filtering.
content_columns
The data inserted into the knowledge base can be classified as content, which is embedded by the embedding model and stored in the underlying vector store.
Note that source data column(s) included in content_columns
cannot be used in metadata_columns
, and vice versa.
This parameter is an array of strings that lists column names from the source data to be used as content and processed into embeddings. If not provided, the content
column is expected by the knowledge base by default.
Here is an example of usage. A user wants to store the following data in a knowledge base.
Go to the Complete Example section below to find out how to access this sample data.
The notes
column can be used as content.
id_column
The data inserted into the knowledge base can be classified as ID, which is a column in the source data that uniquely identifies each row.
It is mandatory for users to provide the id_column
parameter. This parameter is a string that contains the source data ID column name.
Here is an example of usage. A user wants to store the following data in a knowledge base.
Go to the Complete Example section below to find out how to access this sample data.
The order_id
column can be used as ID.
Note that if the source data row is chunked into multiple chunks by the knowledge base (that is, to optimize the storage), then these rows in the knowledge base have the same ID value that identifies chunks from one source data row.
Example
Here is a sample knowledge base that will be used for examples in the following content.
DESCRIBE KNOWLEDGE_BASE
Syntax
Users can get details about the knowledge base using the DESCRIBE KNOWLEDGE_BASE
command.
Here is the sample output:
INSERT INTO
Syntax
Here is the syntax for inserting data into a knowledge base:
Upon execution, it inserts data into a knowledge base, using the embedding model to embed it into vectors before inserting into an underlying vector database.
Note that when inserting the same data into the knowledge base again, then no new rows are inserted, as the content already exists in the knowledge base.
Update Existing Data
In order to update existing data in the knowledge base, insert data with the column ID that you want to update and the updated content.
Here is an example of usage. A knowledge base stores the following data.
A user updated Laptop Stand
to Aluminum Laptop Stand
.
Go to the Complete Example section below to find out how to access this sample data.
Here is how to propagate this change into the knowledge base.
The knowledge base matches the ID value to the existing one and updates the data if required.
Optimize Data Insertion
In order to optimize the performance of data insertion into the knowledge base, users can set up partitions and threads to insert batches of data in parallel. This also enables tracking the progress of data insertion process including cancelling and resuming it if required.
Here is an example.
The parameters include the following:
-
batch_size
defines the number of rows fetched per iteration to optimize data extraction from the source. It defaults to 1000. -
threads
defines threads for running partitions. Note that if the ML task queue is enabled, threads are used automatically. The available values for include:- A number of threads to be used, for example,
threads = 10
, - A boolean value that defines whether to enable threads, setting
threads = true
, or disable threads, settingthreads = false
.
- A number of threads to be used, for example,
-
track_column
defines the column used for sorting data before partitioning. -
error
defines the error processing options. The available values includeraise
, used to raise errors as they come, orskip
, used to subside errors. It defaults toraise
if not provided.
After executing the INSERT INTO
statement with the above parameters, users can view the data insertion progress by querying the information_schema.queries
table.
Users can cancel the data insertion process using the process ID from the information_schema.queries
table.
Note that canceling the query will not remove the already inserted data.
Users can resume the data insertion process using the process ID from the information_schema.queries
table.
Chunking Data
Upon inserting data into the knowledge base, the data chunking is performed in order to optimize the storage and search of data.
Each chunk is identified by its chunk ID of the following format: <id>:<chunk_number>of<total_chunks>:<start_char_number>to<end_char_number>
.
Underlying Vector Store
Each knowledge base has its underlying vector store that stores data inserted into the knowledge base in the form of embeddings.
Users can query the underlying vector store as follows.
- KB with the default ChromaDB vector store:
Example
Here a sample knowledge base created in the previous Example section is inserted into.
When inserting into a knowledge base where the content_columns
parameter was not specified, the column storing content must be aliased AS content
as below.
SELECT FROM KB
Syntax
Knowledge bases provide an abstraction that enables users to see the stored data.
Note that here a sample knowledge base created and inserted into in the previous Example sections is searched.
Here is the sample output:
Data Stored in Knowledge Base
The following columns are stored in the knowledge base.
id
It stores values from the column defined in the id_column
parameter when creating the knowledge base. These are the source data IDs.
chunk_id
Knowledge bases chunk the inserted data in order to fit the defined chunk size. If the chunking is performed, the following chunk ID format is used: <id>:<chunk_number>of<total_chunks>:<start_char_number>to<end_char_number>
.
chunk_content
It stores values from the column(s) defined in the content_columns
parameter when creating the knowledge base.
metadata
It stores the general metadata and the metadata defined in the metadata_columns
parameter when creating the knowledge base.
distance
It stores the calculated distance between the chunk’s content and the search phrase.
relevance
It stores the calculated relevance of the chunk as compared to the search phrase.
Note that the calculation method of relevance_threshold
differs as follows:
- When the ranking model is provided, the default
relevance_threshold
is 0, unless defined otherwise in theWHERE
clause. - When the reranking model is not provided and the
relevance_threshold
is not defined in the query, then no relevance filtering is applied and the output includes all rows matched based on the similarity and metadata search. - When the reranking model is not provided but the
relevance_threshold
is defined in the query, then the relevance is calculated based on thedistance
column (1/(1+ distance)
) and therelevance_threshold
value is compared with this relevance value to filter the output.
Semantic Search
Users can query a knowledge base using semantic search by providing the search phrase (called content
) to be searched for.
Here is the output:
Notice that the output is in this case the same as the one without specifying content
in the WHERE
clause. This is caused by the default relevance_threshold
set to 0, which is its lowest value.
Users can limit the relevance_threshold
in order to get only the most relevant results.
Here is the output:
By providing the relevance_threshold
filter, the output is limited to only data with relevance score of the provided value or higher. The available value of relevance_threshold
are between 0 and 1, and its default value is 0.
Users can limit the number of rows returned.
Here is the output:
Metadata Filtering
Besides semantic search features, knowledge bases enable users to filter the result set by the defined metadata.
Here is the output:
Note that when searching by metadata alone, the relevance
column values are not calculated.
Users can do both, filter by metadata and search by content.
Here is the output:
JOIN
Syntax
Knowledge bases can be used in the standard SQL JOIN statements.
Here is the output:
DELETE FROM
Syntax
Here is the syntax for deleting from a knowledge base:
Upon execution, it identifies matching records based on the user-defined condition and removes all associated data (metadata, content, chunks, embeddings) for matching records from the KB’s storage.
DROP KNOWLEDGE_BASE
Syntax
Here is the syntax for deleting a knowledge base:
Upon execution, it removes the knowledge base with its content.
Complete Example
Here is the data that will be inserted into the knowledge base.
You can access this sample data as below:
Here is how to create a knowledge base specifically for the data.
Here is how to insert the data.
Here is how to query the knowledge base.