Anyscale Endpoints

This documentation describes the integration of MindsDB with Anyscale Endpoints, a fast and scalable API to integrate OSS LLMs into apps. The integration allows for the deployment of Anyscale Endpoints models within MindsDB, providing the models with access to data from various data sources.

Prerequisites

Before proceeding, ensure the following prerequisites are met:

Install MindsDB locally via Docker or Docker Desktop.
To use Anyscale Endpoints within MindsDB, install the required dependencies following this instruction.
Obtain the Anyscale Endpoints API key required to deploy and use Anyscale Endpoints models within MindsDB. Follow the instructions for obtaining the API key.

Setup

Create an AI engine from the Anyscale Endpoints handler.

CREATE ML_ENGINE anyscale_endpoints_engine
FROM anyscale_endpoints
USING
      anyscale_endpoints_api_key = 'api-key-value';

Create a model using anyscale_endpoints_engine as an engine.

CREATE MODEL anyscale_endpoints_model
[FROM integration
         (SELECT * FROM table)]
PREDICT target_column
USING
      engine = 'anyscale_endpoints_engine',   -- engine name as created via CREATE ML_ENGINE
      api_base = 'base-url', -- optional, replaces the default base URL
      mode = 'conversational', -- optional, mode to run the model in
      model_name = 'anyscale_endpoints_model_name',  -- optional, the LLM to use
      prompt = 'You are a helpful assistant. Your task is to continue the chat.',  -- optional, system prompt for the model
      question_column = 'question',  -- optional, column name that stores user input
      context_column = 'context',  -- optional, column that stores context of the user input
      prompt_template = 'Answer the users input in a helpful way: {{question}}', -- optional, base template with placeholders used to provide input to the model 
      max_tokens = 100, -- optional, token limit for model output
      temperature = 0.3, -- optional, randomness setting for the model output
      json_struct = {
        'key': 'value',
        ...
      }' -- optional, the parameter for extracting JSON data from `prompt_template`

It is possilbe to override certain parameters set for a model at prediction time instead of recreating the model. For example, to change the temperature parameter for a specific prediction, use the following query:

SELECT question, answer
FROM anyscale_endpoints_model
WHERE question = 'Where is Stockholm located?'
USING
      temperature = 0.9
      prompt_template = 'Answer the users input as a pirate: {{question}}';

The parameters that can be overridden as shown above are mentioned below in the detailed explanation.

The following is a more detailed explanation of the parameters used in the CREATE MODEL statement:

The implementation of this integration is based on the engine for the OpenAI API, as Anyscale conforms to it. There are a few notable differences, though:

All models supported by Anyscale Endpoints are open source. A full list can be found here for inference-only under section Supported models.
Not every model is supported for fine-tuning. You can find a list here under section Fine Tuning - Supported models.

Please check both lists regularly, as they are subject to change. If you try to fine-tune a model that is not supported, you will get a warning and subsequently an error from the Anyscale endpoint.

This integration only offers chat-based text completion models, either for normal text or specialized for code.
When providing a description, this integration returns the respective HuggingFace model card.
Fine-tuning requires that your dataset complies with the chat format. That is, each row should contain a context and a role. The context is the text that is the message in the chat, and the role is who authored it (system, user, or assistant, where the last one is the model). For more information, please check the fine tuning guide in the Anyscale Endpoints docs.

The base URL for this API is https://api.endpoints.anyscale.com/v1.

Usage

The following usage examples utilize anyscale_endpoints_engine to create a model with the CREATE MODEL statement.

The output generated for a single input will be the same regardless of the mode used. The difference between the modes is in how the model handles multiple inputs.

files.unrelated_questions is a simple CSV file containing a question column with simple (unrelated) questions that has been uploaded to MindsDB, while files.related_questions is a similar file containing related questions. files.unrelated_questions_with_context and files.related_questions_with_context are similar files containing an additional context column.

These files are used in the examples given below to provide multiple inputs to the models created. It is possible to use any other supported data source in the same manner.

In the default mode, the model will generate a separate response for each input provided. No context is maintained between the inputs.

CREATE MODEL anyscale_endpoints_default_model
PREDICT answer
USING
      engine = 'anyscale_endpoints_engine',
      model_name = 'mistralai/Mistral-7B-Instruct-v0.1',
      prompt_template = 'Answer the users input in a helpful way: {{question}}';

To generate a response for a single input, the following query can be used:

SELECT question, answer
FROM anyscale_endpoints_default_model
WHERE question = 'Where is Stockholm located?';

The response will look like the following:

+---------------------------+-------------------------------+
|question                   |answer                         |
+---------------------------+-------------------------------+
|Where is Stockholm located?|Stockholm is the capital city of Sweden, located in the southeastern part of the country. It is situated on an archipelago of 30,000 islands in the Baltic Sea, and is known for its beautiful waterfront, historic buildings, and vibrant cultural scene.|
+---------------------------+-------------------------------+

To generate responses for multiple inputs, the following query can be used:

SELECT d.question, m.answer
FROM files.unrelated_questions AS d
JOIN anyscale_endpoints_default_model AS m

The response will look like the following:

| question | answer |
| -------- | ------ |
| Where is Stockholm located? | Stockholm is the capital city of Sweden, located in the southeastern part of the country. It is situated on an archipelago of 30,000 islands in the Baltic Sea, and is known for its beautiful waterfront, historic buildings, and vibrant cultural scene. |
| What is the fourth planet in the solar system? | The fourth planet in the solar system is Mars. |

CREATE MODEL anyscale_endpoints_default_model
PREDICT answer
USING
      engine = 'anyscale_endpoints_engine',
      model_name = 'mistralai/Mistral-7B-Instruct-v0.1',
      question_column = 'question';

To generate a response for a single input, the following query can be used:

SELECT question, answer
FROM anyscale_endpoints_default_model
WHERE question = 'Where is Stockholm located?';

The response will look like the following:

+---------------------------+-------------------------------+
|question                   |answer                         |
+---------------------------+-------------------------------+
|Where is Stockholm located?|Stockholm is the capital city of Sweden, located in the southeastern part of the country. It is situated on an archipelago of 30,000 islands in the Baltic Sea, and is known for its beautiful waterfront, historic buildings, and vibrant cultural scene.|
+---------------------------+-------------------------------+

To generate responses for multiple inputs, the following query can be used:

SELECT d.question, m.answer
FROM files.unrelated_questions AS d
JOIN anyscale_endpoints_default_model AS m

The response will look like the following:

| question | answer |
| -------- | ------ |
| Where is Stockholm located? | Stockholm is the capital city of Sweden, located in the southeastern part of the country. It is situated on an archipelago of 30,000 islands in the Baltic Sea, and is known for its beautiful waterfront, historic buildings, and vibrant cultural scene. |
| What is the fourth planet in the solar system? | The fourth planet in the solar system is Mars. |

CREATE MODEL anyscale_endpoints_default_model
PREDICT answer
USING
      engine = 'anyscale_endpoints_engine',
      model_name = 'mistralai/Mistral-7B-Instruct-v0.1',
      question_column = 'question',
      context_column = 'context';

To generate a response for a single input, the following query can be used:

SELECT question, context, answer
FROM anyscale_endpoints_default_model
WHERE question = 'What is the main topic of the conference happening next week?' AND context = 'The conference, focusing on advancements in AI, will take place in San Francisco next week.';

The response will look like the following:

+---------------------------+-------------------------------+-------------------------------+
|question                   |context                         | answer                        |
+---------------------------+-------------------------------+-------------------------------+
|What is the main topic of the conference happening next week? | The conference, focusing on advancements in AI, will take place in San Francisco next week.|The main topic of the conference happening next week in San Francisco is advancements in AI.|
+---------------------------+-------------------------------+-------------------------------+

To generate responses for multiple inputs, the following query can be used:

SELECT d.question, d.context, m.answer
FROM files.unrelated_questions_with_context AS d
JOIN anyscale_endpoints_default_model AS m

The response will look like the following:

+---------------------------+-------------------------------+-------------------------------+
|question                   |context                         | answer                        |
+---------------------------+-------------------------------+-------------------------------+
|What is the main topic of the conference happening next week? | The conference, focusing on advancements in AI, will take place in San Francisco next week.|The main topic of the conference happening next week in San Francisco is advancements in AI.|
+---------------------------+-------------------------------+-------------------------------+
|What caused the extension of the project deadline? | The project deadline was extended by two weeks due to team members falling sick unexpectedly. | The extension of the project deadline was caused by unexpected illnesses among team members. |
+---------------------------+-------------------------------+-------------------------------+

In the conversational mode, the model will maintain context between the inputs and generate a single response. This response will be placed in the last row of the result set.

CREATE MODEL anyscale_endpoints_conversational_model
PREDICT answer
USING
      engine = 'anyscale_endpoints_engine',
      mode = 'conversational',
      model_name = 'mistralai/Mistral-7B-Instruct-v0.1',
      prompt = 'You are a helpful assistant. Your task is to continue the chat.',
      prompt_template = 'Answer the users input in a helpful way: {{question}}';

To generate a response for a single input, the following query can be used:

SELECT question, answer
FROM anyscale_endpoints_conversational_model
WHERE question = 'Where is Stockholm located?';

The response will look like the following:

+---------------------------+-------------------------------+
|question                   |answer                         |
+---------------------------+-------------------------------+
|Where is Stockholm located?|Stockholm is the capital city of Sweden, located in the southeastern part of the country. It is situated on an archipelago of 30,000 islands in the Baltic Sea, and is known for its beautiful waterfront, historic buildings, and vibrant cultural scene.|
+---------------------------+-------------------------------+

To generate responses for multiple inputs, the following query can be used:

SELECT d.question, m.answer
FROM files.related_questions AS d
JOIN anyscale_endpoints_conversational_model AS m

The response will look like the following:

question	answer
Where is Stockholm located?
What are some fun activities to do there?	Stockholm is the capital city of Sweden and is located in the southeastern part of the country. Some fun activities to do in Stockholm include visiting the famous Vasa Museum, exploring the beautiful archipelago, taking a stroll through the charming Gamla Stan neighborhood, and trying out some of the local food and drinks.

CREATE MODEL anyscale_endpoints_conversational_model
PREDICT answer
USING
      engine = 'anyscale_endpoints_engine',
      mode = 'conversational',
      model_name = 'mistralai/Mistral-7B-Instruct-v0.1',
      prompt = 'You are a helpful assistant. Your task is to continue the chat.',
      question_column = 'question';

To generate a response for a single input, the following query can be used:

SELECT question, answer
FROM anyscale_endpoints_conversational_model
WHERE question = 'Where is Stockholm located?';

The response will look like the following:

+---------------------------+-------------------------------+
|question                   |answer                         |
+---------------------------+-------------------------------+
|Where is Stockholm located?|Stockholm is the capital city of Sweden, located in the southeastern part of the country. It is situated on an archipelago of 30,000 islands in the Baltic Sea, and is known for its beautiful waterfront, historic buildings, and vibrant cultural scene.|
+---------------------------+-------------------------------+

To generate responses for multiple inputs, the following query can be used:

SELECT d.question, m.answer
FROM files.unrelated_questions AS d
JOIN anyscale_endpoints_conversational_model AS m

The response will look like the following:

question	answer
Where is Stockholm located?
What are some fun activities to do there?	Stockholm is the capital city of Sweden and is located in the southeastern part of the country. Some fun activities to do in Stockholm include visiting the famous Vasa Museum, exploring the beautiful archipelago, taking a stroll through the charming Gamla Stan neighborhood, and trying out some of the local food and drinks.

CREATE MODEL anyscale_endpoints_conversational_model
PREDICT answer
USING
      engine = 'anyscale_endpoints_engine',
      mode = 'conversational',
      model_name = 'mlabonne/NeuralHermes-2.5-Mistral-7B',
      prompt = 'You are a helpful assistant. Your task is to continue the chat.',
      question_column = 'question',
      context_column = 'context';

To generate a response for a single input, the following query can be used:

SELECT question, context, answer
FROM anyscale_endpoints_conversational_model
WHERE question = 'What is the main topic of the conference happening next week?' AND context = 'The conference, focusing on advancements in AI, will take place in San Francisco next week.';

The response will look like the following:

+---------------------------+-------------------------------+-------------------------------+
|question                   |context                         | answer                        |
+---------------------------+-------------------------------+-------------------------------+
|What is the main topic of the conference happening next week? | The main topic of the conference happening next week is advancements in Artificial Intelligence.|
+---------------------------+-------------------------------+-------------------------------+

To generate responses for multiple inputs, the following query can be used:

SELECT d.question, d.context, m.answer
FROM files.related_questions_with_context AS d
JOIN anyscale_endpoints_conversational_model AS m

The response will look like the following:

question	context	answer
Where is Anna planning a trip to next month?	Anna is planning a trip to Kyoto next month.
What does Anna plan on doing there?	Anna plans on going sightseeing.	Anna plans on going sightseeing during her trip to Kyoto next month.

In the conversational-full mode, the model will maintain context between the inputs and generate a response for each input.

CREATE MODEL anyscale_endpoints_conversational_full_model
PREDICT answer
USING
      engine = 'anyscale_endpoints_engine',
      mode = 'conversational-full',
      model_name = 'mistralai/Mistral-7B-Instruct-v0.1',
      prompt_template = 'Answer the users input in a helpful way: {{question}}';

To generate a response for a single input, the following query can be used:

SELECT question, answer
FROM anyscale_endpoints_conversational_full_model
WHERE question = 'Where is Stockholm located?';

The response will look like the following:

+---------------------------+-------------------------------+
|question                   |answer                         |
+---------------------------+-------------------------------+
|Where is Stockholm located?|Stockholm is the capital city of Sweden, located in the southeastern part of the country. It is situated on an archipelago of 30,000 islands in the Baltic Sea, and is known for its beautiful waterfront, historic buildings, and vibrant cultural scene.|
+---------------------------+-------------------------------+

To generate responses for multiple inputs, the following query can be used:

SELECT d.question, m.answer
FROM files.related_questions AS d
JOIN anyscale_endpoints_conversational_full_model AS m

The response will look like the following:

question	answer
Where is Stockholm located?	Stockholm is the capital city of Sweden, located in the southeastern part of the country. It is situated on an island in the Stockholm archipelago, which is made up of more than 30,000 islands. The city is known for its beautiful architecture, museums, and cultural attractions, as well as its vibrant food and nightlife scene.
What are some fun activities to do there?	Stockholm is the capital city of Sweden and is located in the southeastern part of the country, on the east coast of the Stockholm archipelago. Some fun activities to do in Stockholm include visiting the famous Vasa Museum, exploring the charming old town of Gamla Stan, taking a stroll through the beautiful parks and gardens, and trying out some of the local food and drinks. There are also many opportunities for shopping, cultural experiences, and outdoor activities such as hiking and biking

question

answer

Where is Stockholm located?

Stockholm is the capital city of Sweden, located in the southeastern part of the country. It is situated on an island in the Stockholm archipelago, which is made up of more than 30,000 islands. The city is known for its beautiful architecture, museums, and cultural attractions, as well as its vibrant food and nightlife scene.

What are some fun activities to do there?

Stockholm is the capital city of Sweden and is located in the southeastern part of the country, on the east coast of the Stockholm archipelago. Some fun activities to do in Stockholm include visiting the famous Vasa Museum, exploring the charming old town of Gamla Stan, taking a stroll through the beautiful parks and gardens, and trying out some of the local food and drinks. There are also many opportunities for shopping, cultural experiences, and outdoor activities such as hiking and biking

CREATE MODEL anyscale_endpoints_conversational_full_model
PREDICT answer
USING
      engine = 'anyscale_endpoints_engine',
      mode = 'conversational-full',
      model_name = 'mistralai/Mistral-7B-Instruct-v0.1',
      question_column = 'question';

To generate a response for a single input, the following query can be used:

SELECT question, answer
FROM anyscale_endpoints_conversational_full_model
WHERE question = 'Where is Stockholm located?';

The response will look like the following:

+---------------------------+-------------------------------+
|question                   |answer                         |
+---------------------------+-------------------------------+
|Where is Stockholm located?|Stockholm is the capital city of Sweden, located in the southeastern part of the country. It is situated on an archipelago of 30,000 islands in the Baltic Sea, and is known for its beautiful waterfront, historic buildings, and vibrant cultural scene.|
+---------------------------+-------------------------------+

To generate responses for multiple inputs, the following query can be used:

SELECT d.question, m.answer
FROM files.related_questions AS d
JOIN anyscale_endpoints_conversational_full_model AS m

The response will look like the following:

question	answer
Where is Stockholm located?	Stockholm is the capital city of Sweden, located in the southeastern part of the country. It is situated on an island in the Stockholm archipelago, which is made up of more than 30,000 islands. The city is known for its beautiful architecture, museums, and cultural attractions, as well as its vibrant food and nightlife scene.
What are some fun activities to do there?	Stockholm is the capital city of Sweden and is located in the southeastern part of the country, on the east coast of the Stockholm archipelago. Some fun activities to do in Stockholm include visiting the famous Vasa Museum, exploring the charming old town of Gamla Stan, taking a stroll through the beautiful parks and gardens, and trying out some of the local food and drinks. There are also many opportunities for shopping, cultural experiences, and outdoor activities such as hiking and biking

question

answer

Where is Stockholm located?

What are some fun activities to do there?

CREATE MODEL anyscale_endpoints_conversational_full_model
PREDICT answer
USING
      engine = 'anyscale_endpoints_engine',
      mode = 'conversational-full',
      model_name = 'mlabonne/NeuralHermes-2.5-Mistral-7B',
      question_column = 'question',
      context_column = 'context';

To generate a response for a single input, the following query can be used:

SELECT question, context, answer
FROM anyscale_endpoints_conversational_full_model
WHERE question = 'What is the main topic of the conference happening next week?' AND context = 'The conference, focusing on advancements in AI, will take place in San Francisco next week.';

The response will look like the following:

+---------------------------+-------------------------------+-------------------------------+
|question                   |context                         | answer                        |
+---------------------------+-------------------------------+-------------------------------+
|What is the main topic of the conference happening next week? | The conference, focusing on advancements in AI, will take place in San Francisco next week.|The main topic of the conference happening next week in San Francisco is advancements in AI.|
+---------------------------+-------------------------------+-------------------------------+

To generate responses for multiple inputs, the following query can be used:

SELECT d.question, d.context, m.answer
FROM files.related_questions_with_context AS d
JOIN anyscale_endpoints_conversational_full_model AS m

The response will look like the following:

question	context	answer
Where is Anna planning a trip to next month?	Anna is planning a trip to Kyoto next month.	Anna is planning a trip to Kyoto next month.
What does Anna plan on doing there?	Anna plans on going sightseeing.	Anna plans on going sightseeing during her trip to Kyoto next month.

Next Steps

Follow this tutorial to see more use case examples.

Troubleshooting Guide

Authentication Error

Symptoms: Failure to authenticate to the Anyscale.
Checklist:
1. Make sure that your Anyscale account is active.
2. Confirm that your API key is correct.
3. Ensure that your API key has not been revoked.
4. Ensure that you have not exceeded the API usage or rate limit.

SQL statement cannot be parsed by mindsdb_sql

Symptoms: SQL queries failing or not recognizing table and model names containing spaces or special characters.

Checklist:

Ensure table names with spaces or special characters are enclosed in backticks. Examples:

Incorrect:

SELECT input.text, output.sentiment
FROM integration.travel data AS input
JOIN openai_engine AS output

Incorrect:

SELECT input.text, output.sentiment
FROM integration.'travel data' AS input
JOIN openai_engine AS output

Correct:

SELECT input.text, output.sentiment
FROM integration.`travel data` AS input
JOIN openai_engine AS output

Integrations

Data Sources

AI Engines

Anyscale Endpoints

Prerequisites

Setup

Usage

Troubleshooting Guide

Integrations

Data Sources

AI Engines

​Prerequisites

​Setup

​Usage

​Troubleshooting Guide

Prerequisites

Setup

Usage

Troubleshooting Guide