This handler was implemented using the replicate library that is provided by Replicate.

The required arguments to establish a connection are,

  • model_name: Model name which you want to access in MindsDB. e.g ‘air-forever/kandinsky-2’
  • version: version hash/id which you want to use in MindsDB.
  • api_key: API key from Replicate Platform you can found here.
  • model_type: It should be set to ‘LLM’ while using Large language Model else it Optional

Before you can use Replicate, it’s essential to authenticate by setting your API token in an environment variable named REPLICATE_API_TOKEN. This token acts as a key to enable access to Replicate’s features.

  1. Using pip:

If you’re working in a standard Python environment (using pip for package management), set your token as an environment variable by running the following command in your terminal:

On Linux, Mac:

export REPLICATE_API_TOKEN='YOUR_TOKEN'

On Windows:

set REPLICATE_API_TOKEN=YOUR_TOKEN
  1. Using Docker:

For Docker users, the process slightly differs. You need to pass the environment variable directly to the Docker container when running it. Use this command:

docker run -e REPLICATE_API_TOKEN='YOUR_TOKEN' -p 47334:47334 -p 47335:47335 mindsdb/mindsdb

Again, replace ‘YOUR_TOKEN’ with your actual Replicate API token.

​
Usage

To use this handler and connect to a Replicate cluster in MindsDB, you need an account on Replicate. Make sure to create an account by following this link.

To establish the connection and create a model in MindsDB, use the following syntax:

CREATE MODEL vicuna_13b
PREDICT output
USING
    engine = 'replicate',
    model_name= 'replicate/vicuna_13b',
    model_type='LLM',
    version ='6282abe6a492de4145d7bb601023762212f9ddbbe78278bd6771c8b3b2f2a13b',
    api_key = 'r8_HEH............';

You can use the DESCRIBE PREDICTOR query to see the available parameters that you can specify to customize your predictions:

DESCRIBE PREDICTOR mindsdb.vicuna_13b.features;

​
OUTPUT

+--------------------+---------+---------+---------------------------------------------------------------------------------------------------------------------------------------+
| inputs             | type    | default | description                                                                                                                           |
+--------------------+---------+---------+---------------------------------------------------------------------------------------------------------------------------------------+
| seed               | integer | -1      | Seed for random number generator, for reproducibility                                                                                 |
| debug              | boolean | False   | provide debugging output in logs                                                                                                      |
| top_p              | number  | 1       | When decoding text, samples from the top p percentage of most likely tokens; lower to ignore less likely tokens                       |
| prompt             | string  | -       | Prompt to send to Llama.                                                                                                              |
| max_length         | integer | 500     | Maximum number of tokens to generate. A word is generally 2-3 tokens                                                                  |
| temperature        | number  | 0.75    | Adjusts randomness of outputs, greater than 1 is random and 0 is deterministic, 0.75 is a good starting value.                        |
| repetition_penalty | number  | 1       | Penalty for repeated words in generated text; 1 is no penalty, values greater than 1 discourage repetition, less than 1 encourage it. |
+--------------------+---------+---------+---------------------------------------------------------------------------------------------------------------------------------------+

Now, you can use the established connection to query your ML Model as follows:

SELECT *
FROM vicuna_13b
WHERE prompt='Write a humourous  poem on Open Source'
USING
 max_length=200,
 temperature=0.75;

​
OUTPUT

+--------------------------------------------------------------+----------------------------------------+
| output                                                       | prompt                                 |
+--------------------------------------------------------------+----------------------------------------+
| Opensource software, oh how we love thee                     | Write a humourous  poem on Open Source |  
| With bugs and glitches, oh so free                           |                                        |  
| You bring us laughter and joy each day                       |                                        |  
| And we'll never have to pay                                  |                                        |  
|                                                              |                                        |  
| The license is open, the code is there                       |                                        |  
| For all to see and share in cheer                            |                                        |  
| You bring us together, from far and wide                     |                                        |  
| To work on projects, side by side                            |                                        |  
|                                                              |                                        |  
| With open source, there's no end                             |                                        |  
| To the code we can bend                                      |                                        |  
| We can change it, mold it, make it our own                   |                                        |  
| And create something truly great, or really strange          |                                        |  
|                                                              |                                        |  
| So here's to open source, the future is bright               |                                        |  
| With code that's free, and with all of our might             |                                        |  
| We"ll make the future shine, with technology fine            |                                        |  
| And open source will always be our shining line              |                                        |  
+--------------------------------------------------------------+----------------------------------------+

Note: Replicate provides only a few free predictions, so choose your predictions wisely. Don’t let the machines have all the fun, save some for yourself! 😉

Was this page helpful?

PortkeyVertex AI