Cohere Toolkit Installation & Setup on Ubuntu 24.04 | AI & RAG Apps

The Cohere Toolkit is a fully developed, open-source repository tailored for creating generative AI applications. It includes user interface components for knowledge-based assistants, supports AI models from Cohere Platform, Bedrock, and SageMaker, and seamlessly connects with enterprise data sources. With the Cohere Toolkit, you can develop and deploy Retrieval Augmented Generation (RAG) applications that are interactive, adaptable, and based on specialized knowledge domains.

Prerequisites

Before proceeding with the installation, ensure that you have:

• Access to an Ubuntu 24.04 instance with a non-root sudo user.
• Installed Docker and Docker Compose, which are required by Cohere Toolkit.
• Acquired a Cohere Toolkit API Key.

Obtain a Cohere Toolkit API Key

1. Create a Cohere account.
2. Log in to your Cohere account.
3. Navigate to API Keys in the sidebar.
4. Copy the default trial key from the Trial keys section and save it for later use.

Installing Poetry

Poetry is a Python dependency management tool that allows you to create an isolated environment for Cohere Toolkit. Follow these steps to install Poetry and Python 3.11 on Ubuntu 24.04.

Add the Deadsnakes PPA Repository

First, add the Deadsnakes PPA to access multiple Python versions:

$ sudo add-apt-repository ppa:deadsnakes/ppa

Update Package Index

Apply the repository changes:

$ sudo apt update

Install Python 3.11

Install the required Python version:

$ sudo apt install python3.11

Cohere Toolkit requires Python 3.11. If an unsupported version is detected, the build process will automatically use Python 3.11 if available.

Install Poetry

Next, install Poetry:

$ sudo apt install python3-poetry

Verify the Installation

Confirm that Poetry is installed:

$ poetry --version

The expected output is similar to:

Poetry (version 1.8.2)

Initialize a New Project

Create a new pyproject.toml configuration by running:

$ poetry init

During the setup, enter project details as prompted. Ensure to specify ^3.11 as the compatible Python version.

Example Configuration

Package name [linuxuser]:
Version [0.1.0]:
Description []:
Author [None, n to skip]: n
License []:
Compatible Python versions [^3.12]: ^3.11
Would you like to define your main dependencies interactively? (yes/no) [yes]
Package to add or search for (leave blank to skip):
Would you like to define your development dependencies interactively? (yes/no) [yes]
Package to add or search for (leave blank to skip):

Do you confirm generation? (yes/no) [yes]

Activate the Virtual Environment

Use Python 3.11 to activate a new virtual environment:

$ poetry env use 3.11

The expected output should resemble:

Creating virtualenv linuxuser-W6smXhkx-py3.11 in /home/linuxuser/.cache/pypoetry/virtualenvs
Using virtualenv: /home/linuxuser/.cache/pypoetry/virtualenvs/linuxuser-W6smXhkx-py3.11

Installing Cohere Toolkit on Ubuntu 24.04

The Cohere Toolkit provides two configuration templates—configuration.template.yaml and secrets.template.yaml—that help set up new configurations. Follow these steps to clone the Cohere Toolkit repository and configure it accordingly.

Step 1: Navigate to the Home Directory

Ensure you are in your home directory before proceeding:

$ cd

Step 2: Clone the Repository

Download the Cohere Toolkit repository from GitHub:

$ git clone https://github.com/cohere-ai/cohere-toolkit.git

Step 3: Move into the Project Directory

Change into the cloned directory:

$ cd cohere-toolkit

Step 4: Configure Secrets

Copy the secrets.template.yaml file to create a new secrets file:

$ cp src/backend/config/secrets.template.yaml src/backend/config/secrets.yaml

Open the secrets file with a text editor to add your API key:

$ nano src/backend/config/secrets.yaml

Locate the following section and replace the example key with your actual Cohere API key:

deployments:
 cohere_platform:
  api_key: hMtMgJGNbtymq-example

Step 5: Set Up the Configuration File

Copy the configuration.template.yaml file to create a new configuration file:

$ cp src/backend/config/configuration.template.yaml src/backend/config/configuration.yaml

Configuring the Cohere Toolkit

The Cohere Toolkit provides a Makefile that simplifies setup by automating essential tasks such as compilation, initialization, testing, and environment configuration. Follow these steps to properly configure both the backend and frontend components.

Step 1: Navigate to the Toolkit Directory

Before executing any setup commands, make sure you are in the correct directory:

$ cd ~/cohere-toolkit

Step 2: Run the Initial Setup

Start the first-time setup process by executing:

$ sudo make first-run

If an error related to psycopg2 appears, such as:

make[1]: *** [Makefile:104: setup] Error 1
make[1]: Leaving directory '/home/linuxuser/cohere-toolkit'
make: *** [Makefile:128: first-run] Error 2

This happens because psycopg2 (2.9.10) does not support PEP 517 builds. The issue originates from the build backend and is unrelated to Poetry.

Step 3: Update pyproject.toml

To resolve this, open the pyproject.toml file in a text editor:

$ nano pyproject.toml

Locate the [tool.poetry.dependencies] section and comment out the existing psycopg2 entry, replacing it with psycopg2-binary:

[tool.poetry.dependencies]
python = "~3.11"
#psycopg2 = "^2.9.9"
psycopg2-binary = "^2.9.9"

Next, modify the [tool.poetry.group.setup.dependencies] section similarly:

[tool.poetry.group.setup.dependencies]
python = "~3.11"
#psycopg2 = "^2.9.9"
psycopg2-binary = "^2.9.9"

Once updated, save the file and close the editor.

Step 4: Lock the Dependencies

Run the following command to finalize dependency updates:

$ poetry lock

Step 5: Restart the Setup

Now, rerun the setup process to continue with the installation:

$ sudo make first-run

Step 6: Finalizing the Configuration

During the installation process, you will need to confirm several settings:

• Press Enter to keep the default backend API hostname:

Enter your public API Hostname or press enter for default [recommended]: http://localhost:8000

• Select the development environment as the build target:

Select the build target:
> dev
  prod

• Press Enter to disable the Python interpreter (it requires additional configuration).
• Press Enter to disable Tavily Internet Search unless you have a TAVILY_API_KEY.

Enter the value for TAVILY_API_KEY:

• Select Cohere Platform as the model deployment option and press Enter to save the configuration.

Select the model deployments you want to set up:
> [ ] Azure
  [ ] Bedrock
  [X] Cohere Platform
  [ ] SageMaker

Verifying the Configuration

If the setup was successful, you should see the following output:

backend-1    | INFO:     Started server process [9]
backend-1    | INFO:     Waiting for application startup.
backend-1    | INFO:     Application startup complete.

Cohere Toolkit Endpoints

The Cohere Toolkit is now running at the following addresses:

• Frontend: http://localhost:4000
• Backend: http://localhost:8000

Configuring Network Access

To allow external access to the Cohere Toolkit backend, open the required port:

$ sudo ufw allow 8000/tcp

If the firewall utility UFW is not installed, install it and allow SSH access using:

$ sudo apt install ufw -y && sudo ufw allow ssh

Finally, reload UFW to apply the new firewall rules:

$ sudo ufw reload

Making API Calls with Cohere Toolkit

The Cohere Platform model provides two API endpoints for interaction:

• Non-streaming requests: http://localhost:8000/v1/chat
• Streaming requests: http://localhost:8000/v1/chat-stream

Follow the steps below to make an API request to the Cohere backend.

Step 1: Sending a Request

Use a prompt, instruction, or query to send a request to the non-streaming endpoint. The following example demonstrates how to request information:

$ curl --location 'http://localhost:8000/v1/chat' \
--header 'User-Id: user-id' \
--header 'Content-Type: application/json' \
--data '{
  "message": "What is centron?"
}'

Conclusion

Congratulations! You have successfully installed the Cohere Toolkit on Ubuntu 24.04 and executed an API request to the Cohere backend. With this toolkit, you can leverage Cohere's Command, Embed, and Rerank models. Additionally, you can integrate it with external frameworks such as LangChain and LlamaIndex to build applications powered by Large Language Models (LLMs).

For further customization and detailed documentation, refer to the official Cohere Toolkit documentation.