Skip to content

qdrant/qdrant-kafka

Repository files navigation

Qdrant Sink Connector

Use Qdrant as a sink destination in Kafka Connect. Supports streaming dense/sparse/multi vectors into Qdrant collections.

Usage

Important

Qdrant collections have to be created in advance with the appropriate vector dimensions and configurations.

Learn to use the connector with

Configuration

Name Type Default Description
qdrant.grpc.url string http://localhost:6334 Qdrant gRPC URL
qdrant.api.key string null API key for authentication
qdrant.collection.name string null Target collection name. If set, overrides the collection_name field in records.

Message Formats

This sink connector supports messages with multiple dense/sparse vectors.

Click each to expand.

Unnamed/Default vector

Reference: Creating a collection with a default vector.

{
    "collection_name": "{collection_name}",
    "id": 1,
    "vector": [
        0.1,
        0.2,
        0.3,
        0.4,
        0.5,
        0.6,
        0.7,
        0.8
    ],
    "payload": {
        "name": "kafka",
        "description": "Kafka is a distributed streaming platform",
        "url": "https://kafka.apache.org/"
    }
}
Named multiple vectors

Reference: Creating a collection with multiple vectors.

{
    "collection_name": "{collection_name}",
    "id": 1,
    "vector": {
        "some-dense": [
            0.1,
            0.2,
            0.3,
            0.4,
            0.5,
            0.6,
            0.7,
            0.8
        ],
        "some-other-dense": [
            0.1,
            0.2,
            0.3,
            0.4,
            0.5,
            0.6,
            0.7,
            0.8
        ]
    },
    "payload": {
        "name": "kafka",
        "description": "Kafka is a distributed streaming platform",
        "url": "https://kafka.apache.org/"
    }
}
Sparse vectors

Reference: Creating a collection with sparse vectors.

{
    "collection_name": "{collection_name}",
    "id": 1,
    "vector": {
        "some-sparse": {
            "indices": [
                0,
                1,
                2,
                3,
                4,
                5,
                6,
                7,
                8,
                9
            ],
            "values": [
                0.1,
                0.2,
                0.3,
                0.4,
                0.5,
                0.6,
                0.7,
                0.8,
                0.9,
                1.0
            ]
        }
    },
    "payload": {
        "name": "kafka",
        "description": "Kafka is a distributed streaming platform",
        "url": "https://kafka.apache.org/"
    }
}
Multi-vector
{
    "collection_name": "{collection_name}",
    "id": 1,
    "vector": {
        "some-multi": [
            [
                0.1,
                0.2,
                0.3,
                0.4,
                0.5,
                0.6,
                0.7,
                0.8,
                0.9,
                1.0
            ],
            [
                1.0,
                0.9,
                0.8,
                0.5,
                0.4,
                0.8,
                0.6,
                0.4,
                0.2,
                0.1
            ]
        ]
    },
    "payload": {
        "name": "kafka",
        "description": "Kafka is a distributed streaming platform",
        "url": "https://kafka.apache.org/"
    }
}
Combination of named dense and sparse vectors

Reference:

{
    "collection_name": "{collection_name}",
    "id": "a10435b5-2a58-427a-a3a0-a5d845b147b7",
    "vector": {
        "some-other-dense": [
            0.1,
            0.2,
            0.3,
            0.4,
            0.5,
            0.6,
            0.7,
            0.8
        ],
        "some-sparse": {
            "indices": [
                0,
                1,
                2,
                3,
                4,
                5,
                6,
                7,
                8,
                9
            ],
            "values": [
                0.1,
                0.2,
                0.3,
                0.4,
                0.5,
                0.6,
                0.7,
                0.8,
                0.9,
                1.0
            ]
        }
    },
    "payload": {
        "name": "kafka",
        "description": "Kafka is a distributed streaming platform",
        "url": "https://kafka.apache.org/"
    }
}

LICENSE

Apache 2.0

About

Kafka Sink connector for Qdrant. Stream vector data into a Qdrant collection. Supports named/unnamed dense/sparse vectors.

Resources

License

Stars

Watchers

Forks

Packages

 
 
 

Contributors

Languages