Pinecone
The complete guide

What is a Vector Database — and why Pinecone?

A Vector Database is a database built specifically for storing and searching Embeddings — numeric representations of text, images, video and audio. Unlike SQL, which searches by exact values, a Vector DB searches by semantic proximity — it finds the items most semantically similar to your query.

Pinecone is the most popular Vector Database — Managed Service that requires no infrastructure, with a generous Free Tier and Serverless that fits any size. It's the foundation of every RAG (Retrieval Augmented Generation) system.

Core concepts — Vectors, Embeddings and Similarity

Before you start coding, it's important to understand the basic concepts:

What is an Embedding?

An Embedding is a numeric representation of text — an array of decimal numbers (for example, 1536 dimensions in OpenAI text-embedding-3-small). The Embedding model turns each text into a point in a multidimensional mathematical space, where texts with similar meaning are close to each other.

# a simplified visualization of Embeddings:
# "dog"       → [0.23, -0.45, 0.12, ...]  # 1536 dimensions
# "puppy"     → [0.24, -0.44, 0.11, ...]  # very close!
# "cat"       → [0.20, -0.40, 0.15, ...]  # close (animal)
# "bank"      → [-0.12, 0.89, -0.34, ...] # far away

Similarity Metrics

Index Types — Serverless vs Pod

• Serverless: Pay-per-use, no infrastructure management, suited to most projects. Recommended to start.
• Pod-based: Reserved capacity, lower latency, suited to Production with a strict SLA.

Setup & First Index

Getting started with Pinecone is simple: you sign up atapp.pinecone.io, get an API Key, and install the SDK.

pip install pinecone openai python-dotenv

from pinecone import Pinecone, ServerlessSpec
import os
from dotenv import load_dotenv

load_dotenv()
pc = Pinecone(api_key=os.getenv("PINECONE_API_KEY"))

# create an Index (once)
pc.create_index(
    name="automation4mi",
    dimension=1536,        # OpenAI text-embedding-3-small
    metric="cosine",       # cosine / euclidean / dotproduct
    spec=ServerlessSpec(
        cloud="aws",
        region="us-east-1"
    )
)

# connect to the Index
index = pc.Index("automation4mi")
print(index.describe_index_stats())

play_circle

Pinecone Vector Database — RAG Tutorial Python 2024

YouTube • search for tutorials

open_in_new

Upsert — adding Vectors to the store

Upsert = "Insert or Update" — if a Vector with this ID exists, it will be updated; if not — it will be added. Always send in Batches of 100 for optimal performance.

from openai import OpenAI

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def embed(text: str) -> list[float]:
    return client.embeddings.create(
        input=text,
        model="text-embedding-3-small"
    ).data[0].embedding

# preparing the Vectors
vectors = [
    {
        "id": "doc1",
        "values": embed("The n8n automation guide"),
        "metadata": {
            "source": "guide-n8n",
            "title": "n8n Automation Guide",
            "category": "automation",
            "lang": "en"
        }
    },
    {
        "id": "doc2",
        "values": embed("Stable Diffusion ComfyUI workflows"),
        "metadata": {
            "source": "guide-sd",
            "title": "Stable Diffusion Guide",
            "category": "image-ai",
            "lang": "en"
        }
    },
    {
        "id": "doc3",
        "values": embed("Pinecone Vector Database RAG"),
        "metadata": {
            "source": "pinecone-guide",
            "title": "Pinecone Guide",
            "category": "vector-db",
            "lang": "en"
        }
    }
]

# Upsert to a specific namespace
index.upsert(vectors=vectors, namespace="guides")
print(f"Upserted {len(vectors)} vectors")

Batch Upsert for a large corpus

import time

def batch_upsert(documents: list[dict], namespace: str = "default", batch_size: int = 100):
    """Upsert documents in batches for good performance"""
    vectors = []
    for doc in documents:
        embedding = embed(doc["text"])
        vectors.append({
            "id": doc["id"],
            "values": embedding,
            "metadata": {
                "text": doc["text"][:1000],  # keep up to 1000 characters in metadata
                **doc.get("metadata", {})
            }
        })

    # send in batches
    for i in range(0, len(vectors), batch_size):
        batch = vectors[i:i + batch_size]
        index.upsert(vectors=batch, namespace=namespace)
        print(f"Upserted batch {i//batch_size + 1}/{(len(vectors)-1)//batch_size + 1}")
        time.sleep(0.1)  # protection from a rate limit

Query — Semantic Search

At the Query stage, you take a question, turn it into an Embedding, and search Pinecone for the closest Vectors.

def semantic_search(query: str, top_k: int = 5, namespace: str = "guides") -> list[dict]:
    """Search for the documents most similar to the query"""
    query_embedding = embed(query)

    results = index.query(
        vector=query_embedding,
        top_k=top_k,
        include_metadata=True,
        namespace=namespace
    )

    matches = []
    for match in results.matches:
        matches.append({
            "text":   match.metadata.get("text", ""),
            "title":  match.metadata.get("title", ""),
            "source": match.metadata.get("source", ""),
            "score":  round(match.score, 3),
            "id":     match.id
        })
    return matches

# usage:
results = semantic_search("How do you set up a Webhook in n8n?")
for r in results:
    print(f"[{r['score']}] {r['title']}: {r['text'][:80]}...")

Metadata Filtering

Combine vector search with filtering by Metadata — search only in a defined subset of documents for higher accuracy.

# filtering by Metadata
results = index.query(
    vector=query_embedding,
    top_k=10,
    filter={
        "lang":     {"$eq": "he"},
        "category": {"$in": ["automation", "ai"]},
    },
    include_metadata=True,
    namespace="guides"
)

# available Operators:
# $eq, $ne      — equality / inequality
# $gt, $gte     — greater / greater-or-equal
# $lt, $lte     — less / less-or-equal
# $in, $nin     — in / not in a list
# $exists       — field exists
# $and, $or     — logical

Namespaces — multi-tenant organization

A Namespace is a logical partition within an Index. Each Namespace is fully independent — you can search, add and delete without affecting other Namespaces. Perfect for Multi-tenant applications.

# Multi-tenant: a Namespace per user
index.upsert(vectors=user_a_docs, namespace="user-alice-123")
index.upsert(vectors=user_b_docs, namespace="user-bob-456")

# searching in a specific Namespace — perfect isolation
results_alice = index.query(
    vector=query_embedding,
    namespace="user-alice-123",
    top_k=5
)

# statistics per Namespace
stats = index.describe_index_stats()
for ns, info in stats.namespaces.items():
    print(f"{ns}: {info.vector_count} vectors")

# deleting all the data of a namespace
index.delete(delete_all=True, namespace="user-alice-123")

Hybrid Search — Dense + Sparse

Hybrid Search combines Dense (Embeddings — semantic) andSparse (BM25 — keywords). Dense is excellent for meaning; Sparse is excellent for exact words. The combination gives the best of both worlds.

from pinecone_text.sparse import BM25Encoder

# train BM25 on your corpus
bm25 = BM25Encoder()
bm25.fit([doc["text"] for doc in your_documents])

# Upsert with Sparse + Dense
def upsert_hybrid(doc: dict):
    dense = embed(doc["text"])
    sparse = bm25.encode_documents(doc["text"])

    index.upsert(vectors=[{
        "id": doc["id"],
        "values": dense,
        "sparse_values": sparse,
        "metadata": {"text": doc["text"]}
    }])

# Query with Hybrid
def hybrid_query(query: str, alpha: float = 0.5):
    """alpha=1: dense only, alpha=0: sparse only, 0.5: balanced"""
    dense_vec  = embed(query)
    sparse_vec = bm25.encode_queries(query)

    # manual weighting
    def scale(sparse, alpha):
        return {"indices": sparse["indices"],
                "values":  [v * (1-alpha) for v in sparse["values"]]}

    return index.query(
        vector=[v * alpha for v in dense_vec],
        sparse_vector=scale(sparse_vec, alpha),
        top_k=5,
        include_metadata=True
    )

When is Hybrid Search worth it?

• Code and technical terms — "TypeError Python" = Dense struggles, Sparse succeeds excellently.
• Unique proper nouns — product names, people's names, acronyms.
• Mixed language — text with terms in English.
• Search-as-you-type — search-as-you-type with partial words.

A full RAG Pipeline with LangChain

Here's a full RAG Pipeline connecting Pinecone with LangChain and GPT-4o — a production-ready structure:

from langchain_pinecone import PineconeVectorStore
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate
import os

# 1. Embeddings + VectorStore
embeddings  = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = PineconeVectorStore(
    index_name="automation4mi",
    embedding=embeddings,
    namespace="guides"
)

# 2. Retriever
retriever = vectorstore.as_retriever(
    search_type="similarity_score_threshold",
    search_kwargs={
        "k": 4,
        "score_threshold": 0.70,
        "namespace": "guides"
    }
)

# 3. Custom Prompt
prompt = PromptTemplate(
    input_variables=["context", "question"],
    template="""You are an AI assistant for Automation4MI.
Answer the question based only on the following context.
If there isn't enough information — say so explicitly.

Context:
{context}

Question: {question}

Answer:"""
)

# 4. QA Chain
qa = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(model="gpt-4o", temperature=0.1),
    chain_type="stuff",
    retriever=retriever,
    return_source_documents=True,
    chain_type_kwargs={"prompt": prompt}
)

# 5. usage
result = qa.invoke({"query": "How do you use ControlNet?"})
print(result["result"])
print("\nSources:")
for doc in result["source_documents"]:
    print(f"  - {doc.metadata.get('title', 'no title')}: score={doc.metadata.get('score','?')}")

Manual RAG (without LangChain)

from openai import OpenAI

openai_client = OpenAI()

def rag_query(question: str, namespace: str = "guides") -> str:
    # step 1: Embed the question
    q_embedding = embed(question)

    # step 2: search Pinecone
    results = index.query(
        vector=q_embedding,
        top_k=4,
        include_metadata=True,
        namespace=namespace
    )

    # step 3: build the context
    context = "\n\n---\n\n".join([
        f"[{m.metadata.get('title','')}]\n{m.metadata.get('text','')}"
        for m in results.matches if m.score > 0.65
    ])

    if not context:
        return "No relevant information found in the knowledge base."

    # step 4: send to the LLM
    response = openai_client.chat.completions.create(
        model="gpt-4o",
        temperature=0.1,
        messages=[
            {"role": "system", "content": "Answer based only on the context."},
            {"role": "user",   "content": f"Context:\n{context}\n\nQuestion: {question}"}
        ]
    )
    return response.choices[0].message.content

print(rag_query("What is the difference between n8n and Make?"))

5 practical projects

Cheat sheet — everything in one place

Index Configuration

Main SDK Methods

Pricing (April 2026)

Metadata Best Practices

• Store the original text — always metadata["text"] for retrieval at Query time.
• Add source — where the document comes from, for attribution and filtering.
• Timestamps — created_at and updated_at for filtering by date.
• Don't store large Vectors in Metadata — Metadata is limited to 40KB per Vector.
• Normalize values — category as a lowercase tag, lang in ISO 639-1.