nlp-engineering

When this skill is activated, always start your first response with the 🧢 emoji.

NLP Engineering

A practical framework for building production NLP systems. This skill covers the full stack of natural language processing - from raw text ingestion through tokenization, embedding, retrieval, classification, and generation - with an emphasis on making the right architectural choices at each layer. Designed for engineers who know Python and ML basics and need opinionated guidance on building reliable, scalable text processing pipelines.

When to use this skill

Trigger this skill when the user:

• Builds a text preprocessing or cleaning pipeline
• Generates or stores embeddings for documents or queries
• Implements semantic search or similarity-based retrieval
• Classifies text into categories (sentiment, intent, topic, etc.)
• Extracts named entities, relationships, or structured data from text
• Summarizes long documents (extractive or abstractive)
• Chunks documents for RAG (Retrieval-Augmented Generation) pipelines
• Tunes tokenization strategies (BPE, wordpiece, whitespace)

Do NOT trigger this skill for:

• Pure LLM prompt engineering or chain-of-thought with no text processing pipeline
• Speech-to-text or image captioning (separate modalities with different toolchains)

Key principles

1. Preprocessing is load-bearing - Garbage in, garbage out. Inconsistent

casing, stray HTML, and unicode noise degrade every downstream component. Invest in a reproducible cleaning pipeline before touching a model.

1. Match the model to the task - A 66M-parameter sentence-transformer is

often better than GPT-4 embeddings for a narrow domain retrieval task, and 100x cheaper. Pick the smallest model that hits your quality bar.

1. Embed offline, search online - Pre-compute embeddings at index time.

Doing embedding + vector search in the request path is an avoidable latency sink. Only re-embed at write time (new docs) or on model upgrade.

1. Chunk with overlap, not just length - Fixed-length chunking without

overlap splits sentences at boundaries and degrades retrieval recall. Always use a sliding window with 10-20% overlap and respect sentence boundaries.

1. Evaluate before you ship - Define offline metrics (precision@k, NDCG,

ROUGE, F1) before building. An NLP system without evals is a system you cannot improve or regress-test.

Core concepts

Tokenization

Tokenization converts raw text into a sequence of tokens a model can process. Modern models use subword tokenizers (BPE, WordPiece, SentencePiece) rather than whitespace splitting, allowing them to handle out-of-vocabulary words gracefully by decomposing them into known subword units.

Key considerations: token budget (LLMs have context windows), language coverage (multilingual text needs a multilingual tokenizer), and domain vocabulary (medical/legal/code text may have poor tokenization with general-purpose tokenizers).

Embeddings

An embedding is a dense vector representation of text that encodes semantic meaning. Similar texts produce vectors with high cosine similarity. Embeddings are the foundation of semantic search, clustering, and classification.

Two categories: encoding models (sentence-transformers, E5, BGE) are fast, cheap, and purpose-built for retrieval. LLM embeddings (OpenAI text-embedding-3, Cohere Embed) are convenient API calls but cost money per token and introduce external latency.

Attention and transformers

Transformers process the full token sequence in parallel using self-attention, letting every token attend to every other token. This gives transformer-based models long-range context understanding that recurrent models lacked. For NLP tasks, you almost never need to implement attention from scratch - use HuggingFace transformers and fine-tune a pretrained checkpoint.

Vector similarity

Three distance metrics dominate:

| Metric | Formula (conceptual) | Best for | |---|---|---| | Cosine similarity | angle between vectors | Normalized embeddings, most retrieval | | Dot product | magnitude + angle | When vector magnitude carries information | | Euclidean distance | straight-line distance | Rare; prefer cosine for NLP |

Most vector stores (Pinecone, Weaviate, pgvector, FAISS) default to cosine or dot product. Normalize your embeddings before storing them to make cosine and dot product equivalent.

Common tasks

Text preprocessing pipeline

Build a reproducible cleaning pipeline before any modeling step. Apply in this order: decode -> strip HTML -> normalize unicode -> lowercase -> remove noise -> normalize whitespace.

import re
import unicodedata
from bs4 import BeautifulSoup

def preprocess(text: str, lowercase: bool = True) -> str:
    # 1. Decode HTML entities and strip tags
    text = BeautifulSoup(text, "html.parser").get_text(separator=" ")

    # 2. Normalize unicode (NFD -> NFC, remove combining chars if needed)
    text = unicodedata.normalize("NFC", text)

    # 3. Lowercase
    if lowercase:
        text = text.lower()

    # 4. Remove URLs, emails, special tokens
    text = re.sub(r"https?://\S+|www\.\S+", " ", text)
    text = re.sub(r"\S+@\S+\.\S+", " ", text)

    # 5. Collapse whitespace
    text = re.sub(r"\s+", " ", text).strip()

    return text

# Usage
clean = preprocess("<p>Visit https://example.com for more info.</p>")
# -> "visit for more info."

Persist the preprocessing config (lowercase flag, regex patterns) alongside

your model so training and inference use identical transformations.

Generate embeddings

Use sentence-transformers for local, cost-free embeddings or the OpenAI API for convenience. Always batch your calls.

# Option A: sentence-transformers (local, free, fast on GPU)
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")

documents = ["The quick brown fox", "Machine learning is fun", "NLP rocks"]

# encode() handles batching internally; show_progress_bar for large corpora
embeddings = model.encode(documents, normalize_embeddings=True, show_progress_bar=True)
# -> numpy array, shape (3, 384)

# Option B: OpenAI embeddings API
from openai import OpenAI

client = OpenAI()

def embed_batch(texts: list[str], model: str = "text-embedding-3-small") -> list[list[float]]:
    # Strip newlines - they degrade embedding quality per OpenAI docs
    texts = [t.replace("\n", " ") for t in texts]
    response = client.embeddings.create(input=texts, model=model)
    return [item.embedding for item in response.data]

Build semantic search

Index embeddings into a vector store and retrieve by cosine similarity at query time. This example uses FAISS for local search and pgvector for PostgreSQL.

import numpy as np
import faiss
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")

# --- Indexing ---
docs = ["Python is a programming language.", "The Eiffel Tower is in Paris.", ...]
doc_embeddings = model.encode(docs, normalize_embeddings=True).astype("float32")

# Inner product on normalized vectors = cosine similarity
index = faiss.IndexFlatIP(doc_embeddings.shape[1])
index.add(doc_embeddings)

# --- Retrieval ---
def search(query: str, top_k: int = 5) -> list[tuple[str, float]]:
    q_emb = model.encode([query], normalize_embeddings=True).astype("float32")
    scores, indices = index.search(q_emb, top_k)
    return [(docs[i], float(scores[0][j])) for j, i in enumerate(indices[0])]

results = search("programming languages for data science")
# -> [("Python is a programming language.", 0.87), ...]

For production, use faiss.IndexIVFFlat (approximate, faster) or a managed

vector store (pgvector, Pinecone, Weaviate) rather than exact IndexFlatIP.

Text classification with transformers

Fine-tune a pretrained encoder for sequence classification. HuggingFace transformers + `