backend-dev-guidelines

Backend Development Guidelines

Purpose

Establish consistency and best practices across Langfuse's backend packages (web, worker, packages/shared) using Next.js 14, tRPC, BullMQ, and TypeScript patterns.

When to Use This Skill

Automatically activates when working on:

• Creating or modifying tRPC routers and procedures
• Creating or modifying public API endpoints (REST)
• Creating or modifying BullMQ queue consumers and producers
• Building services with business logic
• Authenticating API requests
• Accessing resources based on entitlements
• Implementing middleware (tRPC, NextAuth, public API)
• Database operations with Prisma (PostgreSQL) or ClickHouse
• Observability with OpenTelemetry, DataDog, logger, and traceException
• Input validation with Zod v4
• Environment configuration from env variables
• Backend testing and refactoring

Quick Start

UI: New tRPC Feature Checklist (Web)

• [ ] Router: Define in features/[feature]/server/*Router.ts
• [ ] Procedures: Use appropriate procedure type (protected, public)
• [ ] Authentication: Use JWT authorization via middlewares.
• [ ] Entitlement check: Access resources based on resource and role
• [ ] Validation: Zod v4 schema for input
• [ ] Service: Business logic in service file
• [ ] Error handling: Use traceException wrapper
• [ ] Tests: Unit + integration tests in __tests__/
• [ ] Config: Access via env.mjs

SDKs: New Public API Endpoint Checklist (Web)

• [ ] Route file: Create in pages/api/public/
• [ ] Wrapper: Use withMiddlewares + createAuthedProjectAPIRoute
• [ ] Types: Define in features/public-api/types/
• [ ] Authentication: Authorization via basic auth
• [ ] Validation: Zod schemas for query/body/response
• [ ] Versioning: Versioning in API path and Zod schemas for query/body/response
• [ ] Tests: Add end-to-end test in __tests__/async/

New Queue Processor Checklist (Worker)

• [ ] Processor: Create in worker/src/queues/
• [ ] Queue types: Create queue types in packages/shared/src/server/queues
• [ ] Service: Business logic in features/ or worker/src/features/
• [ ] Error handling: Distinguish between errors which should fail queue processing and errors which should result in a succeeded event.
• [ ] Queue registration: Add to WorkerManager in app.ts
• [ ] Tests: Add vitest tests in worker

Architecture Overview

Layered Architecture

# Web Package (Next.js 14)

┌─ tRPC API ──────────────────┐   ┌── Public REST API ──────────┐
│                             │   │                             │
│  HTTP Request               │   │  HTTP Request               │
│      ↓                      │   │      ↓                      │
│  tRPC Procedure             │   │  withMiddlewares +          │
│  (protectedProjectProcedure)│   │  createAuthedProjectAPIRoute│
│      ↓                      │   │      ↓                      │
│  Service (business logic)   │   │  Service (business logic)   │
│      ↓                      │   │      ↓                      │
│  Prisma / ClickHouse        │   │  Prisma / ClickHouse        │
│                             │   │                             │
└─────────────────────────────┘   └─────────────────────────────┘
                 ↓
            [optional]: Publish to Redis BullMQ queue
                 ↓
┌─ Worker Package (Express) ──────────────────────────────────┐
│                                                             │
│  BullMQ Queue Job                                           │
│      ↓                                                      │
│  Queue Processor (handles job)                              │
│      ↓                                                      │
│  Service (business logic)                                   │
│      ↓                                                      │
│  Prisma / ClickHouse                                        │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Key Principles:

• Web: tRPC procedures for UI OR public API routes for SDKs → Services → Database
• Worker: Queue processors → Services → Database
• packages/shared: Shared code for Web and Worker

See [architecture-overview.md](architecture-overview.md) for complete details.

Directory Structure

Web Package (/web/)

web/src/
├── features/                # Feature-organized code
│   ├── [feature-name]/
│   │   ├── server/          # Backend logic
│   │   │   ├── *Router.ts   # tRPC router
│   │   │   └── service.ts   # Business logic
│   │   ├── components/      # React components
│   │   └── types/           # Feature types
├── server/
│   ├── api/
│   │   ├── routers/         # tRPC routers
│   │   ├── trpc.ts          # tRPC setup & middleware
│   │   └── root.ts          # Main router
│   ├── auth.ts              # NextAuth.js config
│   └── db.ts                # Database client
├── pages/
│   ├── api/
│   │   ├── public/          # Public REST APIs
│   │   └── trpc/            # tRPC endpoint
│   └── [routes].tsx         # Next.js pages
├── __tests__/               # Jest tests
│   └── async/               # Integration tests
├── instrumentation.ts       # OpenTelemetry (FIRST IMPORT)
└── env.mjs                  # Environment config

Worker Package (/worker/)

worker/src/
├── queues/                  # BullMQ processors
│   ├── evalQueue.ts
│   ├── ingestionQueue.ts
│   └── workerManager.ts
├── features/                # Business logic
│   └── [feature]/
│       └── service.ts
├── instrumentation.ts       # OpenTelemetry (FIRST IMPORT)
├── app.ts                   # Express setup + queue registration
├── env.ts                   # Environment config
└── index.ts                 # Server start

Shared Package (/packages/shared/)

shared/src/
├── server/                  # Server utilities
│   ├── auth/                # Authentication helpers
│   ├── clickhouse/          # ClickHouse client & schema
│   ├── instrumentation/     # OpenTelemetry helpers
│   ├── llm/                 # LLM integration utilities
│   ├── redis/               # Redis queues & cache
│   ├── repositories/        # Data repositories
│   ├── services/            # Shared services
│   ├── utils/               # Server utilities
│   ├── logger.ts
│   └── queues.ts
├── encryption/              # Encryption utilities
├── features/                # Feature-specific code
├── tableDefinitions/        # Table schemas
├── utils/                   # Shared utilities
├── constants.ts
├── db.ts                    # Prisma client
├── env.ts                   # Environment config
└── index.ts                 # Main exports

Import Paths (package.json exports):

The shared package exposes specific import paths for different use cases:

| Import Path | Maps To | Use For | | ------------------------------------------ | --------------------------------- | --------------------------------------------------------------- | | @langfuse/shared | dist/src/index.js | General types, schemas, utilities, constants | | @langfuse/shared/src/db | dist/src/db.js | Prisma client and database types | | @langfuse/shared/src/server | dist/src/server/index.js | Server-side utilities (queues, auth, services, instrumentation) | | @langfuse/shared/src/server/auth/apiKeys | dist/src/server/auth/apiKeys.js | API key management utilities | | @langfuse/shared/encryption | dist/src/encryption/index.js | Encryption and signature utilities |

Usage Examples: