midnight-ops-doctor

Notes from running a Midnight Network backend in production. Look up your symptom in the triage table, follow the link to the runbook.

Triage

Match the user's words against the left column. Load only the doc on the right. Don't pre-load others.

Routing rules:

• One symptom, one doc. Never broadcast-load.
• If the user describes two symptoms, fix the one that blocks the other first (sync before deploy, network before sync).
• If no row matches, ask one clarifying question. Don't guess.

Three addresses

The single most-mis-applied concept in Midnight backend code. Inline here so triage doesn't require loading a reference doc.

One seed, three addresses

A single MIDNIGHT_SEED (32-byte or 64-byte hex) derives three distinct addresses through HDWallet.fromSeed(seed).selectAccount(0).selectRoles([Roles.Zswap, Roles.NightExternal, Roles.Dust]).deriveKeysAt(0). The roles enum is exported by @midnight-ntwrk/wallet-sdk-hd. Each role yields a separate keypair with a different on-chain semantic.

import { HDWallet, Roles } from '@midnight-ntwrk/wallet-sdk-hd';
import {
  ZswapSecretKeys,
  DustSecretKey,
} from '@midnight-ntwrk/ledger-v8';
import { createKeystore, PublicKey } from '@midnight-ntwrk/wallet-sdk-address-format';
import { MidnightBech32m, DustAddress } from '@midnight-ntwrk/wallet-sdk-address-format';

const seed = Buffer.from(process.env.MIDNIGHT_SEED!, 'hex');
const hd = HDWallet.fromSeed(new Uint8Array(seed));
if (hd.type !== 'seedOk') throw new Error(`HD seed failed: ${hd.type}`);

const derived = hd.hdWallet
  .selectAccount(0)
  .selectRoles([Roles.Zswap, Roles.NightExternal, Roles.Dust] as const)
  .deriveKeysAt(0);
if (derived.type !== 'keysDerived') throw new Error(`derive failed: ${derived.type}`);

const networkId = 'preview'; // or 'preprod' / 'mainnet'

const zswapKeys = ZswapSecretKeys.fromSeed(derived.keys[Roles.Zswap]);
const dustSecret = DustSecretKey.fromSeed(derived.keys[Roles.Dust]);
const unshieldedKs = createKeystore(derived.keys[Roles.NightExternal], networkId);

const shieldedCoinPublicKey = zswapKeys.coinPublicKey;                     // 32-byte hex
const unshieldedAddress = unshieldedKs.getBech32Address().toString();      // mn_addr_<net>1...
const dustAddress = MidnightBech32m
  .encode(networkId, new DustAddress(dustSecret.publicKey))
  .toString();                                                             // mn_dust_<net>1...

hd.hdWallet.clear();

Address shapes

Operation, address, why

The getWalletAddress() trap

Many wallet adapter wrappers expose a getWalletAddress() method that returns ONLY the shielded coin public key (32-byte hex padded to 0x + 64 hex). Backend code that says "the wallet address" almost always means this one. It's correct for HTLC sender/receiver args, and wrong for everything balance-related (faucet, Lace, indexer).

When in doubt, expose all three on a diagnostics endpoint and pick by use case:

{
  shieldedCoinPublicKey: '0x4f1c...3a9b',
  unshieldedAddress: 'mn_addr_preview1...',
  dustAddress: 'mn_dust_preview1...',
}

Seed normalization gotcha

A 24-word BIP39 mnemonic can be normalized into a seed three ways. Only one matches Lace.

If your derived addresses don't match what Lace shows, you almost certainly used the wrong normalization. The bundled scripts/address-derive.mjs runs the canonical PBKDF2-full path.

For deeper diagnostics (faucet didn't arrive, balance still zero, address mismatch), load references/three-addresses.md.

Quick fixes

Top twelve errors. One-line diagnosis, minimum-viable fix. If the fix doesn't stick, escalate to the deeper doc.

Custom error: 139

Diagnosis: wallet submitted a transaction before chain sync completed. Node rejected stale UTXO inputs.

Fix:

await waitForWalletSyncState(WALLET_SYNC_TIMEOUT_MS, 'startup');
// only then: submitTx, deploy, lock, etc.

Deeper: references/wallet-lifecycle.md § Sync-completion check.

ERR_UNSUPPORTED_DIR_IMPORT from @midnight-ntwrk/*

Diagnosis: @midnight-ntwrk/* packages are ESM-only. CJS require() fails on bare-directory imports.

Fix:

// In a CJS file, use type-only static imports + dynamic import for runtime.
import type { WalletFacade } from '@midnight-ntwrk/wallet-sdk-facade';
const { WalletFacade } = await import('@midnight-ntwrk/wallet-sdk-facade');

Deeper: references/symptom-catalog.md § cjs-esm.

submitTx times out around 25-30s, tx is ~90 KB

Diagnosis: the SDK's HTTP submitTx posts to RPC. AWS WAF rejects bodies >8 KB. Deploy txs are typically 50-100 KB.

Fix: route submission through wallet.submitTransaction(tx). That uses the WebSocket relay (PolkadotNodeClient), bypassing the HTTP body limit.

const midnightProvider = {
  async submitTx(tx) { return wallet.submitTransaction(tx); },
};

Deeper: references/symptom-catalog.md § waf-8kb.

403 Forbidden from https://rpc.preprod.midnight.network (only on cloud VMs)

Diagnosis: