Layer 3 disaster recovery runbook

Restore a self-hosted MPC signer that was lost along with its key-share database, when point-in-time recovery (PITR) and database snapshots cannot bring the database back. You rebuild the signer and restore its key shares from the Layer 3 backup. This applies to self-hosted deployments where you run the signer’s database and backup bucket. On DFNS Cloud, DFNS handles recovery for you.

What Layer 3 is

Each time a signer creates a key share, it writes the same encrypted bytes to two places: its database and your Layer 3 backup bucket. The backup is an exact copy of the encrypted rows. Recovery copies those shares back into a fresh database. They stay encrypted throughout, so no key material is ever exposed.

The backup holds the encrypted shares, not the key that decrypts them. That key is the signer’s Core Keys Pack, stored in AWS Secrets Manager, separate from the database.If the Core Keys Pack is also lost, restore it from its backup first. Without it the shares cannot be decrypted, and you must recover through Layer 4 instead.

Recovery procedure

Confirm the Core Keys Pack is intact

The signer needs its Core Keys Pack to decrypt the restored shares. Confirm the secret still exists before you start:

aws secretsmanager describe-secret --secret-id <core-keys-pack-secret-id>

If it is present, continue. You do not need to open or export it, only confirm it exists.If it is gone, restore it from backup first. If you have no backup, use Layer 4 instead.

Redeploy the missing signer

Redeploy the missing signer using your normal MPC signer deployment process. Contact our if you need help with this step.

The new signer must reuse the original Core Keys Pack: point it at the same core-keys secret as before (Step 1), so it loads its original keys instead of generating new ones.

The new signer comes up with its original identity and an empty database, ready to restore the shares into.

Grant the recovery role read access

The backup bucket is write-only, so grant your recovery role read access for the restore:

aws iam put-role-policy \
  --role-name <recovery-role> \
  --policy-name l3-recovery-read \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::<l3-backup-bucket>" },
      { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::<l3-backup-bucket>/*" }
    ]
  }'

Download the shares

Download every object under your signer’s prefix to local files. Each object’s name is a key_id and its body is the encrypted share.

set -euo pipefail
BUCKET=<l3-backup-bucket>
PREFIX=<signer-prefix>
DEST=./l3-shares

aws s3 sync "s3://$BUCKET/$PREFIX/" "$DEST/"

# The local file count should match the number of objects in the bucket.
aws s3 ls "s3://$BUCKET/$PREFIX/" --recursive | wc -l
find "$DEST" -type f | wc -l

Remove the recovery role's read access

With the shares downloaded, remove the read access so the bucket is write-only again:

aws iam delete-role-policy --role-name <recovery-role> --policy-name l3-recovery-read

Insert the shares into the database

Insert the shares into key_shares, connected as the signer’s own login role.

Connect as the signer’s login role, not a superuser. Row-level security ties each row to the role that inserts it, so restoring as any other role leaves the signer unable to see the shares.

set -euo pipefail
PGURL=<postgres-url>   # connect as the signer's login role
DEST=./l3-shares

find "$DEST" -type f -print0 | while IFS= read -r -d '' f; do
  key_id=$(basename "$f")
  hex=$(xxd -p "$f" | tr -d '\n')
  printf "insert into key_shares (key_id, encrypted_share) values ('%s', decode('%s', 'hex'));\n" \
    "$key_id" "$hex" | psql "$PGURL" -v ON_ERROR_STOP=1 -f -
done

Confirm the row count matches the number of files:

find "$DEST" -type f | wc -l
psql "$PGURL" -c "select count(*) from key_shares;"

Confirm signing resumes

The signer picks up the restored shares automatically, with no restart. Confirm with a test signature (see Validation). If signing does not resume, contact our .

Validation

Row count: key_shares matches the number of objects in the bucket (Step 6).
Byte match: for a few shares, the stored bytes match the downloaded file.

psql "$PGURL" -tAc "select md5(encrypted_share) from key_shares where key_id = '<key_id>';"
md5sum "$DEST/<key_id>"   # macOS: md5 -q "$DEST/<key_id>"

Test signature: sign with a recovered key on a testnet and confirm it broadcasts. A recovered key controls the same address on every chain, so use a testnet so no real assets move.

Introduction

Core Concepts

Platform Features

Networks

Solutions

Guides

Advanced

Layer 3 disaster recovery runbook

What Layer 3 is

Recovery procedure

Validation

​What Layer 3 is

​Recovery procedure

​Validation

What Layer 3 is

Recovery procedure

Validation