> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dfns.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Layer 3 disaster recovery runbook

> Step-by-step procedure to restore a self-hosted MPC signer after total loss of its key-share database, by recreating the key shares rows from the Layer 3 S3 backup.

export const SupportLink = ({children}) => {
  const url = "https://support.dfns.co";
  return <a href={url} target="_blank">{children || url}</a>;
};

Restore a self-hosted MPC signer that was lost along with its key-share database, when point-in-time recovery (PITR) and database snapshots cannot bring the database back. You rebuild the signer and restore its key shares from the Layer 3 backup.

This applies to self-hosted deployments where you run the signer's database and backup bucket. On DFNS Cloud, DFNS handles recovery for you.

## What Layer 3 is

Each time a signer creates a key share, it writes the same encrypted bytes to two places: its database and your Layer 3 backup bucket. The backup is an exact copy of the encrypted rows.

Recovery copies those shares back into a fresh database. They stay encrypted throughout, so no key material is ever exposed.

<Warning>
  The backup holds the encrypted shares, not the key that decrypts them. That key is the signer's Core Keys Pack, stored in AWS Secrets Manager, separate from the database.

  If the Core Keys Pack is also lost, restore it from its backup first. Without it the shares cannot be decrypted, and you must recover through [Layer 4](/advanced/deployment-models/disaster-recovery#layer-4-disaster-recovery) instead.
</Warning>

## Recovery procedure

<Steps>
  <Step title="Confirm the Core Keys Pack is intact">
    The signer needs its Core Keys Pack to decrypt the restored shares. Confirm the secret still exists before you start:

    ```bash theme={null}
    aws secretsmanager describe-secret --secret-id <core-keys-pack-secret-id>
    ```

    If it is present, continue. You do not need to open or export it, only confirm it exists.

    If it is gone, restore it from backup first. If you have no backup, use [Layer 4](/advanced/deployment-models/disaster-recovery#layer-4-disaster-recovery) instead.
  </Step>

  <Step title="Redeploy the missing signer">
    Redeploy the missing signer using your normal MPC signer deployment process. Contact our <SupportLink>Support Team</SupportLink> if you need help with this step.

    <Note>
      The new signer must reuse the original Core Keys Pack: point it at the same core-keys secret as before (Step 1), so it loads its original keys instead of generating new ones.
    </Note>

    The new signer comes up with its original identity and an empty database, ready to restore the shares into.
  </Step>

  <Step title="Grant the recovery role read access">
    The backup bucket is write-only, so grant your recovery role read access for the restore:

    ```bash theme={null}
    aws iam put-role-policy \
      --role-name <recovery-role> \
      --policy-name l3-recovery-read \
      --policy-document '{
        "Version": "2012-10-17",
        "Statement": [
          { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::<l3-backup-bucket>" },
          { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::<l3-backup-bucket>/*" }
        ]
      }'
    ```
  </Step>

  <Step title="Download the shares">
    Download every object under your signer's prefix to local files. Each object's name is a `key_id` and its body is the encrypted share.

    ```bash theme={null}
    set -euo pipefail
    BUCKET=<l3-backup-bucket>
    PREFIX=<signer-prefix>
    DEST=./l3-shares

    aws s3 sync "s3://$BUCKET/$PREFIX/" "$DEST/"

    # The local file count should match the number of objects in the bucket.
    aws s3 ls "s3://$BUCKET/$PREFIX/" --recursive | wc -l
    find "$DEST" -type f | wc -l
    ```
  </Step>

  <Step title="Remove the recovery role's read access">
    With the shares downloaded, remove the read access so the bucket is write-only again:

    ```bash theme={null}
    aws iam delete-role-policy --role-name <recovery-role> --policy-name l3-recovery-read
    ```
  </Step>

  <Step title="Insert the shares into the database">
    Insert the shares into `key_shares`, connected as the signer's own login role.

    <Warning>
      Connect as the signer's login role, not a superuser. Row-level security ties each row to the role that inserts it, so restoring as any other role leaves the signer unable to see the shares.
    </Warning>

    ```bash theme={null}
    set -euo pipefail
    PGURL=<postgres-url>   # connect as the signer's login role
    DEST=./l3-shares

    find "$DEST" -type f -print0 | while IFS= read -r -d '' f; do
      key_id=$(basename "$f")
      hex=$(xxd -p "$f" | tr -d '\n')
      printf "insert into key_shares (key_id, encrypted_share) values ('%s', decode('%s', 'hex'));\n" \
        "$key_id" "$hex" | psql "$PGURL" -v ON_ERROR_STOP=1 -f -
    done
    ```

    Confirm the row count matches the number of files:

    ```bash theme={null}
    find "$DEST" -type f | wc -l
    psql "$PGURL" -c "select count(*) from key_shares;"
    ```
  </Step>

  <Step title="Confirm signing resumes">
    The signer picks up the restored shares automatically, with no restart. Confirm with a test signature (see [Validation](#validation)). If signing does not resume, contact our <SupportLink>Support Team</SupportLink>.
  </Step>
</Steps>

## Validation

* Row count: `key_shares` matches the number of objects in the bucket (Step 6).
* Byte match: for a few shares, the stored bytes match the downloaded file.

```bash theme={null}
psql "$PGURL" -tAc "select md5(encrypted_share) from key_shares where key_id = '<key_id>';"
md5sum "$DEST/<key_id>"   # macOS: md5 -q "$DEST/<key_id>"
```

* Test signature: sign with a recovered key on a testnet and confirm it broadcasts. A recovered key controls the same address on every chain, so use a testnet so no real assets move.