agberohq/keeper: Simple Secure Keeper for Secrets · GitHub

Keeper is a cryptographic secret store for Go. It encrypts arbitrary byte
payloads at rest using Argon2id key derivation and XChaCha20-Poly1305 (default)
authenticated encryption, and stores them in an embedded bbolt database.

It ships as three things you can use independently:

• A Go library — embed a hardened secret store directly in your process,
  with four security levels, per-bucket DEK isolation, and a tamper-evident
  audit chain.
• An HTTP handler (x/keephandler) — mount keeper endpoints on any
  net/http mux in one call, with pluggable hooks, guards, and response
  encoders for access control and audit logging.
• A CLI (cmd/keeper) — a terminal interface with a persistent REPL
  session, no-echo secret entry, and zero shell-history exposure.

Keeper was designed as the foundational secret management layer for the Agbero
load balancer but has no dependency on Agbero and works in any Go project.

Keeper partitions secrets into buckets. Every bucket has an immutable
BucketSecurityPolicy that governs how its Data Encryption Key (DEK) is
protected. Four levels are available.

The URI scheme (vault://, certs://, space://, or any name you register)
is independent of the security level. A scheme is just a namespace prefix that
groups related buckets. The security level is a property of the
BucketSecurityPolicy set at CreateBucket time and cannot be changed
afterwards. You can mix security levels freely within the same scheme.

The bucket DEK is derived from the master key using HKDF-SHA256 with a
domain-separated info string per bucket (keeper-bucket-dek-v1:scheme:namespace).
All LevelPasswordOnly buckets are unlocked automatically when
UnlockDatabase is called with the correct master passphrase. No per-bucket
credential is required at runtime. This level is appropriate for secrets the
process needs at startup without human interaction.

The bucket has a randomly generated 32-byte DEK unique to that bucket. The DEK
is never stored in plaintext. For each authorised admin a Key Encryption Key
(KEK) is derived from HKDF(masterKey‖adminCred, dekSalt) and used to wrap
the DEK via XChaCha20-Poly1305. The bucket is inaccessible until an admin calls
UnlockBucket with their credential. The master passphrase alone cannot
decrypt the bucket. Revoking one admin does not affect any other admin’s wrapped
copy.

The bucket DEK is generated at CreateBucket time and immediately wrapped by
a caller-supplied HSMProvider. The provider performs the wrap and unwrap
operations — keeper never handles the raw DEK after handing it to the provider.
UnlockDatabase automatically calls the provider to unwrap and seed the
Envelope for all registered HSM buckets. Master key rotation does not
re-encrypt these buckets; the DEK is provider-controlled.

A built-in SoftHSM implementation backed by a memguard-protected wrapping
key is available in pkg/hsm for testing and CI environments. Do not use it
in production.

Identical to LevelHSM in key management behaviour, but the HSMProvider is
implemented by pkg/remote.Provider — a configurable HTTPS adapter that
delegates wrap and unwrap to any remote KMS service over TLS. Pre-built
configurations for HashiCorp Vault Transit, AWS KMS, and GCP Cloud KMS are
provided in pkg/remote. For production use, configure TLSClientCert and
TLSClientKey to enable mutual TLS authentication.

salt ← random 32 bytes, generated once, stored as a versioned SaltStore (unencrypted)
masterKey ← Argon2id(passphrase, salt, t=3, m=64 MiB, p=4) → 32 bytes

A verification hash is stored on first derivation:

verifyHash ← Argon2id(masterKey, "verification", t=1, m=64 MiB, p=4) → 32 bytes

Subsequent DeriveMaster calls recompute this hash and compare it with
crypto/subtle.ConstantTimeCompare. A mismatch returns ErrInvalidPassphrase.

The KDF salt is stored unencrypted by design. It must be readable before
UnlockDatabase to derive the master key — encrypting it with a key derived
from the master would be circular. A KDF salt is not a secret; its purpose is
uniqueness, not confidentiality.

Each plaintext value is encrypted with XChaCha20-Poly1305 using the bucket DEK:

nonce ← random 24 bytes
ciphertext ← XChaCha20-Poly1305.Seal(nonce, DEK, plaintext)

The stored record is a msgpack-encoded Secret struct containing the
ciphertext, encrypted metadata, and schema version. Authentication is implicit:
a ciphertext decrypted with the wrong key produces an AEAD authentication
failure before any plaintext is returned.

salt ← random 32 bytes, generated at bucket creation, stored in policy
ikm ← masterKey ‖ adminCredential
KEK ← HKDF-SHA256(ikm, salt, info="keeper-kek-v1") → 32 bytes
wrappedDEK ← XChaCha20-Poly1305.Seal(nonce, KEK, DEK)

The KEK is derived using HKDF rather than a second Argon2 pass. The master key
was already produced by a high-cost KDF; a second Argon2 invocation would add
hundreds of milliseconds of latency to every UnlockBucket call with no
security benefit. HKDF-SHA256 operates in approximately one microsecond.

The neither-alone property holds: an attacker who compromises only the database
obtains the wrapped DEK and the HKDF salt but cannot derive the KEK without the
master key. An attacker who compromises only the master key cannot unwrap any
LevelAdminWrapped DEK without also knowing the admin credential.

Secret metadata (creation time, update time, access count, version) is
encrypted separately from the ciphertext:

metaKey ← HKDF-SHA256(bucketDEK, nil, info="keeper-metadata-v1") → 32 bytes
encryptedMeta ← XChaCha20-Poly1305.Seal(nonce, metaKey, msgpack(metadata))

For LevelAdminWrapped, LevelHSM, and LevelRemote buckets this means
metadata is inaccessible without the bucket credential, preventing an attacker
with read access to the database file from learning access patterns or
timestamps.

All structural metadata is also encrypted at rest. Two keys are derived from
the master key at UnlockDatabase time:

policyEncKey ← HKDF-SHA256(masterKey, nil, info="keeper-policy-enc-v1") → 32 bytes
auditEncKey  ← HKDF-SHA256(masterKey, nil, info="keeper-audit-enc-v1")  → 32 bytes

policyEncKey encrypts: BucketSecurityPolicy values and the rotation WAL.

auditEncKey encrypts: the Scheme, Namespace, and Details fields of
every audit event.

Both keys are cleared from memory at Lock(). The cipher used for metadata
encryption is the same configurable crypt.Cipher interface used for secrets —
the user’s cipher choice (AES-256-GCM for FIPS, XChaCha20-Poly1305 by default)
flows through automatically.

Wire format for all encrypted metadata blobs:

nonce (cipher.NonceSize() bytes) || AEAD-ciphertext

On-disk policy keys are opaque hashes rather than plaintext scheme:namespace
strings, preventing offline enumeration of bucket names:

base ← hex(SHA-256("scheme:namespace"))[:32]   // 32 hex chars = 128-bit key space
_policies/          → encrypted BucketSecurityPolicy
_policies/__hash__  → SHA-256(encrypted policy bytes)
_policies/__hmac__  → HMAC-SHA256(policyKey, encrypted policy bytes)

The in-memory schemeRegistry continues to use "scheme:namespace" as its
key — only the on-disk representation changes.

Each policy record carries two integrity tags written atomically in one bbolt
transaction:

hash ← SHA-256(encryptedPolicyBytes)                         — unauthenticated, pre-unlock integrity
policyKey ← HKDF-SHA256(masterKey, nil, info="keeper-policy-hmac-v1") → 32 bytes
hmac ← HMAC-SHA256(policyKey, encryptedPolicyBytes)          — authenticated, post-unlock integrity

Before UnlockDatabase, only the SHA-256 hash is available. After unlock,
loadPolicy verifies the HMAC tag. UnlockDatabase calls upgradePolicyHMACs
to backfill HMAC tags on policies created before this feature existed.

auditKey ← HKDF-SHA256(masterKey, nil, info="keeper-audit-hmac-v1") → 32 bytes
HMAC ← HMAC-SHA256(auditKey, event fields including Seq)

The signing key is activated at UnlockDatabase and cleared at Lock. When
the master key is rotated, Rotate appends a key-rotation checkpoint event to
every active audit chain, signed with the old audit key as the final event of
the old epoch. History is never rewritten; the checkpoint is the trust bridge
between epochs.

passphrase
    │
    └─ Argon2id(salt) ──→ masterKey (32 bytes, memguard Enclave)
                              │
                              ├─ HKDF("keeper-audit-hmac-v1")  ──→ auditKey    (HMAC signing)
                              ├─ HKDF("keeper-audit-enc-v1")   ──→ auditEncKey (audit field encryption)
                              ├─ HKDF("keeper-policy-hmac-v1") ──→ policyKey   (policy HMAC)
                              ├─ HKDF("keeper-policy-enc-v1")  ──→ policyEncKey (policy/WAL encryption)
                              │
                              ├─ [LevelPasswordOnly]
                              │       └─ HKDF("keeper-bucket-dek-v1:scheme:ns") ──→ DEK
                              │               └─ HKDF("keeper-metadata-v1") ──→ metaKey
                              │
                              ├─ [LevelAdminWrapped]
                              │       ├─ random 32 bytes ──→ DEK
                              │       │       └─ HKDF("keeper-metadata-v1") ──→ metaKey
                              │       │
                              │       └─ HKDF("keeper-kek-v1", masterKey‖adminCred, dekSalt)
                              │                 └─ KEK
                              │                       └─ XChaCha20-Poly1305(KEK, DEK) ──→ wrappedDEK
                              │
                              └─ [LevelHSM / LevelRemote]
                                      ├─ random 32 bytes ──→ DEK
                                      │       └─ HKDF("keeper-metadata-v1") ──→ metaKey
                                      │
                                      └─ HSMProvider.WrapDEK(DEK) ──→ wrappedDEK
                                         (stored; provider controls the wrapping key)

All intermediate keys are zeroed immediately after use. The master key is
never written to disk in any form.

The underlying database is bbolt. All buckets and their contents:

type Secret struct ⚡

The Event struct uses separate plaintext routing fields (Scheme,
Namespace) alongside encrypted payload fields (EncScheme, EncNamespace,
EncDetails). Checksums are computed over the plaintext routing fields and the
encrypted EncDetails bytes, so chain integrity can be verified at three tiers
without any key:

The KDF salt is stored as a msgpack-encoded SaltStore under the salt
metadata key. Each salt rotation appends a new SaltEntry and advances
CurrentVersion. Old entries are retained as an audit trail. The SaltStore is
stored unencrypted — see Security decisions.

Rotate writes a WAL before touching any record. The WAL carries
WrappedOldKey: the pre-rotation master key encrypted with the new master key.
After a crash the old passphrase is gone; WrappedOldKey is the only correct
way to carry the old key across the boundary. At UnlockDatabase, when a WAL
is present, the new master key decrypts WrappedOldKey and rotation resumes
from the WAL cursor. The WAL itself is encrypted with policyEncKey.

Every significant operation appends a tamper-evident event to the bucket’s
audit chain. Chain integrity depends on two mechanisms.

Checksum. SHA-256 over prevChecksum, ID, BucketID, Scheme, Namespace,
EncDetails, EventType, and Timestamp. Using Scheme/Namespace as plaintext
(always preserved alongside the encrypted forms) ensures the checksum is stable
across load paths. EncDetails provides integrity over the encrypted payload.

HMAC. HMAC-SHA256 over all fields including Seq. An attacker who can write
to the database but does not know the audit key cannot produce a valid HMAC.
VerifyIntegrity checks both layers for every event.

Key rotation epoch boundary. At Rotate, a checkpoint event is appended
to every active chain carrying fingerprints of both the outgoing and incoming
audit keys. The checkpoint is signed with the outgoing key. Auditors holding
any epoch key can recover subsequent epoch keys from the wrapped_new_key field
and verify HMAC continuity across the full chain.

Automatic pruning. When AuditPruneInterval is set in Config, a
jack.Scheduler runs periodically and calls PruneEvents on every registered
bucket. LevelHSM and LevelRemote buckets are never pruned regardless of
this setting.

Jack is an optional process supervision library. When a JackConfig is
provided via WithJack, keeper activates background components automatically:
auto-lock Looper, per-bucket DEK Reaper, health monitoring patients (bbolt
read latency + encrypt/decrypt round-trip), audit prune scheduler, and async
event Pool. Keeper never calls pool.Shutdown — the pool lifecycle belongs to
the caller.

x/keepcmd provides reusable keeper operations decoupled from any CLI
framework. Embed it in your own application to get typed, testable secret
management without pulling in the CLI binary.

import "github.com/agberohq/keeper/x/keepcmd"

cmds := &keepcmd.Commands{
    Store: func() (*keeper.Keeper, error) {
        return security.KeeperOpen(cfg)  // your own config
    },
    Out:     keepcmd.PlainOutput{},
    NoClose: false, // true in REPL / session contexts
}

cmds.List()                                          // all keys: scheme://namespace/key
cmds.List("vault")                                   // all keys in scheme vault
cmds.List("vault", "system")                         // all keys in vault://system
cmds.Get("vault://system/jwt_secret")
cmds.Set("vault://system/jwt_secret", "newsecret", keepcmd.SetOptions{})
cmds.Rotate(newPassphraseBytes)    // caller resolved the passphrase — no prompter dependency
cmds.RotateSalt(currentPassBytes)  // same

keepcmd never calls prompter or reads from stdin. Passphrase resolution
is entirely the caller’s responsibility — this keeps the package safe in
headless server contexts.

NoClose: true prevents Commands from calling store.Close() after each
operation. Use this in REPL / session contexts where one store is shared
across many calls.

x/keephandler mounts keeper HTTP endpoints on any net/http mux. No
external router dependency — it uses Go 1.22+ method+pattern routing with
stdlib http.ServeMux.

import "github.com/agberohq/keeper/x/keephandler"

keephandler.Mount(mux, store,
    keephandler.WithPrefix("/api/keeper"),
    keephandler.WithGuard(func(w http.ResponseWriter, r *http.Request, route string) bool {
        if !acl.Allow(r.Header.Get("X-Principal"), route) {
            http.Error(w, `{"error":"forbidden"}`, http.StatusForbidden)
            return false
        }
        return true
    }),
    keephandler.WithHooks(
        keephandler.Hook{
            Route:       keephandler.RouteGet,
            CaptureBody: false,
            After: func(r *http.Request, status int, _ []byte) {
                audit.Log(r.Context(), route, status)
            },
        },
    ),
    keephandler.WithEncoder(func(w http.ResponseWriter, route string, status int, data any) {
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(status)
        json.NewEncoder(w).Encode(map[string]any{
            "ok":    status < 400,
            "route": route,
            "data":  data,
        })
    }),
    keephandler.WithRoutes(func(m *http.ServeMux) {
        m.HandleFunc("POST /api/keeper/totp/{user}", myTOTPHandler)
    }),
)

BeforeFunc returns (allow bool, err error).

• (true, nil) — let the request proceed.
• (false, nil) — abort; the hook has already written a complete response.
• (false, err) — abort; the framework writes a 500 using err.Error().
  The hook must not have written anything to w.

Hook.CaptureBody bool controls whether AfterFunc receives the response
body. false (default) costs one lightweight statusWriter wrapper;
true buffers the full body into a bytes.Buffer for the AfterFunc — one
allocation per request.

store, err := keeper.New(keeper.Config{
    DBPath:              "/var/lib/agbero/keeper.db",
    AutoLockInterval:    30 * time.Minute,
    EnableAudit:         true,
    AuditPruneInterval:  24 * time.Hour,
    AuditPruneKeepLastN: 10_000,
    AuditPruneOlderThan: 90 * 24 * time.Hour,
    DBLatencyThreshold:  200 * time.Millisecond,
    Logger:              logger,
}, keeper.WithJack(keeper.JackConfig{
    Pool:     jackPool,
    Shutdown: jackShutdown,
}))
defer store.Close()

// Shorthand (wraps DeriveMaster + UnlockDatabase):
if err := store.Unlock([]byte(os.Getenv("KEEPER_PASSPHRASE"))); err != nil {
    log.Fatal(err) // ErrInvalidPassphrase on wrong passphrase
}

UnlockDatabase performs the following in order:

1. Derives and activates the audit HMAC signing key
2. Derives and activates the policy HMAC key
3. Derives and activates policyEncKey and auditEncKey
4. Clears and reloads schemeRegistry (decrypts all policy blobs)
5. Resumes any interrupted rotation WAL
6. Upgrades policy HMAC tags
7. Seeds all LevelPasswordOnly bucket DEKs into the Envelope
8. Starts background tasks (migration looper, auto-lock, health patients)

err := store.CreateBucket("vault", "system", keeper.LevelPasswordOnly, "init")

store.Set("vault://system/jwt_secret", []byte("supersecret"))
val, err := store.Get("vault://system/jwt_secret")

// Namespaced convenience wrappers
store.SetNamespaced("admin", "jwt_secret", secretBytes)
val, err = store.GetNamespaced("admin", "jwt_secret")

err := store.CreateBucket("finance", "payroll", keeper.LevelAdminWrapped, "ops-team")
err = store.AddAdminToPolicy("finance", "payroll", "alice", []byte("alicepass"))

store.SetNamespacedFull("finance", "payroll", "salary_key", []byte("AES256..."))

store.LockBucket("finance", "payroll")
err = store.UnlockBucket("finance", "payroll", "bob", []byte("bobpass"))
// ErrAuthFailed — does not distinguish wrong password from unknown admin (CWE-204)

err = store.RevokeAdmin("finance", "payroll", "alice")
err = store.RotateAdminWrappedDEK("finance", "payroll", "bob", []byte("bobpass"))

needs, err := store.NeedsAdminRekey("finance", "payroll")

import (
    "github.com/agberohq/keeper/pkg/hsm"
    "github.com/agberohq/keeper/pkg/remote"
)

// SoftHSM — testing only
provider, _ := hsm.NewSoftHSM()
store.RegisterHSMProvider("secure", "keys", provider)
store.CreateBucket("secure", "keys", keeper.LevelHSM, "ops")

// Vault Transit
cfg := remote.VaultTransit("https://vault.corp:8200", vaultToken, "my-key")
cfg.TLSClientCert = "/etc/keeper/client.crt"
cfg.TLSClientKey  = "/etc/keeper/client.key"
provider, _ = remote.New(cfg)
store.RegisterHSMProvider("tenant", "secrets", provider)
store.CreateBucket("tenant", "secrets", keeper.LevelRemote, "ops")

// Export the audit encryption key to allow a third-party auditor to decrypt
// event details without access to the master passphrase.
auditKey, err := store.ExportAuditKey()
defer zero.Bytes(auditKey)

events, err := auditStore.LoadChain("vault", "system", auditKey)

// Rotate passphrase — crash-safe WAL, resumes on next Unlock if interrupted
store.Rotate([]byte("new-passphrase"))

// Rotate KDF salt — re-derives master key, re-encrypts LevelPasswordOnly
store.RotateSalt([]byte("current-passphrase"))

err := store.CompareAndSwapNamespacedFull("vault", "system", "counter",
    []byte("old"), []byte("new"))
// ErrCASConflict if current value does not match old

f, _ := os.Create("keeper.db.bak")
info, err := store.Backup(f)
// info.Bytes, info.Timestamp, info.DBPath

ErrAuthFailed unifies all UnlockBucket failures (CWE-204 / CVSS 5.3). Both
an unknown admin ID and a wrong password return ErrAuthFailed. This prevents
admin ID enumeration by timing or error-string comparison. RevokeAdmin retains
ErrAdminNotFound because it is an administrative operation on an
already-unlocked store.

Argon2id dominates timing. Argon2id takes 200–500 ms on typical hardware.
Post-derivation comparison differences are four or more orders of magnitude
smaller and are not measurable remotely. No artificial equalisation is applied.

DEK retrieved inside the CAS transaction boundary. CompareAndSwapNamespacedFull
retrieves the bucket DEK inside the bbolt write transaction, eliminating the
window where a concurrent Rotate could change the DEK between retrieval and
use.

Passphrase never stored as a Go string in the HTTP handler. All three
passphrase fields (passphrase, new_passphrase) are decoded from JSON
directly into []byte via raw-map extraction, keeping the string backing array
off the long-lived heap. The []byte copy is zeroed with wipeBytes after use.

No --passphrase flag in the CLI. Flags appear in ps output and shell
history. The CLI accepts the passphrase only from KEEPER_PASSPHRASE env or
an interactive no-echo prompt.

REPL secret values are never visible. set in the REPL without an
inline value uses term.ReadPassword — it does not appear in terminal
scrollback, shell history, or ps. An inline value (set key value) can be
supplied for non-sensitive data when convenient.

SaltStore is intentionally unencrypted. The KDF salt must be readable
before UnlockDatabase to derive the master key. policyEncKey (used for all
other metadata encryption) is itself derived from the master key — encrypting
the salt with policyEncKey would be circular. A KDF salt provides uniqueness,
not confidentiality; there is no security value in encrypting it.

Policy bucket keys are hashed, not plaintext. On-disk policy keys are
hex(SHA-256("scheme:namespace"))[:32] — 128 bits of key space — rather than
readable strings. An offline attacker reading the bbolt file cannot enumerate
bucket names without decrypting the policy blobs.

Metadata encryption uses the same cipher interface as secrets. All
policyEncKey and auditEncKey operations go through s.config.NewCipher(key)
— the same crypt.Cipher interface configured for secret values. The user’s
cipher choice (AES-256-GCM for FIPS 140, XChaCha20-Poly1305 by default) flows
through to policy, WAL, and audit encryption automatically. No code path
hard-codes a specific algorithm.

LevelHSM and LevelRemote buckets skipped during master key rotation.
reencryptAllWithKey and RotateSalt explicitly skip these buckets. The DEK
is provider-controlled; master salt rotation does not affect it.

Crash-safe rotation with WrappedOldKey. Rotate writes a WAL before
touching any record. The WAL carries WrappedOldKey: the pre-rotation master
key encrypted with the new master key. After a crash, UnlockDatabase decrypts
WrappedOldKey using the verified new key and resumes rotation from the cursor.