jira.xml

<?xml version="1.0" encoding="utf-8" ?>
<!--
#    \\ SPIKE: Secure your secrets with SPIFFE.
#  \\\\\ Copyright 2024-present SPIKE contributors.1
# \\\\\\\ SPDX-License-Identifier: Apache-2.0
-->
<stuff>
  <purpose>
    <target>Our goal is to have a minimally delightful product.</target>
    <target>Strive not to add features just for the sake of adding
      features.
    </target>
    <target>Half-assed features shall be completed before adding more
      features.
    </target>
    <meta>
      Minimally Delightful Product Requirements:
      - A Kubernetes SPIKE deployment
      ✅ Minimal policy enforcement
      - Minimal integration tests
      ✅ A demo workload that uses SPIKE to test things out as a consumer.
      ✅ A golang SDK
    </meta>
  </purpose>
  <low-hanging-fruits>
    <issue>
      if crashed and trying to restore (instead of recover)
      you get this:

      spike (main)$ spike operator recover
      2025/02/24 21:30:35 recover: Problem parsing response body
      unexpected end of JSON input

      error could be more explanatory
    </issue>
    <issue>
      spike (main)$ spike operator restore
      (your input will be hidden as you paste/type it)
      Enter recovery shard: 2025/02/24 22:09:34 recover: Problem parsing
      response body
      unexpected end of JSON input

      ^ this happens when the input is invalid; a more explanatory error message
      would be great.
    </issue>
    <issue>
      // TDO: Yes memory is the source of truth; but at least
      // attempt some exponential retries before giving up.
      if err := be.StoreSecret(ctx, path, *secret); err != nil {
      // Log error but continue - memory is the source of truth
      log.Log().Warn(fName,
      "msg", "Failed to cache secret",
      "path", path,
      "err", err.Error(),
      )
      }

      SQLLite can error out if there is a blocked transaction or
      a integrity issue, which a retry can fix it.
    </issue>
    <issue>
      sanitize perms

      err := api.CreatePolicy(name, spiffeIddPattern, pathPattern, perms)
      if err != nil {
      fmt.Printf("Error: %v\n", err)
      return
      }
    </issue>
    <issue>
      return cobra.Command{
      Use: "delete policy-id",
      Short: "Delete a policy",
      Args: cobra.ExactArgs(1),
      Run: func(cmd *cobra.Command, args []string) {
      api := spike.NewWithSource(source)

      // TOD: sanitize policy id.
      // also validate other command line arguments too if it makes sense.
      // better to stop bad data at the client (but still not trust the
      // client fully)

      Go through all cobra commands and validate/sanitize what needs to
      be.
    </issue>
  </low-hanging-fruits>
  <feature title="Security">
    <issue>
      Initialize the root key with 32 bits of zeroes.

      Initializing rootKey with 32 bytes of zeroes (make([]byte, 32)) instead
      of letting it dynamically grow will be a better practice from a security
      perspective for several reasons:
      * Prevents Length Leaks: If we allow the slice to dynamically grow, an
      attacker observing memory patterns might infer key-related operations
      based on the length changes. By pre-allocating 32 bytes, the length
      remains constant.
      * Ensures Expected Size: It enforces that the encryption key will always
      have a fixed size, reducing the risk of unexpected behavior or bugs
      where the key is shorter or longer than expected.
      * Avoids Accidental nil Access: Without initialization, rootKey defaults
      to nil, which can lead to runtime panics if accessed before being
      properly assigned.
      * Predictable Memory Layout: Pre-allocating avoids potential fragmentation
      in memory allocation, making access patterns more predictable.

      Or even better, use an array `rootKey [32]byte` since size is fixed.

      When doing it, we'd need to check everywhere rootKey is used because
      there are nil check which will regress when using an array or a
      non-nil zeroed-out slice.
    </issue>
  </feature>

  <feature title="Containerization">
    <issue>
      Next prio is likely containerization.
      * start by creating dockerhub repos.
      * also containerize things locally.
    </issue>
  </feature>
  <low-hanging-fruits>
    <issue>
      go runtime update to the recent.
      update SPIRE too while you are at it.
    </issue>
    <issue>
      try getent in  checkdomain

      check_domain() {
      # Use getent instead of dig as it respects /etc/hosts
      if getent hosts "$SPIRE_SERVER_DOMAIN" > /dev/null; then
      # Get the full answer for display
      DNS_ANSWER=$(getent hosts "$SPIRE_SERVER_DOMAIN")

      # Print the resolved address(es)
      echo "DNS resolution for $SPIRE_SERVER_DOMAIN:"
      echo "$DNS_ANSWER"
      return 0
      else
      echo "Error: No valid DNS answer for $SPIRE_SERVER_DOMAIN"
      return 1
      fi
      }

    </issue>
    <issue>
      To docs:

      additional comments.

      wrt `mlock`;
      * it requires root privileges or `CAP_IPC_LOCK` in Linux.
      * there is no windows equivalent.

      however without `mlock` if swap is on, zeroing out memory won't be good enough as we won't have any control ove whatever is written on swap.

      --

      @kfox1111 has suggested to add "turning swap off" to the [production guide](https://spike.ist/operations/production/) as a recommendation.

      Turning swap off is recommended for bare-metal k8s control plane and worker nodes anyway; so chances are, it will be turned off if a modern containerized (doker, k8s, etc) environment.

      If turning swap is not an option, encrypting swap could be another option too.
    </issue>
    <issue>
      may want to put a note in docs or something to run with swap off.
      1:17
      not sure if the sanitation bits would work with swap on a system.

      ^
      production guides.

      ^

      also describe how enabling swap might impact the security posture.
    </issue>
    <issue>
      use Mlock optionally

      Also we can either "try and shrug" using `mlock` as in

      ```go
      err := unix.Mlock(data)
      if err != nil {
      // do nothing
      }

      // Zero out memory
      for i := range data {
      data[i] = 0
      }
      ```

      Or we can put the mlock option behind a feature flag.

      If the user has granted `CAP_IPC_LOCK` to SPIKE Nexus, then they can also set something like `SPIKE_NEXUS_USE_MLOCK="true"` as an env var, and try locking memory if the variable is set (will be "false" by default).



      https://github.com/spiffe/spike/issues/68
    </issue>
    <issue>
      sanitize keeper id and shard

      request := net.HandleRequest[
      reqres.ShardContributionRequest, reqres.ShardContributionResponse](
      requestBody, w,
      reqres.ShardContributionResponse{Err: data.ErrBadInput},
      )
      if request == nil {
      return errors.ErrParseFailure
      }

      shard := request.Shard
      id := request.KeeperId

    </issue>
    <issue>
      validate spiffe id and other parameters
      for this and also other keeper endpoints

      func RouteShard(
      w http.ResponseWriter, r *http.Request, audit *log.AuditEntry,
      ) error {
      const fName = "routeContribute"
      log.AuditRequest(fName, r, audit, log.AuditCreate)

      requestBody := net.ReadRequestBody(w, r)
      if requestBody == nil {
      return errors.ErrReadFailure
      }

      here is an example that does that:
      func RoutePutPolicy(
      w http.ResponseWriter, r *http.Request, audit *log.AuditEntry,
      ) error {
      const fName = "routePutPolicy"
      log.AuditRequest(fName, r, audit, log.AuditCreate)

      requestBody := net.ReadRequestBody(w, r)
      if requestBody == nil {
      return errors.ErrParseFailure
      }

      request := net.HandleRequest[
      reqres.PolicyCreateRequest, reqres.PolicyCreateResponse](
      requestBody, w,
      reqres.PolicyCreateResponse{Err: data.ErrBadInput},
      )
      if request == nil {
      return errors.ErrReadFailure
      }

      err := guardPutPolicyRequest(*request, w, r)
      if err != nil {
      return err
      }

    </issue>
    <issue>
      something similar for SPIKE too:
      Dev mode
      The Helm chart may run a OpenBao server in development. This
      installs a
      single OpenBao server with a memory storage backend.

      For dev mode:
      - no keepers
      - no backing store (everything is in memory)
    </issue>

    <issue>
      Ensure the system works w/o keepers in "in memory" mode.
      also document it and also create a video out of it.
    </issue>

    <issue>
      this should be configurable:
      ticker := time.NewTicker(5 * time.Minute)
    </issue>
    <issue>
      nil check wherever Backend() is called.
      var be backend.Backend
    </issue>

    <issue kind="containerization">
      GitHub has now arm64 runners. we can use it for
      cross-compilation/automation.
      https://github.blog/changelog/2025-01-16-linux-arm64-hosted-runners-now-available-for-free-in-public-repositories-public-preview/
      Here's an example:
      https://github.com/kfox1111/cid2pid/blob/main/.github/workflows/release.yaml

      Also, we can (at least temporarily) stop bundling for Mac OS.
      Also, worth checking if ARM linux binaries work on Mac.
    </issue>
  </low-hanging-fruits>
  <later>
    <issue>
      newSecreteUndeleteCommand

      Run: func(cmd *cobra.Command, args []string) {
      // O: we can pass this as a predicate to newSecretUndeleteCommand,
      as a HOF.
      trust.Authenticate(spiffeId)
    </issue>
    <issue>
      add retries to everything under:
      app/nexus/internal/state/persist
      ^ they all talk to db; and sqlite can temporarily lock for
      a variety of reasons.
    </issue>
    <issue priority="important" severity="medium">
      if a keeper crashes it has to wait for the next nexus cycle which is
      suboptimal. Instead, nexus can send a ping that returns an overall
      status
      of keeper (i.e. if it's populated or not)
      this can be more frequent than hydration; and once nexus realizes
      keeper
      is down, it can rehydrate it.

      in addition; nexus can first check the sha hash of the keeper's shard.
      before resending; if the hashes match, it won't restransmit the shard.
    </issue>
    <issue>
      path sanitization:
      ^(?!/)([a-zA-Z0-9._-]+)(/[a-zA-Z0-9._-]+)*$

      Hashi Vault is not as script as the above regex; but I think this
      gives a nice balance.

      ^(?!/) → Ensures the path does not start with / (Vault does not require a leading /).
      ([a-zA-Z0-9._-]+) → The first segment must consist of alphanumeric characters, dots, underscores, or dashes.
      (/[a-zA-Z0-9._-]+)* → Subsequent segments must follow the same pattern and be separated by /.
      No double slashes (//) allowed.
      No spaces, backslashes, or reserved URL characters allowed.
      Example Valid Paths:
      ✅ secret/myapp/config
      ✅ secrets/db-creds/admin-user
      ✅ tenantA/projectX/env1/key

      Example Invalid Paths:
      ❌ /secret/myapp/config (leading /)
      ❌ secret//double-slash (double /)
      ❌ secret\path (backslash used)
      ❌ secret path/with space (contains spaces)
      ❌ secret#invalid?path (reserved URL characters)
    </issue>
    <issue>
      this is for policy creation:

      allowed := state.CheckAccess(
      spiffeid.String(), "*",
      []data.PolicyPermission{data.PermissionSuper},
      )

      instead of a wildcard, maybe have a predefined path
      for access check like "/spike/system/acl"

      also disallow people creating secrets etc under
      /spike/system
    </issue>

    <issue>
      path is still stored plain in DB; use HMAC instead.

      CREATE TABLE "secrets" (
      "hashed_path"	BLOB NOT NULL,
      "version"	INTEGER NOT NULL,
      "nonce"	BLOB NOT NULL,
      "encrypted_data"	BLOB NOT NULL,
      "created_time"	DATETIME NOT NULL,
      "deleted_time"	DATETIME,
      PRIMARY KEY("hashed_path","version")
      );

      same is true for policies; we need to keep them encrypted too.
    </issue>
    <issue>
      Also, create a script in ./hack that does that.
      (i.e. something that forces Nexus to reset its root key
      upon next restart; and see how it impacts the system
      see SQLite db with a db viewer
      or maybe let that script delete the database too.)

      // TODO: if you stop nexus, delete the tombstone file, and
      restart nexus,
      // (and no keeper returns a shard and returns 404)
      // it will reset its root key and update the keepers to store
      the new
      // root key. This is not an attack vector, because an adversary
      who can
      // delete the tombstone file, can also delete the backing store.
      /// Plus no sensitive data is exposed; it's just all data is
      inaccessible
      // now because the root key is lost for good. In either
      // case, for production systems, the backing store needs to be
      backed up
      // and the root key needs to be backed up in a secure place too.
      // ^ add these to the documentation.
    </issue>
    <issue>
      spike operator reset:
      deletes and recreates the ~/.spike folder
      restarts the initialization flow to rekey keepers.

      volkan@spike:~/Desktop/WORKSPACE/spike$ spike secret get /db
      Error reading secret: post: Problem connecting to peer

      ^ I get an error instead of a "secret not found" message.
    </issue>
    <issue>
      verify if the keeper has shard before resending it:
      send hash of the shard first
      if keeper says “I have it”, don’t send the actual shard.
      this will make things extra secure.
    </issue>
    <issue>
      Fleet management:
      - There is a management plane cluster
      - There is a control plane cluster
      - There are workload clusters connected to the control plane
      - All of those are their own trust domains.
      - There is MP-CP connectivity
      - There is CP-WL connectivity
      - MP has a central secrets store
      - WL and CP need secrets
      - Securely dispatch them without "ever" using Kubernetes secrets.
      - Have an alternative that uses ESO and a restricted secrets
      namespace
      that no one other than SPIKE components can see into.
    </issue>
    <issue>
      Wrt DR:

      A 404 response from a keeper meens that it does not have the shard
      if threshold number of keepers do not have the shard too, then there is
      no way that nexus can recover; so does it mean it should stop polling?

      Not quite. Because later, we can have ways to seed keepers, maybe through
      other nexuses, maybe via cloning from a backup keeper, maybe using a
      secure keeper API (i.e. imitating nexus)

      It's better, and simpler to keep the polling always running.
      It makes less assumptions that way.

      make this an ADR.
    </issue>
    <issue>
      configure SPIKE to rekey itself as per NIST guidelines.
      Also maybe `spike operator rekey` to manually initiate that.
      `spike operator rekey` will also change the shamir shares, wheras the
      internal rekey will just change the encryption key, leaving the shamir
      shares intact.
    </issue>
    <issue>
      This is OpenBao's production deployment checlist; check of if any of
      those also applies for SPIKE, of if we need different/additional items
      for SPIKE.

      From the context, OpenBao can be swapped with SPIKE Nexus, I think:

      End-to-End TLS. OpenBao should always be used with TLS in production. If
      intermediate load balancers or reverse proxies are used to front OpenBao,
      they should not terminate TLS. This way traffic is always encrypted in
      transit to OpenBao and minimizes risks introduced by intermediate layers.

      Single Tenancy. OpenBao should be the only main process running on a
      machine. This reduces the risk that another process running on the same
      machine is compromised and can interact with OpenBao. This can be
      accomplished by using OpenBao Helm's affinity configurable.

      Enable Auditing. OpenBao supports several auditing backends. Enabling
      auditing provides a history of all operations performed by OpenBao and
      provides a forensics trail in the case of misuse or compromise. Audit logs
      securely hash any sensitive data, but access should still be restricted to
      prevent any unintended disclosures. OpenBao Helm includes a configurable
      auditStorage option that provisions a persistent volume to store audit
      logs.

      Immutable Upgrades. OpenBao relies on an external storage backend for
      persistence, and this decoupling allows the servers running OpenBao to be
      managed immutably. When upgrading to new versions, new servers with the
      upgraded version of OpenBao are brought online. They are attached to the
      same shared storage backend and unsealed. Then the old servers are
      destroyed. This reduces the need for remote access and upgrade
      orchestration which may introduce security gaps. See the upgrade section
      for instructions on upgrading OpenBao on Kubernetes.

      Upgrade Frequently. OpenBao is actively developed, and updating frequently
      is important to incorporate security fixes and any changes in default
      settings such as key lengths or cipher suites. Subscribe to the OpenBao
      mailing list and GitHub CHANGELOG for updates.

      Restrict Storage Access. OpenBao encrypts all data at rest, regardless of
      which storage backend is used. Although the data is encrypted, an attacker
      with arbitrary control can cause data corruption or loss by modifying or
      deleting keys. Access to the storage backend should be restricted to only
      OpenBao to avoid unauthorized access or operations.
    </issue>
    <issue>
      a mode that enables the admin to load shares to keepers.
      this will be essentially using the keeper REST API and acting as
      SPIKE Pilot with a recover svid.
      The benefit would be; we can have a dedicated recover and restore binaries
      without having to expose the pilot binary.
      Or we can update the keepers, even if we don't have access to SPIKE Nexus
      or SPIKE Nexus is down.
      maybe by using a "seeder" SPIFFE ID.
      but I'm also not sure if it's worth it.
      it would mean more APIs to secure; it would also mean keeping spike
      keepers more intelligent (instead of keeping them dumb)
    </issue>
    <issue>
      A /stats endpoint.

      A dedicated /stats endpoint will be implemented to provide real-time
      metrics about:
      Total number of secrets managed.
      Status of the key-value store.
      Resource utilization metrics (e.g., CPU, memory).
      This endpoint will support integration with monitoring tools for enhanced
      observability.
      These measures will ensure comprehensive monitoring and troubleshooting.
    </issue>
    <issue>
      By design, we regard memory as the source of truth.
      This means that backing store might miss some secrets.
      Find ways to reduce the likelihood of this happening.
      1. Implement exponential retries.
      2. Implement a health check to ensure backing store is up.
      3. Create background jobs to sync the backing store.
    </issue>
    <issue>
      all components shall have
      liveness and readiness endpoints
      (or maybe we can design it once we k8s...ify things.
    </issue>
    <issue>
      in development mode, nexus shall act as a single binary:
      - you can create secrets and policies via `nexus create policy` etc

      that can be done by sharing
      "github.com/spiffe/spike/app/spike/internal/cmd"
      between nexus and pilot

      this can even be an optional flag on nexus
      (i.e. SPIKE_NEXUS_ENABLE_PILOT_CLI)
      running ./nexus will start a server
      but run
      ning nexus with args will register secrets and policies.
    </issue>
    <issue>
      Consider using OSS Security Scorecard:
      https://github.com/vmware-tanzu/secrets-manager/security/code-scanning/tools/Scorecard/status
    </issue>
    <issue>
      SPIKE automatic rotation of encryption key.
      the shards will create a root key and the root key will encrypt the
      encryption key.
      so SPIKE can rotate the encryption key in the background and encrypt
      it with the new root key.
      this way, we won't have to rotate the shards to rotate the
      encryption key.
    </issue>
    <issue>
      SPIKE CSI Driver

      the CSI Secrets Store driver enables users to create
      `SecretProviderClass` objects. These objects define which secret
      provider
      to use and what secrets to retrieve. When pods requesting CSI
      volumes are
      made, the CSI Secrets Store driver sends the request to the OpenBao
      CSI
      provider if the provider is `vault`. The CSI provider then uses the
      specified `SecretProviderClass` and the pod’s service account to
      retrieve
      the secrets from OpenBao and mount them into the pod’s CSI volume.
      Note
      that the secret is retrieved from SPIKE Nexus and populated to the
      CSI
      secrets store volume during the `ContainerCreation` phase.
      Therefore, pods
      are blocked from starting until the secrets are read from SPIKE and
      written to the volume.
    </issue>
    <issue>
      shall we implement rate limiting; or should that be out of scope
      (i.e. to be implemented by the user.
    </issue>
    <issue>
      more fine grained policy management

      1. an explicit deny will override allows
      2. have allowed/disallowed/required parameters
      3. etc.

      # This section grants all access on "secret/*". further restrictions
      can be
      # applied to this broad policy, as shown below.
      path "secret/*" {
      capabilities = ["create", "read", "update", "patch", "delete",
      "list", "scan"]
      }

      # Even though we allowed secret/*, this line explicitly denies
      # secret/super-secret. this takes precedence.
      path "secret/super-secret" {
      capabilities = ["deny"]
      }

      # Policies can also specify allowed, disallowed, and required
      parameters. here
      # the key "secret/restricted" can only contain "foo" (any value) and
      "bar" (one
      # of "zip" or "zap").
      path "secret/restricted" {
      capabilities = ["create"]
      allowed_parameters = {
      "foo" = []
      "bar" = ["zip", "zap"]
      }

      but also, instead of going deep down into the policy rabbit hole,
      maybe
      it's better to rely on well-established policy engines like OPA.

      A rego-based evaluation will give allow/deny decisions, which SPIKE
      Nexus
      can then honor.

      Think about pros/cons of each approach. -- SPIKE can have a
      good-enough
      default policy engine, and for more sophisticated functionality we
      can
      leverage OPA.
    </issue>
    <issue>
      key rotation

      NIST rotation guidance

      Periodic rotation of the encryption keys is recommended, even in the
      absence of compromise. Due to the nature of the AES-256-GCM
      encryption
      used, keys should be rotated before approximately 232
      encryptions have been performed, following the guidelines of NIST
      publication 800-38D.

      SPIKE will automatically rotate the backend encryption key prior to
      reaching
      232 encryption operations by default.

      also support manual key rotation
    </issue>
    <issue>
      Do an internal security analysis / threat model for spike.
    </issue>
    <issue>
      TODO in-memory "dev mode" for SPIKE #spike (i.e. in memory mode will
      not be default)
      nexus --dev or something similar (maybe an env var)
    </issue>
    <issue>
      Use SPIKE in lieu of encryption as a service (similar to transit
      secrets)
    </issue>
    <issue>
      dynamic secrets
    </issue>
    <issue>
      use case:
      one time access to an extremely limited subset of secrets
      (maybe using a one time, or time-bound token)
      but also consider if SPIKE needs tokens at all; I think we can
      piggyback
      most of the authentication to SPIFFE and/or JWT -- having to convert
      various kinds of tokens into internal secrets store tokens is not
      that much needed.
    </issue>
    <issue>
      - TODO Telemetry
      - core system metrics
      - audit log metrics
      - authentication metrics
      - database metrics
      - policy metrics
      - secrets metrics
    </issue>
    <issue>
      "token" secret type
      - will be secure random
      - will have expiration
    </issue>
    <issue>
      double-encryption of nexus-keeper comms (in case mTLS gets
      compromised, or
      SPIRE is configured to use an upstream authority that is
      compromised, this
      will provide end-to-end encryption and an additional layer of
      security
      over
      the existing PKI)
    </issue>

    <issue>
      * Implement strict API access controls:
      * Use mTLS for all API connections
      * Enforce SPIFFE-based authentication
      * Implement rate limiting to prevent brute force attacks
      * Configure request validation:
      * Validate all input parameters
      * Implement request size limits
      * Set appropriate timeout values
      * Audit API usage:
      * Log all API requests
      * Monitor for suspicious patterns
      * Regular review of API access logs
      ----
      * Enable comprehensive auditing:
      * Log all secret access attempts
      * Track configuration changes
      * Monitor system events
      * Implement compliance controls:
      * Regular compliance checks
      * Documentation of security controls
      * Periodic security assessments
      ---
      * Tune for security and performance:
      * Optimize TLS session handling
      * Configure appropriate connection pools
      * Set proper cache sizes
      * Monitor performance metrics:
      * Track response times
      * Monitor error rates
      * Alert on performance degradation
    </issue>

    <issue>
      ability to clone a keeper cluster to another standby keeper cluster
      (for redundancy).
      this way, if the set of keepers become not operational, we can
      hot-switch to the other keeper cluster.
      the assumption here is the redundant keeper cluster either remains
      healthy, or is somehow snapshotted -- since the shards are in
      memory, snapshotting will be hard. -- but stil it's worth thinking
      about.
      an alternative option would be to simplyh increase the number of
      keepers.
    </issue>
    <issue>
      ErrPolicyExists = errors.New("policy already exists")
      ^ this error is never used; check why.
    </issue>
    <issue>
      work on the "named admin" feature (using Keycloak as an OIDC
      provider)
      This is required for "named admin" feature.
    </issue>
    <issue>
      BootstrapOrDie()
      if we are certain that SPIKE nexus cannot bootstrap, maybe it can
      just kill itself.
    </issue>

    <issue>
      Consider using google kms, azure keyvault, and other providers
      (including an external SPIKE deployment) for root key recovery.
      question to consider is whether it's really needed
      second question to consider is what to link kms to (keepers or
      nexus?)
      keepers would be better because we'll back up the shards only then.
      or google kms can be used as an alternative to keepers
      (i.e., store encrypted dek, with the encrypted root key on nexus;
      only kms can decrypt it -- but, to me, it does not provide any
      additional advantage since if you are on the machine, you can talk
      to
      google kms anyway)
    </issue>
    <issue>
      dev mode with "zero" keepers.
    </issue>
    <issue>
      remove symbols when packaging binaries for release.
    </issue>
    <issue severity="important" priority="above-normal">
      consider db backend as untrusted
      i.e. encrypt everything you store there; including policies.
      (that might already be the case actually) -- if so, document it
      in the website.
    </issue>
    <issue>
      exponentially back off here

      log.Log().Info("tick", "msg", "Waiting for keepers to initialize")
      time.Sleep(5 * time.Second)

      or maybe not; I'm not sure if it's worth the effort.
      or maybe this algorithm has changed already; needs to be
      double-checked.
    </issue>

    <issue kind="good-first-issue"
           ref="https://github.com/spiffe/spike/issues/80">
      validations:

      along with the error code, also return some explanatory message

      instead of this for example

      err = validation.ValidateSpiffeIdPattern(spiffeIdPattern)
      if err != nil {
      responseBody := net.MarshalBody(reqres.PolicyCreateResponse{
      Err: data.ErrBadInput,
      }, w)
      net.Respond(http.StatusBadRequest, responseBody, w)
      return err
      }

      do this

      err = validation.ValidateSpiffeIdPattern(spiffeIdPattern)
      if err != nil {
      responseBody := net.MarshalBody(reqres.PolicyCreateResponse{
      Err: data.ErrBadInput,
      Reason: "Invalid spiffe id pattern. Matcher should be a regex that
      can match a spiffe id"
      }, w)
      net.Respond(http.StatusBadRequest, responseBody, w)
      return err
      }
    </issue>
    <issue>
      control these with flags.
      i.e. the starter script can optionally NOT automatically
      start nexus or keepers.

      #echo ""
      #echo "Waiting before SPIKE Keeper 1..."
      #sleep 5
      #run_background "./hack/start-keeper-1.sh"
      #echo ""
      #echo "Waiting before SPIKE Keeper 2..."
      #sleep 5
      #run_background "./hack/start-keeper-2.sh"
      #echo ""
      #echo "Waiting before SPIKE Keeper 3..."
      #sleep 5
      #run_background "./hack/start-keeper-3.sh"

      #echo ""
      #echo "Waiting before SPIKE Nexus..."
      #sleep 5
      #run_background "./hack/start-nexus.sh"
    </issue>
  </later>
  <runner-up>
    <issue>
      read policies from a yaml or a json file and create them.
    </issue>
    <issue>
      test that the timeout results in an error.

      ctx, cancel := context.WithTimeout(
      context.Background(), env.DatabaseOperationTimeout(),
      )
      defer cancel()

      cachedPolicy, err := retry.Do(ctx, func() (*data.Policy, error) {
      return be.LoadPolicy(ctx, id)
      })

    </issue>
    <issue>
      ability to lock nexus programmatically.
      when locked, nexus will deny almost all operations
      locking is done by executing nexus binary with a certain command
      line flag.
      (i.e. there is no API access, you'll need to physically exec the
      ./nexus
      binary -- regular svid verifications are still required)
      only a superadmin can lock or unlock nexus.

      ^ instead of that, you can run a script that removes all SVID
      registrations. That will effectively result in the same thing.
    </issue>
    <issue>
      consider using NATS for cross trust boundary (or nor) secret
      federation
    </issue>
    <issue>
      over the break, I dusted off
      https://github.com/spiffe/helm-charts-hardened/pull/166 and started
      playing with the new k8s built in cel based mutation functionality.
      the k8s cel support is a little rough, but I was able to do a whole
      lot in it, and think I can probably get it to work for everything.
      once 1.33 hits, I think it will be even easier.
      I mention this, as I think spike may want similar functionality?
      csi driver, specify secrets to fetch to volume automatically, keep
      it up to date, and maybe poke the process once refreshed
    </issue>
    <issue>
      wrt: secure erasing shards and the root key >>
      It would be interesting to try and chat with some of the folks under
      the cncf
      (That's a good idea indeed; I'm noting it down.)
    </issue>
    <issue severity="important" urgency="moderate">
      // TDO: check all database operations (secrets, policies, metadata)
      and
      // ensure that they are retried with exponential backooff.
    </issue>

    <issue>
      we need a "reset" command for the restore operation in case
      we pushed an incorrect set of shards.
    </issue>
    <issue>
      better play with OIDC and keycloak sometime.
    </issue>
    <issue>
      attribute-based policy control

      path "secret/restricted" {
      capabilities = ["create"]
      allowed_parameters = {
      "foo" = []
      "bar" = ["zip", "zap"]
      }
      }
    </issue>
    <issue>
      Note: this is non-trivial, but doable.

      Periodic rotation of the encryption keys is recommended, even in the
      absence of compromise. Due to the nature of the AES-256-GCM
      encryption
      used, keys should be rotated before approximately 232 encryptions
      have
      been performed, following the guidelines of NIST publication
      800-38D.

      This can be achieved by having a separate encryption key protected
      by
      the root key and rotating the encryption key, and maybe maintaining
      a
      keyring. This way, we won't have to rotate shards to rotate the
      encryption
      key and won't need to change the shards -- this will also allow the
      encryption key to be rotated behind-the-scenes automatically as per
      NIST guidance.
    </issue>
    <issue severity="important" priority="elevated">
      write an adr about why those asnyc are snyc from now on:

      // TOD we don't have any retry for policies or for recovery info.
      // they are equally important.

      // TO: these xsync operations can cause race conditions
      //
      // 1. process a writes secret
      // 2. process b marks secret as deleted
      // 3. in memory we write then delete
      // 4. but to the backing store it goes as delete then write.
      // 5. memory: secret deleted; backing store: secret exists.
      //
      // to solve it; have a queue of operations (as a go channel)
      // and do not consume the next operation until the current
      // one is complete.
      //
      // have one channel for each resource:
      // - secrets
      // - policies
      // - key recovery info.
      //
      // Or as an alternative; make these xsync operations sync
      // and wait for them to complete before reporting success.
      // this will make the architecture way simpler without needing
      // to rely on channels.
    </issue>
    <issue>
      the backing store is considered untrusted and it stores
      encrypted information
      todo: if it's "really" untrusted then maybe it's better to encrypt
      everything
      (including metadata) -- check how other secrets managers does this.
    </issue>
    <issue kind="good-first-issue">
      create a `spike status` command that shows the current status an
      stats
      of SPIKE Nexus (i.e. whether root key initialized; how many secrets
      are
      in the system etc.)
      It does not have to be too detailed; we can always amend it later.
    </issue>
    <issue>
      Kubernetification
    </issue>
    <issue>
      v.1.0.0 Requirements:
      - Having S3 as a backing store
    </issue>
    <issue>
      Consider a health check / heartbeat between Nexus and Keeper.
      This can be more frequent than the root key sync interval.
    </issue>
    <issue>
      Unit tests and coverage reports.
      Create a solid integration test before.
    </issue>
    <issue>
      Test automation.
    </issue>


    <issue>
      spike dev mode
    </issue>
    <issue>
      document limits and maximums of SPIKE (such as key length, path
      lenght, policy size etc)
    </issue>
    <issue>
      double encryption when passing secrets around
      (can be optional for client-nexus interaction; and can be mandatory
      for
      tools that transcend trust boundaries (as in a relay / message queue
      that
      may be used for secrets federation)
    </issue>
    <issue>
      active/standby HA mode
    </issue>
    <issue>
      pattern-based random secret generation
    </issue>
    <issue>
      admin ui
    </issue>
    <issue>
      guidelines about how to backup and restore
    </issue>
    <issue>
      - AWS KMS support for keepers
      - Azure keyvault support for keepers
      - GCP kms support for keepers
      - HSM support for keepers
      - OCI kms support for keepers
      - keepers storing their shards in a separate SPIKE deployment
      (i.e. SPIKE using another SPIKE to restore root token)
    </issue>
    <issue>
      postgresql backend
    </issue>
    <issue>
      audit targets:
      - file
      - syslog
      - socket
      (if audit targets are enabled then command will not execute unless
      an
      audit trail is started)
    </issue>
    <issue>
      have a mode where nexus does not create tables upon initialization
      (i.e. the admin is expected to create them beforehand)
    </issue>
    <issue>
      maybe ha mode

      HA Mode in OpenBao: In HA mode, OpenBao operates with one active server
      and multiple standby servers. The active server processes all requests,
      while standby servers redirect requests to the active instance. If the
      active server fails, one of the standby servers takes over as the new
      active instance. This mechanism relies on PostgreSQL's ability to manage
      locks and ensure consistency across nodes35.
      Limitations:
      The PostgreSQL backend for OpenBao is community-supported and considered
      in an early preview stage, meaning it may have breaking changes or limited
      testing in production environments2.
      While PostgreSQL supports replication and failover mechanisms for its own
      HA, these features operate independently of OpenBao's HA mode. Proper
      configuration and monitoring of the PostgreSQL cluster are essential to
      ensure database-level resilience
    </issue>
    <issue>
      OIDC authentication for named admins.
    </issue>
    <issue>
      SPIKE Dynamic secret sidecar injector
    </issue>
  </runner-up>
  <future reason="no immediate product value">
    <issue>
      get an OpenSSF badge sometime.
    </issue>
  </future>
  <backlog>

    <issue kind="v1.0-requirement">
      - Run SPIKE in Kubernetes too.
    </issue>
    <issue kind="v1.0-requirement">
      - Postgres support as a backing store.
    </issue>
    <issue kind="v1.0-requirement">
      - Ability to channel audit logs to a log aggregator.
    </issue>
    <issue kind="v1.0-requirement">
      - OIDC integration: Ability to connect to an identity provider.
    </issue>
    <issue kind="v1.0-requirement">
      - ESO (External Secrets Operator) integration
    </issue>
    <issue kind="v1.0-requirement">
      - An ADMIN UI (linked to OIDC probably)
    </issue>
    <issue kind="v1.0-requirement">
      - Ability to use the RESTful API without needing an SDK.
      That could be hard though since we rely on SPIFFE authentication and
      SPIFFE workload API to gather certs: We can use a tool to automate
      that
      part. But it's not that hard either if I know where my certs are:
      `curl --cert /path/to/svid_cert.pem --key /path/to/svid_key.pem
      https://mtls.example.com/resource`
    </issue>
    <issue kind="v1.0-requirement">
      > 80% unit test coverage
    </issue>
    <issue kind="v1.0-requirement">
      Fuzzing for the user-facing API
    </issue>
    <isssue kind="v1.0-requirement">
      100% Integration test (all features will have automated integration
      tests
      in all possible environments)
    </isssue>
    <issue kind="user-request">
      Ability to add custom metadata to secrets.
    </issue>
    <issue>
      We need use cases in the website
      - Policy-based access control for workloads
      - Secret CRUD operations
      - etc
    </issue>
    <issue kind="v1.0-requirement">
      Postgresql support for backing store.
    </issue>
    <issue>
      maybe a default auditor SPIFFEID that can only read stuff (for
      Pilot;
      not for named admins; named admins will use the policy system
      instead)
    </issue>
    <issue>
      Optionally not create tables and other ddl during backing store
      creation
    </issue>
    <issue>
      SPIKE Pilot to ingest a policy YAML file(s) to create policies.
      (similar to kubectl)
    </issue>
    <issue>
      - SPIKE Keep Sanity Tests
      - Ensure that the root key is stored in SPIKE Keep's memory.
      - Ensure that SPIKE Keep can return the root key back to SPIKE
      Nexus.
    </issue>
    <issue>
      Try SPIKE on a Mac.
    </issue>
    <issue>
      Try SPIKE on an x-86 Linux.
    </issue>
    <issue>
      not an adr, but a feature flag to have it start w/o a prepopulated db.
      that can also be useful for brownfield deployments.

      --

      based on the following, maybe move SQLite "create table" ddls to a
      separate file.
      Still a "tool" or a "job" can do that out-of-band.

      update: for SQLite it does not matter as SQLite does not have a
      concept
      of RBAC; creating a db is equivalent to creating a file.
      For other databases, it can be considered, so maybe write an ADR for
      that.

      ADR:

      It's generally considered better security practice to create the
      schema
      out-of-band (separate from the application) for several reasons:

      Principle of Least Privilege:

      The application should only have the permissions it needs for
      runtime
      (INSERT, UPDATE, SELECT, etc.)
      Schema modification rights (CREATE TABLE, ALTER TABLE, etc.) are not
      needed during normal operation
      This limits potential damage if the application is compromised


      Change Management:

      Database schema changes can be managed through proper migration
      tools
      Changes can be reviewed, versioned, and rolled back if needed
      Prevents accidental schema modifications during application restarts


      Environment Consistency:

      Ensures all environments (dev, staging, prod) have identical schemas
      Reduces risk of schema drift between environments
      Makes it easier to track schema changes in version control
    </issue>


    <qa>
      <issue>
        - SPIKE Nexus Sanity Tests
        - Ensure SPIKE Nexus caches the root key in memory.
        - Ensure SPIKE Nexus reads from SPIKE keep if it does not have
        the root
        key.
        - Ensure SPIKE Nexus saves the encrypted root key to the
        database.
        - Ensure SPIKE Nexus caches the user's session key.
        - Ensure SPIKE Nexus removes outdated session keys.
        - Ensure SPIKE Nexus does not re-init (without manual
        intervention)
        after
        being initialized.
        - Ensure SPIKE Nexus adheres to the bootstrapping sequence
        diagram.
        - Ensure SPIKE Nexus backs up the admin token by encrypting it
        with the
        root
        key and storing in the database.
        - Ensure SPIKE Nexus stores the initialization tombstone in the
        database.
      </issue>
      <issue>
        - SPIKE Pilot Sanity Tests
        - Ensure SPIKE Pilot denies any operation if SPIKE Nexus is not
        initialized.
        - Ensure SPIKE Pilot can warn if SPIKE Nexus is unreachable
        - Ensure SPIKE Pilot does not indefinitely hang up if SPIRE is
        not
        there.
        - Ensure SPIKE Pilot can get and set a secret.
        - Ensure SPIKE Pilot can do a force reset.
        - Ensure SPIKE Pilot can recover the root password.
        - Ensure that after `spike init` you have a password-encrypted
        root key
        in the db.
        - Ensure that you can recover the password-encrypted root key.
      </issue>
    </qa>


  </backlog>
  <future>
    <issue kind="idea">
      ideation:

      state is expensive to maintain.
      thats one of the reasons cloud services try and decouple/minimize
      state
      as such, the fewest number of state stores I can get away with
      reasonably the better
      and the state stores that are light weight are much better then the
      state stores that are heavy weight.
      there is no more heavyyweight state store than a network attached
      sql sever.


      It makes sense to argue that "we already have paid the expensive
      cost of a postgresql,
      so we just want to use that rather then add anything else". That,
      can make sense.

      but for those of us not carrying a postgresql already, its better
      not to have to have one added.

      so... it makes sense to make the backing store "plugin-based"

      3 backends that people might want for different reasons:
      * s3 - be stateless for your own admin needs, state managed by
      someone else
      * k8s crds - you are already maintaining an etcd cluster. Might as
      well reuse it
      * postgresql - you maintain a postgresql and want to reuse that

      the first two initially feel different then postgresql code wise...
      they are document stores.
      But posgres is pretty much json-compatible; besides SPIKE does not
      have a complicated ata model.
      So, we can find a common ground and treat all databases that are
      plugged-in as forms of document stores.

      It could keep the code to talk to the db to a real minimum.

      The files should be encrypted by the spike key, so should be fine
      just putting in a k8s crd or something without a second layer of
      encryption
      that can be a big selling point. already have a small k8s cluster?
      just add this component and now you have a secret store too. no
      hassle.
    </issue>
    <issue kind="idea">
      use custom resources as backing store;
      since everything is encrypted and not many people want a fast
      secrets creation throughtput it woudl be useful.
      because then you can do `helm install spiffe/spire` and use it
      without any state tracking.
    </issue>
    <issue kind="idea">
      for k8s instructions (docs)
      Might recommend deploying with the chart rather then from scratch,
      which has a lot of those settings. then we can call out the settings
      in the chart on how to do it
    </issue>
    <issue>
      update the guides: PSP is not a thing anymore; better update it
      to Pod Security Standards.
    </issue>
    <issue>
      An external secrets store (such as Hashi Vault) can use SPIKE Nexus
      to
      auto-unseal itself.
    </issue>

    <issue>
      multiple keeper clusters:

      keepers:
      - nodes: [n1, n2, n3, n4, n5]
      - nodes: [dr1, dr2]

      if it cant assemble back from the first pool, it could try the next
      pool, which could be stood up only during disaster recovery.
    </issue>
    <issue>
      a tool to read from one cluster of keepers to hydrate a different
      cluster of keepers.
    </issue>

    <issue>
      since OPA knows REST, can we expose a policy evaluation endpoint to
      help OPA augment/extend SPIKE policy decisions?
    </issue>
    <issue>
      maybe create an interface for kv, so we can have thread-safe
      variants too.
    </issue>

    <issue>
      maybe create a password manager tool as an example use case
    </issue>

    <issue>
      A `stats` endpoint to show the overall
      system utilization
      (how many secrets; how much memory, etc)
    </issue>
    <issue>
      maybe inspire admin UI from keybase
      https://keybase.io/v0lk4n/devices
      for that, we need an admin ui first :)
      for that we need keycloak to experiment with first.
    </issue>

    <issue>
      the current docs are good and all but they are not good for seo; we
      might
      want to convert to something like zola later down the line
    </issue>

    <issues>
      wrt ADR-0014:
      Maybe we should use something S3-compatible as primary storage
      instead of sqlite.
      But that can wait until we implement other features.

      Besides, Postgres support will be something that some of the
      stakeholders
      want to see too.
    </issues>


    <issue>
      SPIKE Dev Mode:

      * Single binary
      * `keeper` functionality runs in memory
      * `nexus` uses an in-memory store, and its functionality is in the
      single
      binary too.
      * only networking is between the binary and SPIRE Agent.
      * For development only.

      The design should be maintainable with code reuse and should not
      turn into
      maintaining two separate projects.
    </issue>
    <issue>
      rate limiting to api endpoints.
    </issue>
    <issue>
      * super admin can create regular admins and other super admins.
      * super admin can assign backup admins.
      (see drafts.txt for more details)
    </issue>
    <issue>
      Each keeper is backed by a TPM.
    </issue>
    <issue>
      Do some static analysis.
    </issue>
    <to-plan>
      <issue>
        S3 (or compatible) backing store
      </issue>
      <issue>
        File-based backing store
      </issue>
      <issue>
        In memory backing store
      </issue>
      <issue>
        Kubernetes Deployment
      </issue>
    </to-plan>
    <issue>
      Initial super admin can create other admins.
      So that, if an admin leaves, the super admin can delete them.
      or if the password of an admin is compromised, the super admin can
      reset it.
    </issue>
    <issue>
      - Security Measures (SPIKE Nexus)
      - Encrypting the root key with admin password is good
      Consider adding salt to the password encryption
      - Maybe add a key rotation mechanism for the future
    </issue>
    <issue>
      - Error Handling
      - Good use of exponential retries
      - Consider adding specific error types/codes for different failure
      scenarios
      - Might want to add cleanup steps for partial initialization
      failures
    </issue>
    <issue>
      Ability to stream logs and audit trails outside of std out.
    </issue>
    <issue>
      Audit logs should write to a separate location.
    </issue>
    <issue>
      Create a dedicated OIDC resource server (that acts like Pilot but
      exposes
      a
      restful API for things like CI/CD integration.
    </issue>
    <issue>
      HSM integration (i.e. root key is managed/provided by an HSM, and
      the key
      ever leaves the trust boundary of the HSM.
    </issue>
    <issue>
      Ability to rotate the root key (automatic via Nexus).
    </issue>
    <issue>
      Ability to rotate the admin token (manual).
    </issue>
    <issue>
      Encourage to create users instead of relying on the system user.
    </issue>
  </future>
</stuff>