PrivateId Logo

Section 5: Scaling MongoDB Atlas and Redis for 50M–1B Users

Overview:

To support large-scale identity traffic — ranging from tens of millions to over a billion users — your database and cache layers must be resilient, performant, and scalable. This section details production-grade setups for MongoDB Atlas and Redis, with architecture, security, and scaling recommendations aligned to high-throughput biometric workloads.

MongoDB Atlas for High Scale

Cluster Tier & Configuration

  • Start with M50+ dedicated cluster (scale to M200+ for billions of users)
  • Sharded Cluster: Enable for horizontal scaling
  • Global Clusters: Use region-aware data placement to reduce latency
  • Multi-region support: Add read replicas in latency-critical regions
  • Storage auto-scaling: Enable for automatic volume growth

Security & Connectivity

  • TLS in-transit: Always enabled by default in Atlas
  • VPC Peering or PrivateLink: Secure, low-latency connections from your Kubernetes cluster
  • IP Whitelisting / CIDR Filtering: Add only your cluster’s IPs
  • RBAC: Create custom MongoDB roles for app-only access

Observability & Tuning

  • Monitor slow queries and resource metrics in Atlas dashboard
  • Use MongoDB Profiler for hot path inspection
  • Use indexes on commonly queried fields (e.g. api_key, customer_id)
  • Benchmark and load-test with tools like MongoPerf or k6

Section 5.1: Initializing the MongoDB Database

Overview

The cryptonets-api automatically creates the necessary database and collection structure in MongoDB Atlas when it first connects. However, it does not insert access credentials (API keys) by default.


This section walks you through inserting the initial document into the api_key collection — either manually via the MongoDB Atlas UI or programmatically using the CLI or a script. Without this step, the API will reject requests with a "missing API key" or "unauthorized" error.

Step 5.2: Connect to Your MongoDB Cluster

Before proceeding, make sure:

  • Your MongoDB URI in the Kubernetes Secret is correct and accessible
  • The cluster is either IP-whitelisted for your Kubernetes nodes or peered/VPC-connected
  • You have read/write access to the target database

Step 5.3: Insert an API Key Document

You’ll need to insert a document into the api_key collection. This document enables applications to authenticate against the cryptonets-api.


Option A: Use MongoDB Atlas Web UI (Simple & Manual)

  1. Log into https://cloud.mongodb.com
  2. Navigate to your Cluster > Collections
  3. Open the database created by the API (e.g. cryptonets)
  4. Find or create the collection: api_key
  5. Click "Insert Document" and paste: 
  {
    "api_key": "0000000000000000test",
    "customer_id": "private_id",
    "secret": "",
    "api_id": "",
    "usage_plan_id": "",
    "dimension": "",
    "product_code": "",
    "status": "active"
  }

Fields explained:

  • api_key: Unique API key clients will send to the API
  • customer_id: Useful for tracking or multi-tenancy
  • status: Set to "active" to enable usage
  • Other fields can be left empty unless required by your integration logic

Option B: Insert Programmatically (Automation-Friendly)


If you prefer automation or want to include this step in CI/CD:


Using mongosh:

  db = connect("<your-mongo-uri>");
  db.getSiblingDB("cryptonets").api_key.insertOne({
    api_key: "0000000000000000test",
    customer_id: "private_id",
    secret: "",
    api_id: "",
    usage_plan_id: "",
    dimension: "",
    product_code: "",
    status: "active"
  });

Or via MongoDB Data API (if enabled):


Use a curl POST to the Data API endpoint with an API key to insert the document. This is helpful in air-gapped or CI environments where you don’t want to expose mongosh

Step 5.4: Create a Vector Search Index in MongoDB

To enable similarity search using vector embeddings, you need to create a vector index on your collection. MongoDB supports vector search via Atlas Vector Search.

Option A: Use MongoDB Atlas Web UI (Simple & Visual)

  1. Log into https://cloud.mongodb.com
  2. Navigate to your Cluster > Browse Collections
  3. Select the relevant "cryptonets-prod-sdk" DB and "feature" collection where your vectors (e.g., feature_vector) are stored
  4. Go to the Search Indexes tab
  5. Click Create Search Index
  6. Choose JSON Editor and paste the following definition:
  {
    "mappings": {
      "dynamic": false,
      "fields": {
          "feature_vector": {
          "type": "vector",
          "similarity": "euclidean",
          "dimensions": 512
        }
      }
    }
  }

Option B: Use MongoDB Compass (Developer-Friendly GUI)

If you're using MongoDB Compass v1.45.4 or later, you can now create vector search indexes directly from the UI:

  1. Open MongoDB Compass
  2. Connect to your cluster
  3. Navigate to your "cryptonets-prod-sdk" DB → "feature" collection → Indexes tab
  4. Click the green Create button
  5. Select Search Index from the dropdown
  6. Paste the following JSON in the editor:
  {
    "mappings": {
      "dynamic": false,
      "fields": {
          "feature_vector": {
          "type": "vector",
          "similarity": "euclidean",
          "dimensions": 512
        }
      }
    }
  }
  1. Click Create Search Index

Step 5.5: Restart the API Pod (If Needed)

Some deployments cache API keys in memory on startup. To ensure your new key is recognized:

  kubectl rollout restart deployment/privateid-api

Verify readiness:

  kubectl get pods -l app=privateid-api

With your API key inserted, your cryptonets-api is now fully operational and ready to handle authenticated traffic.

Was this page helpful?