Ling's Notes

Edge Infra

4 min read

Building a globally distributed edge infrastructure for software package delivery requires a synergy of multi-tiered caching, intelligent edge computation, and robust networking. By combining these technologies, we can ensure that developers worldwide can pull packages, binaries, and containers with minimal latency and high availability.12

Here is the design and implementation strategy for a global edge network tailored for package delivery.

Edge Architecture Overview

Our system utilizes a tiered architecture to move data as close to the end user as possible while protecting the origin from traffic spikes.31

                    +-------------------+
                    |    Developer      |
                    | (npm/docker pull) |
                    +---------+---------+
                              |
                              v
                    +-------------------+
                    | Anycast / Geo-DNS |
                    +---------+---------+
                              | (Routes to nearest PoP)
                              v
+-----------------------------------------------------------+
|                       EDGE PoP                            |
|                                                           |
|  +-------------+    +---------------+   +--------------+  |
|  | Edge Compute|--->| L1 Cache      |   | WAF / DDoS   |  |
|  | (Auth/Route)|    | (Memory/NVMe) |   | Protection   |  |
|  +-------------+    +-------+-------+   +--------------+  |
+-----------------------------|-----------------------------+
                              | (Cache Miss)
                              v
+-----------------------------------------------------------+
|                  REGIONAL SHIELD CACHE                    |
|  +-----------------------------------------------------+  |
|  |   L2 Cache (High Capacity SSD, Request Collapsing)  |  |
|  +--------------------------+--------------------------+  |
+-----------------------------|-----------------------------+
                              | (Cache Miss)
                              v
+-----------------------------------------------------------+
|                   ORIGIN INFRASTRUCTURE                   |
|  +--------------------+       +------------------------+  |
|  | Blob Storage       |       | Global Metadata DB     |  |
|  | (S3 / GCS)         |       | (Spanner / DynamoDB)   |  |
|  +--------------------+       +------------------------+  |
+-----------------------------------------------------------+

The Full Stack

To achieve this scale, the technology stack must be highly concurrent and lightweight:

  • Edge Routing & Proxy: NGINX, Envoy, or Rust-based proxies to handle millions of concurrent TCP connections and perform TLS termination.4
  • Edge Compute: WebAssembly (Wasm) or V8 Isolates running directly on the CDN edge to execute custom logic like authentication, A/B testing, and request filtering without routing back to the origin.31
  • Caching Layer: Varnish or custom memory-mapped file systems for L1 edge caching, backed by high-capacity NVMe drives for L2 regional shields.
  • Data & Origin: Geographically replicated object storage (like AWS S3) for immutable package blobs, and a globally distributed database (like Google Cloud Spanner) for mutable package metadata and user entitlements.

Edge Caching and CDN Optimizations

Software packages often experience “thundering herd” traffic patterns, such as when a popular CI/CD pipeline kicks off thousands of identical container pulls simultaneously.

  • Tiered Cache Hierarchy: Implementing an L1 edge cache and an L2 regional shield reduces origin calls and optimizes global latency.3
  • Request Collapsing: If 10,000 clients request the same uncached package simultaneously, the edge node collapses these into a single origin request, preventing origin overload.
  • Predictive Caching: By analyzing package dependency trees (e.g., package.json), edge servers can pre-cache required dependencies before the client explicitly requests them.1
  • Cache Invalidation: Using stale-while-revalidate headers allows the CDN to serve a slightly outdated metadata file (like a Docker latest tag) while asynchronously fetching the updated version in the background.

Distributed Networking Solutions

Routing users efficiently is critical for minimizing latency and ensuring high availability during regional outages.

  • Anycast IP Routing: Advertising the same IP address from multiple global locations allows the Border Gateway Protocol (BGP) to naturally route the user’s TCP connection to the topologically closest datacenter.
  • Dynamic Server Selection: The system actively monitors Round Trip Time (RTT) and CPU usage to dynamically route traffic away from congested or degraded Points of Presence (PoPs).51
  • Protocol Optimizations: Utilizing TCP BBR congestion control and HTTP/3 (QUIC) reduces connection setup time and mitigates the impact of packet loss on unstable mobile networks.

System Data Flows

When a user pulls a package, the request follows a strict path to ensure authorization and speed:

  1. Resolution: The client’s DNS query hits a Geo-DNS provider, returning the Anycast IP of the nearest Edge PoP.
  2. Edge Auth: The request reaches the Edge Proxy. An Edge Function executes immediately, verifying the user’s API token against a highly cached subset of the metadata database.1
  3. Cache Lookup: The proxy checks the L1 Cache. If the package is found, it is returned instantly.
  4. Shield Fallback: On an L1 miss, the request goes to the Regional Shield. If the package is present in the L2 cache, it is returned and populated in L1.
  5. Origin Fetch: On an L2 miss, the shield fetches the blob from Origin Storage, caches it, and streams it back down the chain to the client.

Performance Impact Chart

This tiered networking approach dramatically reduces latency across the distribution lifecycle.

Average Response Latency (ms) by Retrieval Tier
------------------------------------------------------------
Origin Fetch       |################################ (250ms)
Regional Shield L2 |########### (85ms)
Edge PoP L1        |### (20ms)
Predictive Cache   |# (5ms)
------------------------------------------------------------

Footnotes

  1. https://notionhive.com/blog/edge-computing-cdn-strategies ↩ ↩2 ↩3 ↩4 ↩5 ↩6

  2. https://talents.studysmarter.co.uk/companies/cloudsmith-ltd/belfast/senior-software-engineer-edge-29145650/ ↩

  3. https://www.daydreamsoft.com/blog/edge-caching-and-cdn-optimization-delivering-lightning-fast-web-experiences ↩ ↩2 ↩3

  4. https://builtin.com/job/senior-software-engineer-tech-platform/6578449 ↩

  5. https://arxiv.org/html/2412.09474v1 ↩

Graph View