docs: detail underlying network probing binaries and optimization roadmap

Author: jsuzanne

Roadmap/PROBE_OPTIMIZATIONS.md

@@ -0,0 +1,68 @@
+# Connectivity Probe Optimizations Roadmap
+
+This document outlines a high-level assessment of the Digital Experience Monitoring (DEM) underlying probe engines, evaluating the potential for deeper integration directly into native `Node.js` APIs versus the current external OS-level subprocess approach.
+
+---
+
+## 1. TCP Port Reachability
+
+### Current Implementation
+- **Tool**: Shell `nc` (Netcat) via `child_process.exec`.
+- **Workflow**: Spawns an external Bash shell, which then launches the `nc` binary.
+
+### Proposed Optimization
+- **Rewrite to**: Pure Native Node.js `net.Socket`.
+- **Assessment**: **High ROI / Low Effort**. Replacing `nc` with a native `new net.Socket().connect(port, host)` completely eliminates the heavy OS context-switch and bash wrapper overhead. It would lower container CPU spikes instantly while identically validating standard TCP handshake completion. We can track connect times perfectly within the Node Event Loop.
+- **Priority**: High (Whenever next refactoring phase is initiated).
+
+---
+
+## 2. DNS Resolution Speed
+
+### Current Implementation
+- **Tool**: Shell `dig` utility.
+- **Workflow**: Forked `dig` subprocess targeting specific nameservers using `+time` thresholds.
+
+### Proposed Optimization
+- **Rewrite to**: Pure Native Node.js `dns` module (`dns.promises.resolve` combined with `dns.promises.setServers`). 
+- **Assessment**: **High ROI / Medium Effort**. Node internally uses `c-ares`, an extremely robust asynchronous DNS C library. Bypassing `dig` means the backend never leaves Node memory space to perform a query. We'd manually attach `Date.now()` wrapping logic to replicate the pristine latency outputs that `dig` currently supplies.
+- **Priority**: Medium. 
+
+---
+
+## 3. HTTP / HTTPS (Digital Experience)
+
+### Current Implementation
+- **Tool**: Shell `curl` utility.
+- **Workflow**: Uses `curl`'s heavily formatted `-w` flags to independently isolate TLS Handshakes, TCP Handshakes, TTFB, and namelookups.
+
+### Proposed Optimization
+- **Rewrite to**: **N/A (Keep `curl`)**.
+- **Assessment**: **Negative ROI**. While Node.js `fetch` or `https.request` operates in-memory and skips subprocesses, capturing explicit sub-layer timing events (such as tracking when identically the TLS Handshake succeeds vs the physical socket binding) requires exceptionally complex network hooks (`socket.on('secureConnect')`). The external `curl` process is heavier, but provides unparalleled, undisputed raw metric precision automatically. 
+- **Priority**: Do not optimize.
+
+---
+
+## 4. ICMP Ping
+
+### Current Implementation
+- **Tool**: Shell `ping` utility.
+- **Workflow**: Spawns `-c 1` ICMP pings through the host environment.
+
+### Proposed Optimization
+- **Rewrite to**: **N/A (Keep `ping`)**.
+- **Assessment**: **Negative ROI**. Operating raw ICMP sockets inside Node.js programmatically requires massive security escalations (running Node entirely as `root` or mapping explicit `CAP_NET_RAW` Linux capabilities into the Docker image). Leveraging the pre-escalated native `ping` OS binary is the industry-standard secure approach.
+- **Priority**: Do not optimize.
+
+---
+
+## 5. UDP (Real-time QoS)
+
+### Current Implementation
+- **Tool**: Shell `iperf3` utility.
+- **Workflow**: Client UDP execution binding to proprietary `iperf3` ports.
+
+### Proposed Optimization
+- **Rewrite to**: **N/A (Keep `iperf3`)**.
+- **Assessment**: **Negative ROI**. `iperf3` handles complex proprietary packet accounting including server-negotiation to calculate end-to-end Packet Loss and Jitter. A native Node `dgram` UDP packet mapping engine would require an identical Node instance to live on the target server just to acknowledge the receipt of the connectionless packet chunks. `iperf3` is irreplaceable for interacting with third-party testing nodes.
+- **Priority**: Do not optimize.

docs/CONNECTIVITY_ENDPOINTS.md

@@ -9,28 +9,36 @@ The **Connectivity Probes** (formerly Synthetic Endpoints) provide real-time vis
 The platform supports three primary probe types, each measuring different aspects of the digital experience:
 
 ### 1. HTTP (Digital Experience)
-- **Mechanism**: Performs a standard HTTP/HTTPS `GET` request to the target.
+- **Mechanism**: Orchestrates a native OS `curl` subprocess to perform HTTP/HTTPS `GET` queries.
+- **Primary Benefit**: `curl` provides military-grade precision for extracting intricate low-level execution timings out-of-the-box, allowing us to perfectly isolate DNS lookup delays, TCP Handshake times, and TLS Handshake overhead from the raw Time-To-First-Byte (TTFB).
 - **Metrics**: 
-  - **Latency (ms)**: Time to first byte.
+  - **Latency (ms)**: Total Time to first byte breakdown.
   - **Status**: Success (2xx/3xx) or Failure (4xx/5xx/Timeout).
 - **Scoring**: Weighted calculation: `100 - (30% Latency + 35% TTFB + 25% TLS)`. Penalized heavily if Latency > 2s, TTFB > 1s, or TLS Handshake > 800ms.
 
 ### 2. PING (Network Reachability)
-- **Mechanism**: Sends standard ICMP Echo Requests.
+- **Mechanism**: Executes the native OS `ping` binary to dispatch ICMP Echo Requests.
+- **Primary Benefit**: Using the host's native `ping` utility intelligently avoids the strict capability/root privileges required to open raw ICMP sockets programmatically, ensuring secure, unprivileged execution environments (like Docker containers) map reachability flawlessly.
 - **Metrics**: 
   - **RTT (ms)**: Round-trip time.
 - **Scoring**: Good if < 100ms (Score 100). Reaches 0 at 500ms.
 
 ### 3. DNS (Resolution Speed)
-- **Mechanism**: Queries the target domain.
+- **Mechanism**: Queries the target domain leveraging the `dig` system utility.
+- **Primary Benefit**: `dig` bypasses systemic OS-level caching interfaces, providing the exact unadulterated response time of the raw nameserver for highly faithful resolution mapping.
 - **Metrics**:
   - **Resolution Time (ms)**: Real-world mapping speed.
 - **Scoring**: Good if < 80ms (Score 100). Reaches 0 at 400ms.
 
 ### 4. UDP (Voice/Real-time Quality)
-- **Mechanism**: UDP reachability probe.
+- **Mechanism**: Triggers an `iperf3` client process (`-u` mode) aimed at the target port.
+- **Primary Benefit**: `iperf3` is the undisputed industry standard for UDP throughput mapping. It intrinsically calculates complex networking permutations including packet loss percentages and millisecond Jitter natively without requiring manual script math.
 - **Scoring**: `100 - (Loss % * 10) - Jitter penalty`. Jitter over 30ms reduces the score (max -50). 10% packet loss results in a score of **0**.
 
+### 5. TCP (Port Reachability)
+- **Mechanism**: Executes `nc` (Netcat) to simulate a standard TCP socket connection.
+- **Primary Benefit**: Netcat securely tests port exposure and firewall routing viability without risking incomplete handshakes that some specialized application daemons reject.
+
 ## 🏆 Scoring Methodology
 
 All probes return a score from **0 to 100**.