Custom dir prefix generator for generateSavePath

#731

BREAKING CHANGE — generateSavePath on ProcessingLifeCycle changes from a single optional function to an array-based life cycle stage. CreateResourceArgument gains required fields and loses generateSavePathFn.

Interacts with: #993 (binary misdetected as html — Phase 2 re-generates save paths, should use this life cycle stage instead of calling generateSavePath directly), #150 (existing local file — uses res.savePath to check local files; the generateSavePath stage runs before download, so save paths are finalized before the existingResource check).

Motivation

Users need to map different hosts and paths to different save directories. For example:

Save cdn.example.com assets under assets/ instead of cdn.example.com/
Route example.com/blog/ to blog/ and example.com/docs/ to docs/
Deduplicate resources from multiple CDN hosts into a single directory

The current generateSavePathFn on CreateResourceArgument is a single function override — it replaces the entire save-path logic. There is no composable way to prepend/append path transformations, and the function receives only (uri, isHtml, keepSearch, localSrcRoot) with no access to the resource type, parent, or options.

Additionally, createResource is a single function on ProcessingLifeCycle (not an array), making it impossible to compose multiple transformations without wrapping/replacing the whole function.

Design

New life cycle stage: `generateSavePath`

Split save-path generation out of createResource into its own pipeline stage as an array of functions, inserted between detectResourceType and createResource.

export interface GenerateSavePathContext {
  /** The resolved absolute URI */
  uri: URI;
  /** Resource type (determines isHtml logic) */
  type: ResourceType;
  /** Depth from root */
  depth: number;
  /** The raw url string before resolution */
  rawUrl: string;
  /** Parent resource URL */
  refUrl: string;
  /** True if URL resolution failed (protocol/host errors) */
  replacePathHasError: boolean;
  /** Parent save path (if known) */
  refSavePath?: string;
  /** Parent resource type */
  refType?: ResourceType;
  /** Options */
  options: StaticDownloadOptions;
}

export interface GenerateSavePathResult {
  /** The save path relative to localRoot */
  savePath: string;
  /** Optional: override the refSavePath used for replacePath calculation */
  refSavePath?: string;
}

export interface GenerateSavePathFunc {
  /**
   * Generate or transform the save path for a resource.
   *
   * The first function in the array receives a default savePath computed
   * by the built-in generateSavePath logic. Each subsequent function
   * receives the savePath returned by the previous function.
   *
   * Return void to discard the resource.
   */
  (savePath: string, ctx: GenerateSavePathContext):
    AsyncResult<string | GenerateSavePathResult | void>;
}

Updated `ProcessingLifeCycle`

export interface ProcessingLifeCycle {
  init: InitLifeCycleFunc[];
  linkRedirect: LinkRedirectFunc[];
  detectResourceType: DetectResourceTypeFunc[];
  generateSavePath: GenerateSavePathFunc[];  // CHANGED: was `GenerateSavePathFn | void`
  createResource: typeof createResource;     // simplified, no longer owns save-path logic
  processBeforeDownload: ProcessResourceBeforeDownloadFunc[];
  download: DownloadResourceFunc[];
  processAfterDownload: ProcessResourceAfterDownloadFunc[];
  saveToDisk: SaveToDiskFunc[];
  dispose: DisposeLifeCycle[];
  statusChange: StatusChangeFunc[];
  existingResource?: ExistingResourceFunc;
}

Pipeline stages (updated order)

linkRedirect — skip or redirect URLs
detectResourceType — determine resource type
generateSavePath — compute save path (NEW)
createResource — build Resource object using the save path from step 3
processBeforeDownload — filter/modify; link replacement happens after
download — fetch resource
processAfterDownload — parse content, discover children
saveToDisk — write to filesystem

URL resolution: folded into `PipelineExecutorImpl.createResource`

The URL resolution logic currently in the createResource function in resource.ts (lines 517-548) — handling protocol-relative URLs, absolute paths, relative URLs, file:// URLs — is extracted from that function and folded into PipelineExecutorImpl.createResource(). It is not moved into the generateSavePath hooks themselves; the hooks receive a pre-resolved URI.

The pipeline executor's createResource method resolves the URI via a private _resolveUri helper, runs the generateSavePath hooks, then delegates to this.lifeCycle.createResource() with pre-computed values. This means all callers of pipeline.createResource() — including processHtml, processCss, processSiteMap, and createAndProcessResource — automatically get generateSavePath hook support without changing their call sites.

The _resolveUri private helper:

/**
 * Resolve a raw URL against a refUrl into an absolute URI.
 * Extracted from the createResource function in resource.ts.
 * Called internally from PipelineExecutorImpl.createResource().
 */
private _resolveUri(
  rawUrl: string, refUrl: string, type: ResourceType
): { uri: URI; url: string; replacePathHasError: boolean } {
  let url = rawUrl;
  const refUri = URI(refUrl);
  let replacePathHasError = false;

  if (url.startsWith('file:///') || refUrl.startsWith('file:///')) {
    url = resolveFileUrl(url, refUrl,
      this.options.localSrcRoot, this.options.skipReplacePathError);
    if (!url) {
      replacePathHasError = true;
      url = rawUrl;
    }
  }
  if (!replacePathHasError && url.startsWith('//')) {
    url = refUri.protocol() + ':' + url;
  } else if (!replacePathHasError && url[0] === '/') {
    url = refUri.protocol() + '://' + refUri.host() + url;
  }
  let uri = URI(url);
  if (!replacePathHasError && uri.is('relative')) {
    uri = uri.absoluteTo(refUri);
    url = uri.toString();
  }
  if (!replacePathHasError &&
    checkAbsoluteUri(uri, refUri, this.options.skipReplacePathError,
      url, refUrl, type)) {
    replacePathHasError = true;
  }

  return { uri, url, replacePathHasError };
}

This is a refactor, not a behavior change — the same resolution logic runs, just relocated from the createResource function in resource.ts into the pipeline executor so the URI is available for save-path generation before the Resource object is built.

Initial resources (depth 0)

Initial resources have no parent. Currently createResource computes refSavePath via generateSavePath(refUri, refType === ResourceType.Html, ...) when refSavePath is not provided. In the new design:

For child resources, refSavePath is the parent's savePath (already known).
For initial resources added via addInitialResource in the downloader, refSavePath is computed by the built-in generateSavePath function as today — the generateSavePath life cycle hooks are not run for the refSavePath calculation. This keeps the hooks focused on the current resource's path.

Execution semantics

The private generateSavePath method on PipelineExecutorImpl is called internally from createResource. It follows the same short-circuit-on-void pattern as other stages:

async generateSavePath(
  uri: URI, type: ResourceType, depth: number,
  rawUrl: string, refUrl: string,
  replacePathHasError: boolean,
  refSavePath?: string, refType?: ResourceType
): Promise<{ savePath: string; refSavePath?: string } | void> {
  const isHtml = type === ResourceType.Html;
  const keepSearch = !this.options.deduplicateStripSearch;
  const localSrcRoot = this.options.localSrcRoot;

  // When URL resolution failed, use the raw URL as the save path
  // instead of computing from the (potentially malformed) URI.
  let savePath = replacePathHasError
    ? rawUrl
    : builtinGenerateSavePath(uri, isHtml, keepSearch, localSrcRoot);
  let resultRefSavePath = refSavePath;

  if (!this.lifeCycle.generateSavePath.length) {
    return { savePath, refSavePath: resultRefSavePath };
  }

  const ctx: GenerateSavePathContext = {
    uri, type, depth, rawUrl, refUrl,
    replacePathHasError,
    refSavePath, refType, options: this.options
  };

  for (const fn of this.lifeCycle.generateSavePath) {
    const result = await fn(savePath, ctx);
    if (result === undefined) return undefined;
    if (typeof result === 'string') {
      savePath = result;
    } else {
      savePath = result.savePath;
      if (result.refSavePath !== undefined) {
        resultRefSavePath = result.refSavePath;
      }
    }
  }

  return { savePath, refSavePath: resultRefSavePath };
}

Changes to `CreateResourceArgument`

The createResource function receives the pre-computed savePath and resolved URL:

export interface CreateResourceArgument {
  type: ResourceType;
  depth: number;
  /** The resolved absolute URL (from _resolveUri) */
  url: string;
  /** The pre-resolution URL (post-linkRedirect, pre-resolve) for replaceUri error path */
  rawUrl: string;
  refUrl: string;
  localRoot: string;
  encoding?: ResourceEncoding;
  keepSearch?: boolean;
  skipReplacePathError?: boolean;
  localSrcRoot?: string;
  // NEW: pre-computed by the generateSavePath life cycle
  savePath: string;
  refSavePath: string;
  // NEW: propagated from _resolveUri so createResource can set
  // shouldBeDiscardedFromDownload and compute replaceUri correctly
  replacePathHasError: boolean;
}

Removed fields:

generateSavePathFn — replaced by the life cycle array
refType — only needed for save-path generation, now handled in the life cycle

Changes to `this.lifeCycle.createResource` function body

With savePath, refSavePath, and replacePathHasError pre-computed and passed in via CreateResourceArgument, the createResource function in resource.ts simplifies:

URL resolution removed — the file:///, protocol-relative, absolute-path, relative URL, and checkAbsoluteUri blocks (current lines 517-548) are deleted. The url parameter is already resolved by _resolveUri.
Save-path generation removed — the implGenerateSavePath call and refSavePath fallback computation (current lines 558-566) are deleted. Both values arrive as parameters.
replaceUri computation preserved — uses the pre-computed savePath and refSavePath, with the existing replacePathHasError branch:
```
const replaceUri = replacePathHasError ? URI(rawUrl) :
  URI(urlOfSavePath(savePath)).relativeTo(urlOfSavePath(refSavePath));
```
rawUrl is the pre-resolution URL (post-linkRedirect, pre-resolve) — now a dedicated field on CreateResourceArgument. In the current code, rawUrl was captured at the top of createResource before URL resolution mutated url; in the new design, since resolution is extracted to _resolveUri, rawUrl is passed in explicitly.

shouldBeDiscardedFromDownload preserved — set when replacePathHasError is true:

if (replacePathHasError) {
  resource.shouldBeDiscardedFromDownload = true;
}

downloadLink computation preserved — computed from the (pre-resolved) URI, unchanged.

Changes to `PipelineExecutorImpl.createResource`

The executor's createResource method now orchestrates URL resolution, save-path generation, and resource creation. Its public signature is unchanged — the same arguments as today — but it becomes async and can return void when a generateSavePath hook discards the resource:

async createResource(
  type: ResourceType,
  depth: number,
  url: string,
  refUrl: string,
  localRoot?: string,
  encoding?: ResourceEncoding,
  refSavePath?: string,
  refType?: ResourceType
): Promise<Resource | void> {
  // Step 1: resolve URI (extracted from resource.ts createResource)
  const resolved = this._resolveUri(url, refUrl, type);

  // Step 2: generate save path (runs hooks)
  const savePathResult = await this.generateSavePath(
    resolved.uri, type, depth,
    url, refUrl, resolved.replacePathHasError,
    refSavePath, refType);
  if (!savePathResult) return undefined;

  // Step 3: delegate to the life cycle's createResource
  const arg: CreateResourceArgument = {
    type,
    depth,
    url: resolved.url,
    rawUrl: url,
    refUrl,
    localRoot: localRoot ?? this.options.localRoot,
    localSrcRoot: this.options.localSrcRoot,
    encoding: encoding ?? this.options.encoding[type] ?? 'utf8',
    keepSearch: !this.options.deduplicateStripSearch,
    skipReplacePathError: this.options.skipReplacePathError,
    savePath: savePathResult.savePath,
    refSavePath: savePathResult.refSavePath ?? '',
    replacePathHasError: resolved.replacePathHasError,
  };
  return this.lifeCycle.createResource(arg);
}

This design means every caller of pipeline.createResource() gets generateSavePath hooks for free — processHtml, processCss, processSiteMap, processSourceMap, and createAndProcessResource all go through this method.

Changes to `PipelineExecutor` interface

The createResource return type widens to include void (a generateSavePath hook can discard the resource):

createResource(
  type: ResourceType,
  depth: number,
  url: string,
  refUrl: string,
  localRoot?: string,
  encoding?: ResourceEncoding,
  refSavePath?: string,
  refType?: ResourceType
): AsyncResult<Resource | void>;  // CHANGED: was AsyncResult<Resource>

The signature (parameters) is unchanged. Existing callers that already handle void from processBeforeDownload need only an additional null check after createResource (see Changes to processHtml).

Changes to `PipelineExecutorImpl.createAndProcessResource`

With _resolveUri and generateSavePath now inside createResource, createAndProcessResource simplifies back to essentially the current code — just adding await on createResource (now async):

async createAndProcessResource(
  rawUrl: string,
  defaultType: ResourceType,
  depth: number | void | null,
  element: Cheerio | null,
  parent: Resource
): Promise<Resource | void> {
  const url: string | void = await this.linkRedirect(rawUrl, element, parent);
  if (!url) return;
  const type = await this.detectResourceType(url, defaultType, element, parent);
  if (!type) return;
  const refUrl = parent.redirectedUrl || parent.url;
  const refSavePath = refUrl === parent.url ? parent.savePath : undefined;
  const r = await this.createResource(type, depth || parent.depth + 1, url,
    refUrl, parent.localRoot, this.options.encoding[type],
    refSavePath, parent.type);
  if (!r) return;
  return await this.processBeforeDownload(r, element, parent, this.options);
}

Changes to `processHtml`

processHtml (src/life-cycle/process-html.ts:88-98) currently calls pipeline.createResource() without a null check because the method previously always returned a Resource. With the new void return possibility, a null check is needed:

// Before
let resource: Resource | void = await pipeline.createResource(
  linkType, depth, link, refUrl,
  res.localRoot, options.encoding[linkType],
  savePath, res.type);
resource = await pipeline.processBeforeDownload(resource, elem, res, options);
if (!resource) {

// After
let resource: Resource | void = await pipeline.createResource(
  linkType, depth, link, refUrl,
  res.localRoot, options.encoding[linkType],
  savePath, res.type);
if (!resource) {
  continue;
}
resource = await pipeline.processBeforeDownload(resource, elem, res, options);
if (!resource) {

This is the only caller-side change needed. All other callers (processCss, processSiteMap, processSourceMap) use createAndProcessResource which already handles void.

Default life cycle

export const defaultLifeCycle = (): ProcessingLifeCycle => ({
  init: [],
  linkRedirect: [skipLinks],
  detectResourceType: [detectResourceType],
  generateSavePath: [],  // empty = use built-in default, no transformations
  createResource,
  processBeforeDownload: [],
  download: [downloadResource, downloadStreamingResource, readOrCopyLocalResource],
  processAfterDownload: [
    processRedirectedUrl, processHtml, processHtmlMetaRefresh,
    processSvg, processCss, processSiteMap
  ],
  saveToDisk: [saveHtmlToDisk, saveResourceToDisk],
  dispose: [],
  statusChange: [defaultStatusListener]
});

When the array is empty, PipelineExecutorImpl.generateSavePath() computes the default save path via the built-in function and returns it without further transformation.

Usage examples

Map CDN host to local directory

const lifeCycle = defaultLifeCycle();
lifeCycle.generateSavePath.push((savePath, ctx) => {
  if (ctx.uri.hostname() === 'cdn.example.com') {
    // Strip the host prefix and put under assets/
    const pathPart = savePath.slice('cdn.example.com/'.length);
    return 'assets/' + pathPart;
  }
  return savePath;
});

Flatten multiple CDN hosts

lifeCycle.generateSavePath.push((savePath, ctx) => {
  const cdnHosts = ['cdn1.example.com', 'cdn2.example.com', 'static.example.com'];
  const host = ctx.uri.hostname();
  if (cdnHosts.includes(host)) {
    return savePath.replace(host, 'static');
  }
  return savePath;
});

Route by path prefix

lifeCycle.generateSavePath.push((savePath, ctx) => {
  const p = ctx.uri.path();
  if (p.startsWith('/blog/')) {
    return 'blog' + savePath.slice(savePath.indexOf('/blog/') + '/blog'.length);
  }
  return savePath;
});

Limitations

replacePath depends on both savePath and refSavePath — If a hook changes savePath for some resources but not others, the relative replacePath between parent and child will still be correct (it is computed from the two save paths). But if a hook changes the directory structure in a way that makes the relative path very long (e.g., moving a resource from a/b/c/ to x/), the replacement link will be a long ../../../x/... path. This is correct but may surprise users.
redirectedSavePath runs through the full pipeline — Post-download redirect handling (processRedirectedUrl in adapters.ts) calls pipeline.createAndProcessResource() for the redirected URL, which runs the generateSavePath hooks. This means custom path mappings are applied consistently to both original and redirected URLs. If a consumer needs different behavior for redirected paths (e.g., skip certain hooks), they can check ctx.depth or ctx.rawUrl patterns inside the hook.
No access to the element or parent Resource — Unlike linkRedirect and detectResourceType, the generateSavePath hook does not receive the source element or parent Resource object. It receives refUrl, refSavePath, and refType instead. This is intentional: save-path generation should depend on the URL, not on the DOM context. Consumers who need element-based logic should use processBeforeDownload to modify savePath after resource creation.

Non-Goals

It is not a goal to make createResource itself an array-based hook. The function assembles a Resource from computed values — there is no meaningful composition of multiple "create resource" functions.
It is not a goal to support async this.lifeCycle.createResource (the function in resource.ts). The function is pure computation (no I/O). The PipelineExecutorImpl.createResource method IS async because it runs generateSavePath hooks before delegating to the life cycle function.
It is not a goal to change how replacePath is computed. It remains derived from savePath and refSavePath inside createResource.
It is not a goal to add element/parent access to the generateSavePath context. See Limitations.

Interaction with other proposals

#993 (binary misdetected as html)

Phase 2 of #993 proposes re-generating the save path when a content-type mismatch is detected after download. Since generateSavePath hooks now run inside PipelineExecutorImpl.createResource(), the simplest approach for #993 is to call pipeline.createResource() again with the corrected type and update savePath/replacePath on the existing Resource. Alternatively, PipelineExecutorImpl could expose the private generateSavePath method for direct re-invocation.

#150 (existing local file)

The existingResource callback uses res.savePath to locate the local file. Since generateSavePath runs before createResource, which runs before download, the save path is finalized before the existingResource check at the download stage. No ordering issue.

#204 (configurable logger)

No interaction. The generateSavePath stage does not produce log output (the built-in function logs only for keepSearch hash replacement via log.debug, which is internal to the function).

Migration

Breaking changes

ProcessingLifeCycle.generateSavePath changes from GenerateSavePathFn | void (single optional function) to GenerateSavePathFunc[] (array with new signature).
CreateResourceArgument.generateSavePathFn is removed.
CreateResourceArgument gains required savePath, refSavePath, rawUrl, and replacePathHasError fields; loses refType and generateSavePathFn.
The exported GenerateSavePathFn type is removed.
CreateResourceArgument.url is now the resolved absolute URL (previously it was the pre-resolution URL, with resolution happening inside createResource).
PipelineExecutor.createResource return type widens from AsyncResult<Resource> to AsyncResult<Resource | void>. Callers that pass the result directly to processBeforeDownload without a null check (e.g., processHtml) must add one.

Migration wrapper

A compatibility wrapper converts old-style GenerateSavePathFn to the new GenerateSavePathFunc:

import type {GenerateSavePathFunc} from './life-cycle/types.js';

/**
 * Wrap a legacy GenerateSavePathFn (old signature) into
 * the new GenerateSavePathFunc life cycle hook.
 *
 * @deprecated Use the new GenerateSavePathFunc signature directly.
 */
export function wrapLegacyGenerateSavePath(
  legacyFn: (uri: URI, isHtml?: boolean, keepSearch?: boolean,
    localSrcRoot?: string) => string
): GenerateSavePathFunc {
  return (_savePath, ctx) => {
    const isHtml = ctx.type === ResourceType.Html;
    const keepSearch = !ctx.options.deduplicateStripSearch;
    return legacyFn(ctx.uri, isHtml, keepSearch, ctx.options.localSrcRoot);
  };
}

Note: The wrapper ignores the incoming _savePath because legacy functions compute the path from scratch. A wrapped legacy function is a full replacement — if placed after another hook in the array, it discards the previous hook's output. Legacy wrappers should typically be the only entry in the generateSavePath array.

Users migrate by replacing:

// Before
const lifeCycle = defaultLifeCycle();
lifeCycle.generateSavePath = myCustomGenerateSavePath;

// After
const lifeCycle = defaultLifeCycle();
lifeCycle.generateSavePath.push(
  wrapLegacyGenerateSavePath(myCustomGenerateSavePath)
);

Consumers using `defaultLifeCycle()`

No change needed if not using generateSavePath. The array is empty by default, behavior is identical.

Consumers building `ProcessingLifeCycle` from scratch

Add generateSavePath: [] to the life cycle object. The pipeline executor uses the built-in default when the array is empty.

resource: custom dir prefix generator for generateSavePath #731

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions