BREAKING CHANGE — see Breaking Changes and Migration sections below.
Feature
- logger: make logger implementation configurable (#204) — Replace hardcoded log4js with a pluggable
Loggerinterface. Consumers provide a factory viaDownloadOptions.createLogger. Default implementation writes toconsole. Built-in log4js adapter atlib/logger/log4js-adapter.jsfor backward compatibility. - life-cycle: add statusChange listener hook (#102) — New
statusChangearray onProcessingLifeCycleallows consumers to observe resource progression through the pipeline. Default listener logs skipped/discarded resources and errors. - life-cycle: add existingResource callback for local file handling (#150) — Optional
existingResourcecallback onProcessingLifeCycleto decide what to do when a local file already exists (skip, overwrite, if-modified-since, skipSave). - life-cycle: expose submit resource to init hook (#1131) —
InitLifeCycleFuncreceives an optionalsubmitcallback to add URLs to the download queue during initialization.
Fix
- download: enable warnForNonHtml by default, improve warning (#993) —
warnForNonHtmlis now enabled by default. Warning message includesres.typefor clarity.
Breaking Changes
DownloadOptions.configureLoggerreplaced bycreateLogger?: (options: StaticDownloadOptions) => Logger. The default iscreateDefaultLogger(console-based).log4jsmoved fromdependenciestooptionalDependencies. Consumers who need file-based logging mustnpm install log4jsand use the built-in adapter:import {createLog4jsLogger} from 'website-scrap-engine/lib/logger/log4js-adapter.js'; const options = { createLogger: (opts) => createLog4jsLogger(opts.localRoot, opts.logSubDir), };
- Public
loggernamespace exports are typed asCategoryLoggerinstead of log4jsLogger. Method signatures are compatible (.trace(),.debug(),.info(),.warn(),.error(),.isTraceEnabled()), but consumers using log4js-specific properties will need to update. - Worker log message protocol:
WorkerLog.logger(category string) replaced byWorkerLog.logType(LogType string). Affects custom worker implementations only. ProcessingLifeCyclegains a requiredstatusChange: StatusChangeFunc[]field. Consumers building the life cycle from scratch must addstatusChange: [].warnForNonHtmlis now enabled by default (was opt-in).
New Exports
Loggerinterface — the pluggable logger contractLogTypetype — discriminated union of log categories (io.http.request,system.error, etc.)CategoryLoggerinterface — the per-category proxy type (whatlogger.error,logger.skipetc. are)createDefaultLogger()— factory for the console-based default loggerlogger.setLogger(instance)— configure the logger instance at runtimelogger.getLogger()— retrieve the current logger instance
Misc
- docs: rewrite README with usage examples, architecture details, and adapter helpers
- build(deps): bump picomatch, @typescript-eslint/eslint-plugin, @typescript-eslint/parser, handlebars, ts-jest