Raw and inline asset loader support

[kamranayub]

We are migrating a Monaco-based playground over and are using tsup currently. That seems to work in the original repository but as part of migrating over to our core repo, I thought we'd take a look at tsdown.

It seems we need some special handling of .glsl files because we import them either inline or raw using .glsl?raw and .glsl?inline.

In tsup we passed --loader .glsl=text and that works.

In tsdown I made a new config and it seems to not support those inline and raw import statements.

tsdown.config.ts

import { defineConfig } from 'tsdown'

export default defineConfig({
  entry: '../src/engine/index.ts',
  dts: true,
  tsconfig: '../src/engine/tsconfig.json',
  outDir: './types',
  loader: {'.glsl': 'text'}
})

output

Excalibur\playground on  docs-migrate-playground [$!] is 📦 v1.0.0 via  v24.11.0 took 5s
❯ npm run build:types

> excalibur-playground@1.0.0 build:types
> npx tsdown

ℹ tsdown v0.17.0-beta.6 powered by rolldown v1.0.0-beta.53
ℹ config file: E:\Development\Contrib\excaliburjs\Excalibur\playground\tsdown.config.ts 
ℹ entry: ..\src\engine\index.ts
ℹ tsconfig: ..\src\engine\tsconfig.json
ℹ Build start

 ERROR  Error: Build failed with 23 errors:

[UNLOADABLE_DEPENDENCY] Error: Could not load ../src/engine/Util/Toaster.css?inline
   ╭─[ ../src/engine/Util/Toaster.ts:1:24 ]
   │
 1 │ import toasterCss from './Toaster.css?inline';
   │                        ───────────┬──────────
   │                                   ╰──────────── The filename, directory name, or volume label syntax is incorrect. (os error 123)
───╯

[UNLOADABLE_DEPENDENCY] Error: Could not load ../src/engine/Director/Loader.css?inline
   ╭─[ ../src/engine/Director/Loader.ts:6:23 ]
   │
 6 │ import loaderCss from './Loader.css?inline';
   │                       ──────────┬──────────
   │                                 ╰──────────── The filename, directory name, or volume label syntax is incorrect. (os error 123)
───╯

[UNLOADABLE_DEPENDENCY] Error: Could not load ../src/engine/Graphics/PostProcessor/color-blind-fragment.glsl?raw
   ╭─[ ../src/engine/Graphics/PostProcessor/ColorBlindnessPostProcessor.ts:1:37 ]
   │
 1 │ import colorBlindCorrectSource from './color-blind-fragment.glsl?raw';
   │                                     ────────────────┬────────────────
   │                                                     ╰────────────────── The filename, directory name, or volume label syntax is incorrect. (os error 123)    
───╯

[UNLOADABLE_DEPENDENCY] Error: Could not load ../src/engine/Graphics/Context/particle-renderer/particle-fragment.glsl?raw
   ╭─[ ../src/engine/Graphics/Context/particle-renderer/particle-renderer.ts:5:36 ]
   │
 5 │ import particleFragmentSource from './particle-fragment.glsl?raw';
   │                                    ───────────────┬──────────────
   │                                                   ╰──────────────── The filename, directory name, or volume label syntax is incorrect. (os error 123)        
───╯

[UNLOADABLE_DEPENDENCY] Error: Could not load ../src/engine/Graphics/Context/particle-renderer/particle-vertex.glsl?raw
   ╭─[ ../src/engine/Graphics/Context/particle-renderer/particle-renderer.ts:4:34 ]
   │
 4 │ import particleVertexSource from './particle-vertex.glsl?raw';
   │                                  ──────────────┬─────────────
   │                                                ╰─────────────── The filename, directory name, or volume label syntax is incorrect. (os error 123)
───╯

...

Should this be supported? Are we doing something weird? The tsup output I get is:

❯ npm run build:types:tsup

> excalibur-playground@1.0.0 build:types:tsup
> npx tsup --loader .glsl=text ../src/engine/index.ts --dts --tsconfig ../src/engine/tsconfig.json --out-dir types

CLI Building entry: ../src/engine/index.ts
CLI Using tsconfig: ..\src\engine\tsconfig.json
CLI tsup v8.5.0
CLI Target: es2018
CJS Build start
CJS types\Loader.logo-DZZNB4UP.png     2.73 KB
CJS types\monogram-bitmap-7RZLAWJP.png 1.83 KB
CJS types\index.css                    2.56 KB
CJS types\index.cjs                    1.11 MB
CJS ⚡️ Build success in 263ms
DTS Build start
DTS ⚡️ Build success in 4508ms
DTS types\index.d.cts 607.23 KB

[kamranayub]

We use Vite for the main build and it looks like that supports (through esbuild?) the inline and raw asset handling. So this would be a problem for us migrating to Rolldown as well I expect.

https://vite.dev/guide/assets#explicit-inline-handling

[anntnzrb]

Query parameters (?raw, ?inline) not supported in tsdown yet. The loader config only handles extension mapping. Rolldown underneath supports it via plugins, but tsdown doesn't expose this. Feature gap - use loader: {'.glsl': 'text'} as workaround without query params.

[kamranayub]

Thanks. I don't think we can do that, since that might be a breaking change with our existing build and distribution 😃 We might just need to keep using tsup until tsdown and rolldown support that.

edit: Further, wouldn't that then cause issues with our stable Vite build? So it would "fix" tsdown but break Vite... 🤔

We do plan to try to migrate to Vite Rolldown Preview for TSGo support. If we do that, we could align both builds and tooling. So if there's not an easy temp workaround, we could wait for that.

[kamranayub]

Related: We could possibly solve this across bundlers / runtimes with import attributes.

rolldown/rolldown#2758

[sxzz]

You can try https://github.com/unplugin/unplugin-raw plugin for ?raw support

[sxzz]

I'm investigating

[kamranayub]

I am trying a custom Vite plugin that inlines CSS by default and it seems to work but I need to verify the outputs.

[kamranayub]

OK, this is hacky but this works and has a byte-for-byte equivalent output in our Vite build when I remove ?inline from CSS imports.

function inlineCssByDefault(): Plugin {
  return {
    name: 'inline-css-by-default',
    enforce: 'pre',

    transform(code, id) {
      // Only touch JS/TS/JSX/TSX
      if (!/\.(mjs|cjs|js|ts|jsx|tsx)$/.test(id)) {
        return null;
      }
      if (!code.includes('.css')) {
        return null;
      }

      // Match both:
      //   import styles from "./foo.css";
      //   import * as styles from "./foo.css";
      //   import { x } from "./foo.css";
      //
      // And *not*:
      //   import "./foo.css";
      //
      const importRegex = /import\s+(?<bindings>[^'"]+?)\s+from\s+['"](?<spec>[^'"]+\.css)['"];?/g;

      let match: RegExpExecArray | null;
      let s: MagicString | undefined;

      while ((match = importRegex.exec(code))) {
        const groups = match.groups!;
        const spec = groups.spec;

        // Don’t touch inline/url/custom-query or CSS Modules
        if (spec.includes('?')) {
          continue;
        }
        if (spec.endsWith('.module.css')) {
          continue;
        }

        const start = match.index + match[0].indexOf(spec);
        const end = start + spec.length;

        if (!s) {
          s = new MagicString(code);
        }
        s.overwrite(start, end, `${spec}?inline`);
      }

      if (!s) {
        return null;
      }

      return {
        code: s.toString(),
        map: s.generateMap({ hires: true })
      };
    }
  };
}

I don't know how to build Vite plugins so I used ChatGPT so... I'm sure this is not a canonical way to do it?

[sxzz]

After reviewing your PR excaliburjs/Excalibur#3598, I noticed you're using tsup to bundle the playground, correct? However, tsup doesn't handle ?inline properly, it simply converts .css?inline to:

// ../src/engine/Director/Loader.css?inline
var Loader_default = {};

which isn't correct.

Currently, both tsdown and tsup have limited support for CSS. If you only need dts types, you can externalize your CSS imports.

https://tsdown.dev/options/dependencies#external

[kamranayub]

Interesting! I think all we need is the .d.mts file for Monaco typings. The inline CSS by default "works" but let me try just externalizing the CSS for the tsdown build 🤔

[kamranayub]

@sxzz Yep, that works, and doesn't require any changes to the core build pipeline (which includes both Vitest and Storybook).