Use in embedded system

[X-Ryl669]

I'm wondering how hard it would be to use this on embedded system (like a ESP32, in esp-idf or a STM32 dev board).
It already has numerous features that are required for embedded system, like no dynamic malloc, simple code base hence also small binary size. I haven't tested the speed but I'm sure it's not too bad, the code is well written and clearly readable.

However, other libraries usually implements theses features:

1. Windowed rendering. On embedded CPU, the RAM memory is very limited, it's not possible to store a complete RGBA buffer for the whole output screen in memory (or very slowly with PSRAM on ESP32). So all libraries implement a trick: only store a fraction of the RGBA buffer (for example 1/16th of the buffer, or 10 lines) so it can fit in SRAM, and render 1/fraction times, each time only clipping in what's displayed (for example, the first render will render any path in from y = 0 to y = height/16 - 1, then the second render will be for height/16 to 2*height/16 and so on)
2. Abstracting the buffer pixel type (I think it's not too hard to template the RGBA type in the code to support 16bits 5.6.5 RGB that many screens supports or 1bit pixels for monochrome screens)
3. Implementing double buffering and callbacks when some rendering is done. Typically, once a buffer is rendered, it's sent via DMA to the screen and the rendering continues on another buffer, doubling the achievable FPS.
4. (optional) Implementing a kind of damage tracking. Since many screens are SPI based, and thus have a very limited bus bandwidth, the screen itself stores its pixels' state, there's no point in sending twice the same pixel. It's possible to only update part of the screen (a region of interest). Thus the library can be modified to reset and compute the union of modified bounding box area while drawing so only the modified area is rendered and resent to the screen. Once the whole screen is rendered, the damage area is reset and can be used for the next frame.

Do you think it'd be possible to add these feature set (if they both match your development goals and pick up your interest)?

[a-e-k]

Hi, thanks for the question and your interest. Sorry for the delayed reply. Having given this a little thought:

1. For windowed/tiled rendering, I think you could do this through careful use of the transformations. I think if you appended a translation to the end of each transform to shift everything to the window/tile that would work. You could likely do it by saving/restoring the canvas state before/after the view transform whenever you need to update the local-to-world transform.

   But I could see that being rather clumsy and having a way to set a view transform that gets automatically applied while being kept separate from the local-to-world transform would be much more convenient and not-to-difficult.

   Likewise, for resetting the state and clearing, I had imagined that you could just destroy the canvas instance and recreate it if you wanted that. Or if you wanted to do it all within one instance so that you're not reallocating much, you could save the canvas state as soon as it's created to have a way of completely resetting the state. Then restore, save, draw to clear the frame buffer, restore, and save to start the next tile.

   But again, that's rather a clumsy dance. A single new call to just reset everything would make that much more convenient, not to mention faster.

   Until now, I've tried to keep the API as small and as close as reasonably possible to the HTML canvas API, given C++ and not JS, and without any real extensions to it. But these do seem like some nice quality-of-life features that wouldn't bloat the library too much.

2. Hmm. The only places that actually read or write from the framebuffer, bitmap, are in render_shadow(), render_main(), get_image_data(), and put_image_data(). I'm loath to template the canvas class here, but perhaps if I routed all access through private get and put pixel functions? Those could then unpack and pack the pixels and could easily be replaced. I'd want to profile to make sure that gets inlined in a way that doesn't impact size or time too much.

   I also worry a bit about the reduced fidelity with respect to gamma compression, since it's designed to operate on linearized RGB internally. But I could also see somebody choosing to drop that feature for embedded and just do the "incorrect" thing and blend in sRGB anyway.

   Alternatively, it's meant to be pretty easy to just swap the existing rgba class implementation out.

3. For simplicity, everything here is immediate, so once a call returns the drawing is done. You might be able to overlap the call to get_image_data() on the previous frame with drawing a new frame from scratch. But for that sort of use, I had just envisioned having two instances of the canvas that you swap between. I'm also loath to introduce anything that might require synchronization.

4. Hmm, that's an interesting one that I'd never considered. But I could see how maybe adding one call to query the damage region and another to reset it could be useful. (Or may just one function and it just automatically resets when you query it?) Again, I'd been trying to stick close to the HTML API, but that's a small extension that wouldn't add too much to the footprint and the lack of it can't easily be worked around right now using the existing API.

   Are you thinking of checking if a pixel is actually modified from what was there before - e.g., if I draw opaque blue on identical blue, or a transparent pixel with zero alpha then it doesn't count as damage? Or more just about pixels under the stroke and fill whether they actually changed or not? The later would be more performant and still pretty tightly bound, given the coverage span architecture of the rasterizer.

[a-e-k]

On second thought, for #1, the scan-conversion for the rasterizer already takes a translation offset that it uses for drop shadows. A viewport offset would be very easy to add there with no real size or performance penalty. Though I kind of like a full view transform for allowing zooms and rotations too.

[X-Ryl669]

For 1, the solution can be outside of this library (provided the library adds some support for clipping, that is, avoid drawing when something is outside the predefined clipping rect), with the main program redoing the drawing operation multiple times as required. This would avoid having to buffer anything (and any reallocation that's a no-no for embedded systems).

For 2, yes I think it's easy. For most embedded use, when the display is 16bits per pixel (5 bits per component), you don't really care about the gamma curve of the color emitter, it's already too bad as is, so the 2.2 gamma for sRGB to linear RGB is completely ignored here and it's fine

For 3, ok, it should be doable, but it'd cause twice the memory. Maybe it could be possible to set up the rendering context (that is the buffers to render onto) externally via another constructor, so they can be reused and shared for DMA (creating DMA compatible memory, IMHO, is out of the context of your library, so there should be a way for me to give you the buffer you want)

For 4, it's more a mathematical bounding box computation for each operation. Computing damage at the pixel level is too costly, IMHO, so simply, if you draw to region (X1,Y1)->(X2,Y2), you compute the union of all regions and that's the damaged area (without any drawing, to speed up computation). Then you set the clipping area to only the expected region. The embedded system will use that "union of regions" to set the drawing window on the display (it's an hardware feature that most slow screen (like SPI) implements) and then you render the pixels to this area only and send those to the HW. This usually saves a lot of bandwidth and increase the FPS by a large margin.

[a-e-k]

For 1, it already performs Sutherland-Hodgman clipping to the screen window before scan conversion. So if the framebuffer is sized to a tile, that's already done.

Let me think more about 3.

For 4, what I'd meant was that as part of the scan conversion, it slices everything into horizontal spans of pixels where the fill rule applies:

···········································
···········································
···········································
·········<----->···························
········<>··<-->···························
·······<->···<->···························
······<-->····<>···························
······<-->·································
·····<--->·································
·····<--->·································
·····<--->·································
···<-------->····<---->········<---->······
···<-------->··<-->·<--->····<-->·<--->····
·····<--->····<-->···<--->··<-->···<--->···
·····<--->····<-->···<--->··<-->···<--->···
·····<--->···<--->····<-->·<--->····<-->···
·····<--->···<--->····<-->·<--->····<-->···
·····<--->···<--->····<-->·<--->····<-->···
·····<--->···<--->···<--->·<--->···<--->···
·····<--->····<--->··<--->··<--->··<--->···
·····<--->·····<-->··<-->····<-->··<-->····
···<------->····<----->·······<----->······
···········································
···········································
···········································

And furthermore, it finds the intersection of these with the clip path spans. So it would be fairly straightforward to union these pixel spans with the bounding box to track the damage region with a fairly tight box. It was just making this still tighter by checking if the pixels within the spans meaningfully changed that I didn't think was worth it. (E.g., ignoring pixels that don't actually change due to the global compositing operation, or where their alpha is zero, or where we're painting over with the exact same color.)