[Feature Request] Shared web indexedDB database without using Worker or SharedWorker

[AlexDochioiu]

Is your feature request related to a problem? Please describe.
I am working on a chrome extension app (using manifest V3). This app has a background service. I want to access the same database (shared) between the foreground app of the extension and the background service.

The problem
The current implementation of shared web database uses the Worker and SharedWorker classes. Those do not exist in the context of chrome extension background service.

Can this even work
The indexedDB is shared between the extension tab and the background service. So it should be possible for both to read/write on the same database.

[simolus3]

The indexedDB is shared between the extension tab and the background service. So it should be possible for both to read/write on the same database.

The problem is that IndexedDB reads are asynchronous, whereas sqlite3, being a C library, expects a synchronous file system. With OPFS that's no problem, but with IndexedDB we need to cheat. Essentially, the implementation works by loading the entire database into memory before opening it, and then asynchronously updating IndexedDB when the in-memory copy changes.
The downside of this is that two copies of the database aren't synchronized - they can't be without an asynchronous file system, which sqlite3 doesn't support. So I don't actually think this is possible.

Is the background service a singleton? If so, you could essentially use that service as a shared worker, with the foreground app connecting to it to access the database.

[AlexDochioiu]

The problem is that IndexedDB reads are asynchronous, whereas sqlite3, being a C library, expects a synchronous file system. With OPFS that's no problem, but with IndexedDB we need to cheat. Essentially, the implementation works by loading the entire database into memory before opening it, and then asynchronously updating IndexedDB when the in-memory copy changes.
The downside of this is that two copies of the database aren't synchronized - they can't be without an asynchronous file system, which sqlite3 doesn't support. So I don't actually think this is possible.

I see

Is the background service a singleton? If so, you could essentially use that service as a shared worker, with the foreground app connecting to it to access the database.

Yes, it is a singleton. I tried to do this but adding the following code in the background service

  driftWorkerMain(() {
    return WebDatabase.withStorage(
      DriftWebStorage.indexedDb(
        'my_app_db',
        migrateFromLocalStorage: false,
        inWebWorker: true,
      ),
    );
  });

throws StateError('This worker is neither a shared nor a dedicated worker')

[AlexDochioiu]

Update: I managed to get it working with the following web implementation:

import 'dart:html';

import 'package:drift/drift.dart';
import 'package:drift/web.dart';
import 'package:drift/web/worker.dart';

QueryExecutor connect({required bool isWebBackgroundService}) {
  if (isWebBackgroundService) {
    WorkerGlobalScope.instance.importScripts('sql-wasm.js');

    final QueryExecutor queryExecutor = WebDatabase.withStorage(
      DriftWebStorage.indexedDb(
        'my_app_db',
        migrateFromLocalStorage: false,
        inWebWorker: true,
      ),
    );

    driftWorkerMain(() => queryExecutor);

    return queryExecutor;
  } else {
    return DatabaseConnection.delayed(
      connectToDriftWorker(
        "background_service.web.js",
        mode: DriftWorkerMode.shared,
      ),
    );
  }
}

AND with the following change for drift to force accept my background service as a shared worker:

void driftWorkerMain(QueryExecutor Function() openConnection) {
  final self = WorkerGlobalScope.instance;
  final _RunningDriftWorker worker = _RunningDriftWorker(true, openConnection);

  // if (self is SharedWorkerGlobalScope) {
  //   worker = _RunningDriftWorker(true, openConnection);
  // } else if (self is DedicatedWorkerGlobalScope) {
  //   worker = _RunningDriftWorker(false, openConnection);
  // } else {
  //   throw StateError('This worker is neither a shared nor a dedicated worker');
  // }

  worker.start();
}

[AlexDochioiu]

Another Update:

While the above solution runs with no visible errors, the 2 databases seem to be out of sync.

When I update the database via the extension window/page, the QueryExecutor of the background service does not pick up the change.

When I fully restart the background service (and it loads again the DB from indexedDB), it picks up the change.

[simolus3]

Is the background worker opening an independent database connection or are you connecting it to the drift server already? If there are two separate WebDatabase instances, they won't share the underlying connection.

There's no API for this at the moment, but since you're making modifications you can use worker._startedServer ??= worker._establishModeAndLaunchServer(DriftWorkerMode.shared) to obtain a DriftServer. Then you'll want to do something like this to open a connection to the server in the same context:

DriftServer server = ...;
final channel = StreamChannelController();
server.serve(channel.foreign, serialize: false);

final connection =
    await connectToRemoteAndInitialize(channel.local, serialize: false);

connection implements QueryExecutor and can be passed to the constructor of your database class. This will make the background service and the foreground contexts share a database connection (including things like stream queries synchronizing between them). You can set serialize to false because you're opening an in-memory channel to the database server, that makes it a lot cheaper than going over MessagePorts.

[AlexDochioiu]

I was trying to not rely on the drift server at all for the DB connection of the background service. So, what I have is:

1. Background service creates a query executor and wraps it with a database connection. Then it starts a drift server based on this DatabaseConnection. It then returns the same database connection to be used by the DriftDatabase in the service.
2. The foreground window uses connectToDriftWorker to connect.

I think this approach should normally work (for the most part) to share the same database between the two.

---

Update since my last comment:

I think the issue is actually a bigger one, making this more like a completely new feature as opposed to something to be slightly retrofitted.

The APIs used to communicate between the chrome background service and the extension window is actually not exactly the same as the one used by workers.

To create a MessagePort, I need:

1. The background service to call and listen at chrome.runtime.onConnect.addListener((port) => _newConnection(port))
2. The foreground service to get port by using port = chrome.runtime.connect({name: "drift db"})

This needs quite a bit more dart-js interop done ofc. I already did a rough proof of concept for this. Now, The foreground and background are sharing the same database (after updating the DB in foreground, the background db also picks up the change)

The issues:
a) I see a couple of errors in console as it starts, something about message types received by background not being expected type
b) Using watch on the background service doesn't automatically get changes pushed to stream when foreground app changes data. I expect this is more about a misuse of database connection in service as opposed to an issue caused by drift implementation.

---

I think this can work, but also seems different enough to warrant a new mode altogether. Is this something you'd consider including in the main drift package? Or too niche to be worthwhile adding/supporting over time?

[AlexDochioiu]

#2921

I added this draft pull request with my changes to have a better idea of what I did.

P.s. That is insanely rough proof of concept code to test if it would work. There's no "good coding practices" being followed.

[simolus3]

a) I see a couple of errors in console as it starts, something about message types received by background not being expected type

Did your debugging code to print the type reveal the problem? If you're using the new interop APIs you might have to call dartify and jsify before passing Dart objects to a chrome port.

b) Using watch on the background service doesn't automatically get changes pushed to stream when foreground app changes data

Huh, it should. It might be that the foreground thinks its the only connection user if the mode is not shared and thus doesn't send any update notifications.

Is this something you'd consider including in the main drift package? Or too niche to be worthwhile adding/supporting over time?

It's very cool work, but I think it's too niche to be included in package:drift. Given that you're always using a shared connection setup though, it hopefully shouldn't be too hard to copy existing parts from drift sources and release this as an independent package.
I think drift should expose the underlying APIs to share drift databases across any kind of communication channel (it does with package:drift/remote.dart). Isolates and drift's support for web workers are thin wrappers around the remote API. External use cases can use that API too.