Support for NodeJS Buffer

[jessmorecroft]

🚀 Feature request

Firstly, great work on this lib and fp-ts in general. Definitely has changed the way I think about writing software big time. Secondly, why I'm here, hoping you can help me use parser-ts more easily/efficiently for binary messages in a Node environment.

Any help or advice greatly appreciated!

Current Behavior

Currently it's not easy to use a Stream<A> with a buffer of the Node Buffer type. I'm able to make it work by casting to an Array<number> on construction, which works fine since the lib code is only doing lookups, length checks and slices I believe, all of which are supported by Buffer and/or its parent interface Uint8Array.

Desired Behavior

It'd be nice to support the Buffer type also. It'd also be nice to have getMany/ getManyAndNext functions for extracting a slice of a buffer. For example in the wire protocol I'm looking at (PostgreSQL) some messages have 32 bit big endian length prefixes which indicate the length of the buffer following that pertain to that message.

Suggested Solution

I've been able to get some stuff working with a bit of casting here and there. Here's a test program to demonstrate:

import { pipe } from 'fp-ts/lib/function';
import * as O from 'fp-ts/lib/Option';
import * as E from 'fp-ts/lib/Either';
import * as P from 'parser-ts/Parser';
import * as PR from 'parser-ts/ParseResult';
import * as S from 'parser-ts/Stream';

type Byte = number;

const stream: (buf: Buffer, cursor?: number) => S.Stream<Byte> = (
  buf,
  cursor
) => S.stream(buf as unknown as Array<number>, cursor); // Buffer behaves enough like an Array for use by parser-ts - that is, supports: slice, at/[], length

function buffer(i: Buffer): P.Parser<Byte, Buffer>;
function buffer(i: string, enc?: BufferEncoding): P.Parser<Byte, Buffer>;
function buffer(
  i: string | Buffer,
  enc?: BufferEncoding
): P.Parser<Byte, Buffer> {
  let buf: Buffer;
  if (typeof i === 'string') {
    buf = Buffer.from(i, enc);
  } else {
    buf = i;
  }
  return P.expected(
    P.ChainRec.chainRec<Byte, Buffer, Buffer>(buf, (acc) =>
      pipe(
        O.fromNullable(acc.at(0)),
        O.fold(
          () => P.of(E.right(buf)),
          (c) =>
            pipe(
              P.sat((b: Byte) => b === c),
              P.chain(() => P.of(E.left(acc.slice(1))))
            )
        )
      )
    ),
    JSON.stringify(buf)
  );
}

const getManyAndNext: <A>(
  i: S.Stream<A>,
  count: number
) => O.Option<{
  value: A[];
  next: S.Stream<A>;
}> = (i, count) => {
  const endIndex = count + i.cursor;
  if (endIndex <= i.buffer.length) {
    return O.some({
      value: i.buffer.slice(i.cursor, endIndex), // our buffer using Buffer actually outputs a Buffer here, not an Array<number>
      next: S.stream(i.buffer, endIndex)
    });
  }
  return O.none;
};

const items: <A>(count: number) => P.Parser<A, Array<A>> =
  (count: number) => (i) =>
    pipe(
      getManyAndNext(i, count),
      O.fold(
        () => PR.error(i),
        ({ value, next }) => PR.success(value, next, i)
      )
    );

const uint32BE = pipe(
  items<number>(4),
  P.map((buf) => Buffer.from(buf).readUInt32BE()) // not ideal - creating a buf here is not necessary
  // P.map((buf) => (buf as unknown as Buffer).readUInt32BE()) // this is more efficient/ works too but relies on casting
);

const bufParser = pipe(
  buffer('x'),
  P.chain(() => uint32BE),
  P.chain((n) => items(n))
);

const buf = Buffer.allocUnsafe(11);
buf.write('x');
buf.writeUint32BE(5, 1);
buf.writeUint8(1, 5);
buf.writeUint8(2, 6);
buf.writeUint8(3, 7);
buf.writeUint8(4, 8);
buf.writeUint8(5, 9);
buf.writeUint8(6, 10);

console.log(bufParser(stream(buf)));

// Output:
// {
//   _tag: 'Right',
//   right: {
//     value: <Buffer 01 02 03 04 05>,
//     next: { buffer: <Buffer 78 00 00 00 05 01 02 03 04 05 06>, cursor: 10 },
//     start: { buffer: <Buffer 78 00 00 00 05 01 02 03 04 05 06>, cursor: 0 }
//   }
// }

Who does this impact? Who is this for?

This is for being able to create a parser that deals easily and efficiently with binary content.

Software	Version(s)
fp-ts	2.11.9
parser-ts	0.6.16
TypeScript	4.5.2

[jessmorecroft]

Hi again. Just an FYI, this one not too big a deal for me any more as just coded around much as discussed above.

Here's a lib I've written for querying Postgres, that also does logical replication, which heavily uses parser-ts to do the protocol parsing legwork.

https://www.npmjs.com/package/@jmorecroft67/pg-stream-core