Skip to content

flatbuffers-idl.md

Latest commit

 

History

History
232 lines (178 loc) · 8.44 KB

flatbuffers-idl.md

File metadata and controls

232 lines (178 loc) · 8.44 KB
Raw
title FlatBuffers IDL Support
sidebar_position 7
id flatbuffers_idl
license Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

This page explains how Apache Fory consumes FlatBuffers schemas (.fbs) and translates them into Fory IR for code generation.

What This Page Covers

  • When to use FlatBuffers input with Fory
  • Exact FlatBuffers to Fory mapping behavior
  • Supported Fory-specific attributes in .fbs
  • Adoption notes and generated-code differences

Why Use Apache Fory

  • Idiomatic generated code: Fory generates language-idiomatic classes/structs that can be used directly as domain objects.
  • Java performance: In Java object-serialization workloads, Fory is faster than FlatBuffers in Fory benchmarks.
  • Other languages: serialization performance is generally in a similar range.
  • Deserialization in practice: FlatBuffers can be faster when callers read directly from its buffer, but applications that need native objects still require conversion, and that conversion step can dominate read cost. In those cases, Fory deserialization is often faster end-to-end.
  • Easier APIs: Fory uses direct native objects, so you do not need to reverse-build tables or manually manage offsets.
  • Better graph modeling: Shared and circular references are first-class features in Fory.

Quick Decision Guide

Situation Recommended Path
You already have .fbs schemas and want Fory APIs/codegen Use FlatBuffers input
You are starting new schema work and want full Fory syntax control Use native Fory IDL
You need FlatBuffers wire compatibility Keep FlatBuffers stack
You need Fory object-graph semantics (ref, weak refs, etc.) Use Fory

FlatBuffers to Fory Mapping

Schema-Level Rules

  • namespace maps to Fory package namespace.
  • include entries map to Fory imports.
  • table is translated as evolving=true.
  • struct is translated as evolving=false.
  • root_type is parsed but ignored by Fory codegen.
  • file_identifier and file_extension are parsed but not used by Fory codegen.

Field Numbering

FlatBuffers fields do not have explicit field IDs. Fory assigns field numbers by source declaration order, starting at 1.

Scalar Type Mapping

FlatBuffers Fory Type
byte int8
ubyte uint8
short int16
ushort uint16
int int32
uint uint32
long int64
ulong uint64
float float32
double float64
bool bool
string string

Vectors ([T]) map to Fory lists.

Unions

FlatBuffers unions map to Fory unions.

  • Case IDs are assigned by declaration order, starting at 1.
  • Case names are derived from type names using snake_case field naming.

FlatBuffers

union Payload {
  Note,
  Metric
}

table Container {
  payload: Payload;
}

Fory shape after translation

union Payload {
    Note note = 1;
    Metric metric = 2;
}

message Container {
    Payload payload = 1;
}

Services

FlatBuffers rpc_service definitions are translated to Fory services. With --grpc, the compiler emits gRPC service companions for supported outputs such as Java, Python, Go, Rust, C#, Dart, Scala, Kotlin, and JavaScript. JavaScript browser clients are generated with --grpc-web. These companions use Fory serialization for request and response payloads.

rpc_service SearchService {
  Lookup(SearchRequest):SearchResponse;
  StreamLookup(SearchRequest):SearchResponse (streaming: "server");
}
foryc api.fbs --java_out=./generated/java --python_out=./generated/python --go_out=./generated/go --rust_out=./generated/rust --csharp_out=./generated/csharp --dart_out=./generated/dart --scala_out=./generated/scala --kotlin_out=./generated/kotlin --javascript_out=./generated/javascript --grpc

Generated service code imports grpc APIs, so applications must provide grpc-java, grpc-kotlin, Scala grpc-java APIs, grpcio, grpc-go, Rust tonic and bytes, @grpc/grpc-js, C# Grpc.Core.Api plus server/client dependencies, or Dart package:grpc when they compile or run those files. Python companions use grpc.aio by default and can be generated in sync mode with --grpc-python-mode=sync. Fory packages do not add gRPC as a hard dependency. Use --grpc-web with JavaScript output to generate browser clients that import grpc-web.

Defaults and Metadata

  • FlatBuffers default values are parsed but not applied as Fory defaults.
  • Non-Fory metadata attributes are preserved as generic options in IR and may be consumed by downstream tooling.

Fory-Specific Attributes in FlatBuffers

FlatBuffers metadata attributes use key:value. For Fory-specific options, use fory_ (or fory.) prefix in .fbs; the prefix is removed during parsing.

Supported Field Attributes

FlatBuffers Attribute Effect in Fory
fory_ref:true Enable reference tracking for the field
fory_nullable:true Mark field optional/nullable
fory_weak_ref:true Enable weak reference semantics and implies ref
fory_thread_safe_pointer:false For ref fields, select Rust Rc/RcWeak instead of the default Arc/ArcWeak

Semantics:

  • fory_weak_ref:true implies ref.
  • fory_thread_safe_pointer defaults to true, only takes effect when the field is ref-tracked, and does not change the wire format.
  • In Rust codegen, fory_weak_ref:true uses ArcWeak by default and switches to RcWeak only when fory_thread_safe_pointer:false is set.
  • For list fields, fory_ref:true applies to list elements.

Example:

table Node {
  parent: Node (fory_weak_ref: true);
  children: [Node] (fory_ref: true);
  local: Node (fory_ref: true, fory_thread_safe_pointer: false);
}

Generated Code Differences

Using .fbs as input to Fory still produces normal Fory-generated code, not FlatBuffers ByteBuffer-style APIs.

  • Java, Scala, and Kotlin: JVM model types with Fory metadata and registration helpers
  • Python: dataclasses plus registration helpers
  • C++, Go, and Rust: native structs and Fory metadata
  • JavaScript/TypeScript: TypeScript interfaces and schema helpers
  • C#, Swift, and Dart: annotated or macro-based model types with registration helpers

The serialization format is Fory binary protocol, not FlatBuffers wire format.

Usage

Compile a FlatBuffers schema directly:

foryc schema.fbs --lang java,python --output ./generated

Inspect translated schema syntax for debugging:

foryc schema.fbs --emit-fdl --emit-fdl-path ./translated

Adoption Notes

  1. Keep existing namespace values stable to keep type registration stable.
  2. Review fields that relied on FlatBuffers default literals and set explicit defaults in application code if needed.
  3. Add fory_ref/fory_weak_ref where object-graph semantics are required.
  4. Validate generated model behavior with roundtrip tests before replacing existing serialization paths.

Summary

FlatBuffers input lets you reuse existing .fbs schemas while moving to Fory's serialization and code generation model. This is useful for incremental adoption while preserving schema investment and using Fory-native object APIs.