print: a simpler JSON schema for export

[simonmichael]

Related to #1099, #1195, and others:

There's common perception that the JSON output of print (and maybe other reports) is complex and harder-than-necessary for humans to work with. It is a bit complex, because it tries to represent the internal data types faithfully and losslessly, as far as possible.

Chunhui Ouyang has been working on a simpler, easier-to-consume JSON schema, in chat. Here is the latest draft as of today:

[
  {
    "code": "",
    "comment": "",
    "date": "",
    "date2": "",
    "payee": "",
    "description": "",
    "status": "",
    "tags": {
      "key1": "value1",
      "key2": "value2"
    },
    "postings": [
      {
        "account": "USD",
        "amount": [
          {
            "commodity": "",
            "cost": {
              "commodity": "",
              "display_value": "",
              "value": ""
            },
            "display_value": "1000.53",
            "value": "1000.52860"
          }
        ],
        "comment": "",
        "tags": {
          "key1": "value1",
          "key2": "value2"
        }
      }
    ]
  }
]

The main difference from the current hledger print -O json schema is a simpler (and more lossy, because of using json's native numbers) representation of amounts.

Since print is often used for export, not just reporting, I feel the lossiness might be a problem (disrupting balanced transactions, balance assertions, etc.) But it's hard to say how much of a problem, without some testing.

Someone could add this alternate json schema to print fairly easily. (We need to keep the existing non-lossy schema also.) Call it print -O json2 or something for now. I think what it needs next is an app using it, with real world data, so that we could get some experience with it and judge its effectiveness.

Note, simple/easier to work with is the goal here. A related but different goal could be "use (or be more like) a standard schema used by others". We searched briefly and didn't find an obvious one of those. Microsoft Common Data Model and XBRL (eXtensible Business Reporting Language) were some possibilities.

[simonmichael]

Related:

I would omit date2, personally. And add status.

I wouldn't try to keep the JSON / CSV / SQL schemas perfectly in sync. Eg because CSV and SQL are tabular, JSON isn't

https://github.com/simonmichael/hledger/blob/main/hledger-lib/Hledger/Data/Json.hs is where hledger serialises to JSON. Most types are serialised automatically (requiring just a single "instance JSON" line of code). But here's one where the fields are customised: https://github.com/simonmichael/hledger/blob/main/hledger-lib/Hledger/Data/Json.hs#L178

[simonmichael]

For comparison, here is print's current json schema (this time with some real data):

$ hledger print
2026-01-01 opening balances
    assets:usd      10 USD
    assets:eur      20 EUR
    assets:equity

[
  {
    "tcode": "",
    "tcomment": "",
    "tdate": "2026-01-01",
    "tdate2": null,
    "tdescription": "opening balances",
    "tindex": 1,
    "tpostings": [
      {
        "paccount": "assets:usd",
        "pamount": [
          {
            "acommodity": "USD",
            "acost": null,
            "acostbasis": null,
            "aquantity": {
              "decimalMantissa": 10,
              "decimalPlaces": 0,
              "floatingPoint": 10
            },
            "astyle": {
              "ascommodityside": "R",
              "ascommodityspaced": true,
              "asdecimalmark": ".",
              "asdigitgroups": null,
              "asprecision": 0,
              "asrounding": "NoRounding"
            }
          }
        ],
        "pbalanceassertion": null,
        "pcomment": "",
        "pdate": null,
        "pdate2": null,
        "poriginal": null,
        "preal": "RealPosting",
        "pstatus": "Unmarked",
        "ptags": [],
        "ptransaction_": "1"
      },
      {
        "paccount": "assets:eur",
        "pamount": [
          {
            "acommodity": "EUR",
            "acost": null,
            "acostbasis": null,
            "aquantity": {
              "decimalMantissa": 20,
              "decimalPlaces": 0,
              "floatingPoint": 20
            },
            "astyle": {
              "ascommodityside": "R",
              "ascommodityspaced": true,
              "asdecimalmark": ".",
              "asdigitgroups": null,
              "asprecision": 0,
              "asrounding": "NoRounding"
            }
          }
        ],
        "pbalanceassertion": null,
        "pcomment": "",
        "pdate": null,
        "pdate2": null,
        "poriginal": null,
        "preal": "RealPosting",
        "pstatus": "Unmarked",
        "ptags": [],
        "ptransaction_": "1"
      },
      {
        "paccount": "assets:equity",
        "pamount": [
          {
            "acommodity": "EUR",
            "acost": null,
            "acostbasis": null,
            "aquantity": {
              "decimalMantissa": -20,
              "decimalPlaces": 0,
              "floatingPoint": -20
            },
            "astyle": {
              "ascommodityside": "R",
              "ascommodityspaced": true,
              "asdecimalmark": ".",
              "asdigitgroups": null,
              "asprecision": 0,
              "asrounding": "NoRounding"
            }
          },
          {
            "acommodity": "USD",
            "acost": null,
            "acostbasis": null,
            "aquantity": {
              "decimalMantissa": -10,
              "decimalPlaces": 0,
              "floatingPoint": -10
            },
            "astyle": {
              "ascommodityside": "R",
              "ascommodityspaced": true,
              "asdecimalmark": ".",
              "asdigitgroups": null,
              "asprecision": 0,
              "asrounding": "NoRounding"
            }
          }
        ],
        "pbalanceassertion": null,
        "pcomment": "",
        "pdate": null,
        "pdate2": null,
        "poriginal": null,
        "preal": "RealPosting",
        "pstatus": "Unmarked",
        "ptags": [],
        "ptransaction_": "1"
      }
    ],
    "tprecedingcomment": "",
    "tsourcepos": [
      {
        "sourceColumn": 1,
        "sourceLine": 1,
        "sourceName": "/Users/simon/src/hledger/a.j"
      },
      {
        "sourceColumn": 1,
        "sourceLine": 5,
        "sourceName": "/Users/simon/src/hledger/a.j"
      }
    ],
    "tstatus": "Unmarked",
    "ttags": []
  }
]

[jack9603301]

It is worth noting that this new mode uses string representation to represent numbers, so it can express decimals of any precision.

[jack9603301]

The above is the v2 draft. Although I believe that string notation may be simple enough and sufficient for JSON representation, for improvement purposes, there is a v3 draft here. It is an upgrade of v2 and uses scientific notation to faithfully represent the internal representation structure of large number algorithms. The following is the v3 draft:

[
  {
    "code": "",
    "comment": "",
    "date": "",
    "date2": "",
    "payee": "",
    "description": "",
    "status": "",
    "tags": {
      "key1": "value1",
      "key2": "value2"
    },
    "postings": [
      {
        "account": "USD",
        "amount": [
          {
            "commodity": "",
            "cost": {
              "commodity": "",
              "display_value": "",
              "value": {
                "base-k": "10",
                "segments": {
                  "0": "100053"
                },
                "exponent": "-2",
                "signbit": "1"
              }
            },
            "display_value": "1000.53",
            "value": {
              "base-k": "10",
              "segments": {
                "0": "100053"
              },
              "exponent": "-2",
              "signbit": "1"
            }
          }
        ],
        "comment": "",
        "tags": {
          "key1": "value1",
          "key2": "value2"
        }
      }
    ]
  }
]

The following is an example of the internal representation of scientific notation in v3.

[
  {
    "code": "",
    "comment": "",
    "date": "",
    "date2": "",
    "payee": "",
    "description": "",
    "status": "",
    "tags": {
      "key1": "value1",
      "key2": "value2"
    },
    "postings": [
      {
        "account": "USD",
        "amount": [
          {
            "commodity": "",
            "cost": {
              "commodity": "",
              "display_value": "",
              "value": {
                "base-k": "10",
                "segments": {
                  "0": "100053"
                },
                "exponent": "-2",
                "signbit": "1"
              }
            },
            "display_value": "-525525256258025525623322255.552562582526258236",
            "value": {
              "base-k": "18",
              "segments": {
                "0": 552562582526258236,
                "1": 258025525623322255,
                "2": 525525256
              },
              "exponent": "18",
              "signbit": "-1"
            }
          }
        ],
        "comment": "",
        "tags": {
          "key1": "value1",
          "key2": "value2"
        }
      }
    ]
  }
]

The reason I'm writing this here is for record-keeping purposes. If I need it someday, I can quickly find it and won't forget it.

This is just a record; it seems that both @simonmichael and I want to adopt the v2 expression because it aligns with the goals of this issue.

Scientific notation follows the formula:

base-k: Indicates the segmentation of the integer part of a large number; for example, a 20-digit integer is forcibly split into two segments. In the formula, it is the value of k.

exponent: Indicates the exponent and the decimal point position.

segments: Data segments, following the formula:

$$ \left( \sum_{i=0}^{n} a_i (10^k)^i \right) \times 10^{\text{exponent}} \times \text{signbit} $$

Verification of bc:

>>> scale=38
>>> (552562582526258236 + 258025525623322255*((10^18)^1) + 525525256*((10^18)^2))*10^-18 * -1
-525525256258025525623322255.55256258252625823600000000000000000000