custom / __tostring printer results incorrectly put into quotes

[fab13n]

I'm evaluating the feasibility of using Serpent as Metalua's pretty-printer. I'm stuck with the fact that the result of __tostring metamethods are put into quotes. There's a similar issue with custom printers. For instance:

token_mt = { }
function token_mt :__tostring()
    return string.format("token(%s, %s)", self.tag, self.value)
end

function token(tag, value)
    return setmetatable({tag=tag; value=value}, token_mt)
end

stream = { token("Keyword", "local"), token("Id", "x") }

s = require 'serpent'
print(s.block(stream))
-- incorrectly produces {"token(Keyword, local)", "token(Id, x)"} which can't be deserialized


pp = require'metalua.pprint'
pp.print(stream)
-- produces { token(Keyword, local), token(Id, x) }, readily accepted by loadstring().

[agladysh]

I'd say that __tostring results should be put into quotes — they are, indeed, strings.

Maybe __serialize would help? (Didn't look into it closely.)

[fab13n]

@agladysh Adding this produces the same result:

token_mt.__serialize=token_mt.__tostring
print(s.serialize(stream,{ }))

Besides, it could be debatable whether this is a bug for __tostring, but the same issue exists with custom printers.

[pkulchenko]

@fab13n, I agree, it may make sense to handle __serialize differently. I think for consistency, __tostring should still be quoted.

Is it be sufficient for your purposes if __serialize outputs its result as is?

[fab13n]

I think tostring results shouldn't be quoted by Serpent itself. Otherwise, we can't tell the difference between a generated output and the corresponding literal string. In the example above I couldn't tell the difference between the outputs corresponding to the two following lines:

stream1 = { token("Keyword", "local"), token("Id", "x") }
stream2 = { [[token("Keyword", "local")]], [[token("Id", "x")]] }
assert(s.line(stream1)==s.line(stream2))

Sure, the version with mandatory quotes is valid Lua, but a valid Lua that's semantically different from the original one. It's arguably more treacherous, rather than safer. If users want to add quotes, they can add them in their __tostring() implementation. But the point of a pretty-printer is, well, to print something pretty, or at least readable. Compilability is mandatory for serialization, not for display.

As for using serialization instead of pretty-printing when one intends to pretty-print in a prettier way, I find this rather confusing :-) Besides, I don't even know whether I can use readability-improving options, such as indentation, with s.serialize().

[pkulchenko]

Yes, I do see this point and has struggled with the same considerations in the beginning. The main goal was to make the output consistent between serialization and pretty-printing as it's convenient in many situations take the "printed" output and paste it into a script. For example, when you use a console in the IDE to execute commands, is the result serialization or pretty-printing? It's probably mostly the latter, but I've seen on several occasions people taking the output from the console and pasting it into their code.

stream1 = { token("Keyword", "local"), token("Id", "x") }
stream2 = { [[token("Keyword", "local")]], [[token("Id", "x")]] }
assert(s.line(stream1)==s.line(stream2))

Right, but these are different values and I wanted them to look different. Similar to that, there is ambiguity with nil vs "nil", true vs. "true" and so on (which doesn't exist with the current output).

I can add a parameter to disable quoting of string values and would prefer keep the current option as the default.

[fab13n]

On Mon, Dec 30, 2013 at 7:01 PM, Paul Kulchenko notifications@github.comwrote:

assert(s.line(stream1)==s.line(stream2))

Right, but these are different values and I wanted them to look
different.

My point is, they are identical with the current implementation, and I want
them to look different!

[pkulchenko]

My point is, they are identical with the current implementation, and I want them to look different!

I understand, but they only look the same because your token() call returns its own representation.

There is still a difference between:

function token_mt :__tostring() return "true" end

-- and

function token_mt :__tostring() return true end

I pushed a change (in rawstring branch) that adds rawstring parameter:

print(s.block(stream, {rawstring = true}))

-- generates

{
  token(Keyword, local),
  token(Id, x)
} --[[table: 00C8C248]]

You can make it a default for all methods, so that your users don't need to specify it.

Will this work?

[agladysh]

I'm not sure if this is a correct approach. I think this should be configurable on per-object level. It makes little sense to me to configure a thing like this once per serialization call.

[pkulchenko]

I thought so as well; for example, adding something like __raw attribute to the metatable would signal that the result string should not be quoted (and we'd not need the serialization parameter), but I'd prefer not to clash with (possible) Lua values, like __mode. Maybe _raw or _rawstring?

[agladysh]

For example, when you use a console in the IDE to execute commands, is the result serialization or pretty-printing?

There is yet another use-case for pretty-printing while serializing — human-readable config files. This is a primary reason why in Lua Núcleo we have tpretty as well as tserialize (along with debug visualization of complex tables, of course).

So, we have at least three reasons to convert a Lua object to string:

• Machine-readable serialization (can load data back, don't care about readability)
• Human-readable serialization (can load data back, do care about readability)
• Human-readable visualization (don't care to load data back, do care about readability)

Human-readable serialization and visualization should be highly configurable to conform to usage modes (single-line logging, multi-line config files) and to coding guidelines (indentation rules, line width etc.)

[agladysh]

Maybe I'm missing something (didn't study serpent design too close yet, sorry), but I'm not sure why __tostring and __serialize are supported by default at all.

I'd start with __serpent metamethod, which will have (read-only) access to the configuration and will allow object author emit custom serialized data — to be pasted literally to the resulting string.

Then I'd consider enabling __tostring support for human-readable visualization by default — but only there.

To me __serialize metamethod name looks too generic — there is no well-established protocol for it, as far as I'm aware. But maybe it will make sense to support it as a generic serialization fallback for serialization (and only there), accepting no parameters and returning a string to, again, be pasted literally to the output.

[agladysh]

Furthermore, all this metamethod handling is a sugar that will slow down the basic implementation. Why not leave it to user to sort out in a custom serialization callback function? (Well, OK, and maybe provide some useful common implementations in an auxiliary module...)

[agladysh]

Maybe _raw or _rawstring?

Please don't use such generic names to control module-specific behaviour.

If you really want a flag, use __serpent_raw or something.