✨ Proposal: redefine hostname and idn-hostname formats

[SorinGFS]

Describe the inspiration for your proposal

Not long ago I read many times the RFCs related to idn-hostname parser I made, 5890/5891, which then took me through 5892/5893 to finally reach UTS#46. I realized at that time that in fact 5890/5891 were just a framework that prepared what was to come. Then, I started reading RFCs 3986/3987 for the identifier-js uri/iri validator to see exactly the same thing: they were not the rigid texts that I had in mind but were also frameworks that distribute and delegate responsibilities to various other specifications. While building the validator I always compared it to whatwg-url, which incorporates both uri/iri in a single specification, and I was struck by the fact that they treat urls differently depending on the scheme. I looked to see where they got the specifications from and found that they go back exactly to 3986/3987 RFCs. I have often made the mistake of jumping directly to the relevant normative texts when reading RFCs, but here I realized that something was missing, and I sought to understand more precisely what the authors of the texts were thinking. In a field of "URI/IRI" complexity, they couldn't put all the possible variations into a single text. So, they established this general framework for any valid URI/IRI, and delegated each scheme specific documentation to the related RFC's from the very beginning.

Abstract
A Uniform Resource Identifier (URI) is a compact sequence of
characters that identifies an abstract or physical resource. This
specification defines the generic URI syntax and a process for
resolving URI references that might be in relative form, along with
guidelines and security considerations for the use of URIs on the
Internet. The URI syntax defines a grammar that is a superset of all
valid URIs, allowing an implementation to parse the common components
of a URI reference without knowing the scheme-specific requirements
of every possible identifier. This specification does not define a
generative grammar for URIs; that task is performed by the individual
specifications of each URI scheme.

So, what are those specific schemes:

• http, https RFC7230
• ws, wss RFC6455
• file RFC8089
• mailto RFC6068
• data RFC2397
  There are few more notorious schemes, like the obsoleted ftp, or whatwg specific blob.

I made a research about each of those schemes specificity (not 100% sure about accuracy yet):

Scheme	Userinfo	Host (reg_name)	Port	Path	Query	Fragment
http	optional	required	optional	required	optional	optional
https	optional	required	optional	required	optional	optional
ws	useless	required	optional	required	optional	useless
wss	useless	required	optional	required	optional	useless
file	optional	optional	forbidden	required	optional	optional
ftp	optional	required	optional	required	useless	useless
mailto	useless	useless	useless	required	optional	useless
data	useless	useless	useless	required	useless	useless
blob	useless	useless	useless	required	useless	useless

When we read the related specs we find that the network resource specific schemes define their uris like: http-URI, https-URI, ws-URI, wss-URI, file-URI, all being subsets of the generic URI, and all pointing their base back to RFC3986. As for the other uri parts, for example this is how are they defined in http:

URI-reference = <URI-reference, see [RFC3986], Section 4.1>
absolute-URI  = <absolute-URI, see [RFC3986], Section 4.3>
relative-part = <relative-part, see [RFC3986], Section 4.2>
scheme        = <scheme, see [RFC3986], Section 3.1>
authority     = <authority, see [RFC3986], Section 3.2>
uri-host      = <host, see [RFC3986], Section 3.2.2>
port          = <port, see [RFC3986], Section 3.2.3>
path-abempty  = <path-abempty, see [RFC3986], Section 3.3>
segment       = <segment, see [RFC3986], Section 3.3>
query         = <query, see [RFC3986], Section 3.4>
fragment      = <fragment, see [RFC3986], Section 3.5>

I think I'm not wrong if I say that in IT when we say hostname we think of url.hostname. Maybe we were thinking that the URI RFC is too rigid (considering only that abnf) to cover this option. But perhaps judging things in the light of the above we can see that this is not the case. The RFC3986 has that generic syntax for reg_name as a fallback for non-specific cases, because it defines it:

A URI resolution implementation might use DNS,
host tables, yellow pages, NetInfo, WINS, or any other system for
lookup of registered names.

but, for the specific case when resources are located on the network sais:

A registered name intended for lookup in the DNS uses the syntax
defined in Section 3.5 of [RFC1034] and Section 2.1 of [RFC1123].

and:

If the URI scheme defines a default for host, then that default
applies when the host subcomponent is undefined or when the
registered name is empty (zero length). For example, the "file" URI
scheme is defined so that no authority, an empty host, and
"localhost" all mean the end-user's machine, whereas the "http"
scheme considers a missing authority or empty host invalid.

Note that the reg_name definition is here differentiated based on scheme.

and:

The reg-name syntax allows percent-encoded octets in order to
represent non-ASCII registered names in a uniform way that is
independent of the underlying name resolution technology. Non-ASCII
characters must first be encoded according to UTF-8 [STD63], and then
each octet of the corresponding UTF-8 sequence must be percent-
encoded to be represented as URI characters. URI producing
applications must not use percent-encoding in host unless it is used
to represent a UTF-8 character sequence. When a non-ASCII registered
name represents an internationalized domain name intended for
resolution via the DNS, the name must be transformed to the IDNA
encoding [RFC3490] prior to name lookup. URI producers should
provide these registered names in the IDNA encoding, rather than a
percent-encoding, if they wish to maximize interoperability with
legacy URI resolvers.

Here the RFC3490 infact must be read as RFC5980/5981 that obsoletes it. I read this as a back delegation that allows the storage of an IRI as URI, facilitating back and forth conversion between.

In conclusion, what I understand is this:

yellow.pages//john%20doe/phone is an allowed use of % in reg_name, but if the scheme is http it can only be a transitory state to transform that input into its unicode version delegated to RFC5890/5891 (and its delegations to RFC5892, RFC5893, and UTS#46) in order to give back the URI version of it (as we know the DNS is ASCII-only and will not change in our lifetime). Also, for network involving schemes, the other DNS rules are also mandatory (hyphen-rules, non empty labels, so on...).

Describe the proposal

• replace the hostname format requirement rule to RFC 3986 reg_name abnf when scheme implies a network resource (http,https,ws,wss,file, or any other similar)
• replace the idn-hostname format requirement rule to RFC 3987 ireg_name abnf when scheme implies a network resource (http,https,ws,wss,file, or any other similar)

Implicitly, clarify the specs for uri, iri, uri-reference and iri-reference formats about scheme specific rules.

Describe alternatives you've considered

No response

Additional context

No response

[jdesrosiers]

• replace the hostname format requirement rule to RFC 3986 reg_name abnf when scheme implies a network resource (http,https,ws,wss,file, or any other similar)
• replace the idn-hostname format requirement rule to RFC 3987 ireg_name abnf when scheme implies a network resource (http,https,ws,wss,file, or any other similar)

I don't think that makes sense. There is no scheme in context for hostname/idn-hostname. The scheme is part of the URI.

clarify the specs for uri, iri, uri-reference and iri-reference formats about scheme specific rules.

We talked about scheme-specific requirements in a recent discussion about URI normalization. In that case we decided that scheme-based (and protocol-based) normalization rules should be optional. There are currently over 400 registered URI schemes. I don't think it's reasonable to expect implementations to support the specific rules for each scheme, not to mention having to keep up when new schemes are registered.

I think there are a couple of options here and either way, I think we need a spec update to clarify.

1. Any scheme-based validation is optional
2. Schema-based validation is not allowed
3. Require an implementation to throw an error if it encounters a scheme it doesn't implement
4. Define a set of schemes that implementations must support scheme-based rules

My preference would be option 1. I think implementations shouldn't have to implement URI validation themselves. Most language's have a standard library that already has an implementation and we want people to be able to use those. That means there's a little variation that will be allowed, but it would be the same situation we have with regular expressions. We don't expect people to implement a specific regular expression engine, we just document that you can't always rely on all regex features. I think that's fine for URI validation as well.

If we really want to make sure that all validator behave the same, we could go with option 2 or 3, but I think it's not always known what schemes a standard library supports and you can't turn scheme validation off if you wanted to. Do we even know if standard libraries typically support schemes? I'm not sure they do.

[SorinGFS]

We talked about scheme-specific requirements in a recent discussion about URI normalization. In that case we decided that scheme-based (and protocol-based) normalization rules should be optional. There are currently over 400 registered URI schemes. I don't think it's reasonable to expect implementations to support the specific rules for each scheme, not to mention having to keep up when new schemes are registered.

Nice conversation you had there, but I disagree with the conclusion. Sure, there are over 400 registered uri schemes and counting, but a user should only care about the correctness of that one that he uses. As RFC3986 - Abstract states:

This specification does not define a generative grammar for URIs; that task is performed by the individual specifications of each URI scheme.

So, if the scheme is IANA registered an implementer should follow:

• http points to RFC9110, Section 4.2.1
• https points to RFC9110, Section 4.2.2
• ...

Then, the referenced protocols are pointing back to RFC3986 for the generic rules they use from there.

Here we talk about hostname and idn-hostname so here we should only care about the schemes that are used to retrieve resources over the network.

I analyzed whatwg-url spec which has this goal:

Align RFC 3986 and RFC 3987 with contemporary implementations and obsolete the RFCs in the process. (E.g., spaces, other "illegal" code points, query encoding, equality, canonicalization, are all concepts not entirely shared, or defined.) URL parsing needs to become as solid as HTML parsing. [RFC3986] [RFC3987]

First, they start from RFC3986/3987 in their journey, and then inplementing scheme specific rules. I looked for things that are not conform with RFC3986/3987 + scheme specific rules in their implementation, and so far I got none! Here is how they deal with reg_name if the scheme is generic or http:

console.log(new URL('yellow.pages://john%20doe').href); // yellow.pages://john%20doe
console.log(new URL('http://john%20doe').href); // throws invalid URL input: 'http://john%20doe'

I think there are a couple of options here and either way, I think we need a spec update to clarify.
1. Any scheme-based validation is optional
2. Schema-based validation is not allowed
3. Require an implementation to throw an error if it encounters a scheme it doesn't implement
4. Define a set of schemes that implementations must support scheme-based rules

Here, 1) I think would lead to maximum ambiguity, 2) would ignore what I presented above, 3) would be wrong because it would reject the generic URI's, so I think the only viable option would be 4) and the list is not that long because we talk about retrieving resources over the network:

• http
• https
• file
• ws
• wss

Now, if we talk about uri, iri, uri-reference, iri-reference format definitions in general, I think they shouldn't be limited to those 5 defined above, but the spec should state that if the format has a scheme that is a IANA registered scheme, then should obey the rules indicated by the corresponding IANA scheme. This would eliminate any ambiguity about what those formats means, and if one wants to define an identifier that does not colide with any defined scheme, then should use the generic uri:// and iri:// schemes that shall never be defined by IANA.

• replace the hostname format requirement rule to RFC 3986 reg_name abnf when scheme implies a network resource (http,https,ws,wss,file, or any other similar)
• replace the idn-hostname format requirement rule to RFC 3987 ireg_name abnf when scheme implies a network resource (http,https,ws,wss,file, or any other similar)

I don't think that makes sense. There is no scheme in context for hostname/idn-hostname. The scheme is part of the URI.

I left this part at the end intentionally. Sure, there is no scheme in context for hostname/idn-hostname, but there is an implicit requirement for reg_name if the resource is located on the network, as presented above. I chose that expression because that is the part that has variable meaning when the schema involves a network resource. Each protocol has its own naming convention for that part (e.g: http-host), but the URI spec is the one that defines the general name: reg_name.

So, if you are choosing to revise the uri, iri, uri-reference, iri-reference format definitions to keep account the shema specificity, I think you can also make that connection for hostname (and idn-hostname) to represent uri/iri reg_name for the cases when schema implies the retrieval of the resource from the network: http, https, ws, wss and file. I don't know you, but for me when I hear hostname I think of uri.hostname.

[gregsdennis]

I don't think it's reasonable to expect implementations to support the specific rules for each scheme, not to mention having to keep up when new schemes are registered. - @jdesrosiers

This is my assessment as well.

but a user should only care about the correctness of that one that he uses - @SorinGFS

That may be, but as an implementation provider, I don't know which ones my client may use. This suggests that I support them all, which, as stated above, is unreasonable.

Of the 4 options @jdesrosiers gives, I think (3) is probably most in line with the way we've defined requirements around format in general, but personally, I'd rather not have to have specific support.

[SorinGFS]

That may be, but as an implementation provider, I don't know which ones my client may use. This suggests that I support them all, which, as stated above, is unreasonable.

The RFC 3986 sais:

This specification does not define a generative grammar for URIs; that task is performed by the individual specifications of each URI scheme.

This basically sais that does not support anything as is! And making the validator for uri/iri/uri-reference/iri-reference formats that I'm currently working on I understood just that! It would be useless in real programs to define uris that are not meant for something specific. Using such strings as identifiers for some resources would basically carry the same worth as a uuid. It would be an identifier string for an abstract resource, not a real one. So, that's why I think that adding scheme speciicity support would empower those formats with some meaning.

I'm aware that no implementation can support them all, not even 10% of them, that is not a desirable objective. But what you can do IMO is to narrow down the scheme specificity to those that your spec is actually using:

• http, https, ws, wss, file for network resources
• uri/iri for abstract identifiers
• and implementations should throw an error if none of the above (or just ignore the unknown ones, like whatwg does)

[jdesrosiers]

I left this part at the end intentionally. Sure, there is no scheme in context for hostname/idn-hostname, but there is an implicit requirement for reg_name if the resource is located on the network, as presented above.

I think you're saying that uri/uri-reference should verify that if the reg_name part is matched in the URI, then that part should be validated according to hostname. (And similar for iri/iri-reference and idn-hostname.) Is that correct? That's a reasonable argument and we can talk about that, but I'm still not sure that's what you're saying. The issue description says, "redefine hostname and idn-hostname formats", which is not the same thing. This suggestion would redefine the uri/etc formats, not hostname/idn-hostname.

I'm aware that no implementation can support them all, not even 10% of them, that is not a desirable objective. But what you can do IMO is to narrow down the scheme specificity to those that your spec is actually using:

Our spec doesn't have any limitations on which schemes can be used. I don't see any reason why it should. If someone is writing a schema for some data that includes a URI with an obscure scheme, it makes perfect sense for them to use the uri format in their schema. Even the way JSON Schema uses URIs itself doesn't limit what schemes can be used. Users can register schemas using whatever URI schemes they want.

This basically sais that does not support anything as is!

I don't agree with that. I agree that it has limitations, but it's far from useless. For example, the email format only checks if something looks like an email. It doesn't check if the domain is registered and a mail server is accessible or that the account actually exists. If you need to know that something is a valid email, the only foolproof way to validate it is to send an email and see if it gets delivered. Obviously, a JSON Schema implementation would never be expected to do that. But, that doesn't make the email format useless. It can check what's reasonable to check and leave further validation to the application.

I see the uri/etc formats the same way. They validate that something conforms to the RFC 3986/7 structure. If you need a specific kind of URI, that further validation is left to the application. It's the same as if you need to validate that the URI actually exists on the web. You'd have to do an HTTP request to confirm. The question is just about where to draw the line. My position is just that we draw the line where it's easiest for implementers to use existing standard libraries.

I looked for things that are not conform with RFC3986/3987 + scheme specific rules in their implementation, and so far I got none!

I think that's a good indication that it's common that implementations support scheme-specific rules, but you're testing a WHATWG URL implementation, not an IETF URI implementation. One difference I'm aware of is that WHATWG URLs don't support URLs without an authority like urn: or tag: or the rootless version of file:. So, JSON Schema implementations can't use a WHATWG URL implementation, at least not for format validation. Which means, I'm not sure the results are representative. We need a more information. I still think option (1) is sufficient. It allows most implementations to use standard libraries and I don't think the variation is likely to be significant. If it turns out that most standard libraries implement scheme-specific rules, then that's what we'll get. Most implementations will support scheme-specific rules.

[SorinGFS]

I think you're saying that uri/uri-reference should verify that if the reg_name part is matched in the URI, then that part should be validated according to hostname. (And similar for iri/iri-reference and idn-hostname.) Is that correct?

Yes, but only if the uri/iri scheme is one of those requiring that reg_name to be interpreted as hostname/idn-hostname (http, https, ws, wss, file).

The issue description says, "redefine hostname and idn-hostname formats", which is not the same thing. This suggestion would redefine the uri/etc formats, not hostname/idn-hostname.

Why not both? If uri/etc formats are redefined, couldn't hostname/idn-hostname to be redefined too as reg_name for the case when scheme implies network resources? And you know what: the way you choose to solve this puzzle is not important for me. I just noticed some missing pieces and I guess the best resolution would come from you.