[db-wg] Proposal to allow non-ASCII characters in "org-name:", "person:" and "role:" attributes

Dear colleagues,

To follow-up on the UTF-8 discusssion in January, the DB team plans to implement support for UTF-8 in 3 phases:

(1) Add a flag to allow a client to choose a character set

In the Whois release 1.112, we have added the "-Z / --charset" query flag to allow clients to specify which character set they expect. The server response will encode RPSL objects using that character set.

This new flag can already be tested in the RC environment, e.g. the SHRYANE-MNT object contains "remarks:" attributes with non-ASCII (but still latin-1) characters:

    $ whois -h whois-rc.ripe.net -r shryane-mnt
    $ whois -h whois-rc.ripe.net -r -Z utf8 shryane-mnt

This flag has no impact on the default behaviour of the RIPE database. This change only affects port 43, and the default character set remains latin-1.

This flag will already be useful for example, to capture responses as UTF-8 to file or use UTF-8 encoding in your terminal. In future, if the default on port 43 changes to UTF-8, then clients can keep latin-1 by using "-Z/--charset latin1".

(2) Convert the database schema to UTF-8

In the following Whois release, the DB team plans to switch the RIPE database schema character set from latin-1 to UTF-8. This will allow Whois to store UTF-8 strings in the database index tables.

Switching the database schema character set will involve about 1 hour of downtime to Whois updates, and Whois queries will not be affected. We will announce this change in advance.

This change will have no impact on the default behaviour of the RIPE database. All interfaces will behave as before, and RPSL objects will remain latin-1 encoded internally.

(3) Allow UTF-8 to be used in RPSL objects

Once the RIPE database schema supports the UTF-8 character set, the DB team will create a further Whois release that will allow UTF-8 to be used in RPSL objects, in addition to the index tables.

The default behaviour of the RIPE database will remain the same. All interfaces will behave as before, but RPSL objects will use UTF-8 internally.

In future, if the DB-WG decides to allow UTF-8 characters in RPSL, the database will already support it.

Regards
Ed Shryane
RIPE NCC


> On 18 Jan 2024, at 10:34, Edward Shryane <eshryane at ripe.net> wrote:
> 
> Dear colleagues,
> 
> Based on the discussion regarding UTF-8 in the RIPE database during the interim meeting yesterday, I suggest that we implement support for UTF-8 in the database (i.e. convert the schema and add a flag to allow a client to choose a character set), but we do not allow additional characters for now, pending further DB-WG discussion. Our intention is to lay the groundwork for future support, without breaking existing functionality. If you have any concerns or objections please let me know.
> 
> We will now prepare an implementation plan / impact analysis of these changes.
> 
> Regards
> Ed Shryane
> RIPE NCC
> 
> 
>> On 24 Nov 2023, at 10:03, Edward Shryane via db-wg <db-wg at ripe.net> wrote:
>> 
>> Dear colleagues,
>> 
>> Currently the RIPE database only allows a subset of ASCII characters in the "org-name:", "person:" and "role:" attributes, for a few reasons including:
>> 
>> * These attributes are also a look-up key and the Whois protocol does not allow specifying character sets in queries.
>> * RPSL names are ASCII according to RFC2622
>> * Using a normalised name makes the object easier to query
>> * Reading a normalised name is easier to interpret
>> 
>> However there are some drawbacks to forcing names to only use a subset of ASCII characters:
>> 
>> * Organisations, roles and persons cannot use their actual name if it includes characters outside this subset.
>> * Normalisation is not standard, but is an interpretation done by each maintainer, e.g. characters could be excluded or converted in different ways.
>> 
>> Since we support the Latin-1 character set in the RIPE database, I propose we also allow non-ASCII Latin-1 characters in these attributes.
>> 
>> Querying for a name can be done either using the latin-1 characters (proposed) or a normalised, ASCII representation (currently). The normalised version will be generated by Whois and stored in a database index for querying. The primary key will also be generated from the normalised version.
>> 
>> Please let me know your feedback.
>> 
>> Regards
>> Ed Shryane
>> RIPE NCC
>> 
>> ---
>> 
>> Whois attribute verbose description (copied from the help text).
>> 
>> org-name
>> --------
>> Specifies the name of the organisation that this organisation object
>> represents in the RIPE Database. This is an ASCII-only text attribute.
>> The restriction is because this attribute is a look-up key and the
>> whois protocol does not allow specifying character sets in queries.
>> The user can put the name of the organisation in non-ASCII character
>> sets in the "descr:" attribute if required.
>> 
>> A list of 1 to 30 words separated by white space. 
>> A word is made up of ASCII alphanumeric characters and additionally: ][)(._"*@,&:!'`+/-
>> A word may have up to 64 characters and is not case sensitive. 
>> Each word can have any combination of the above characters with no restriction on the start or end of a word.
>> 
>> person
>> ------
>> Specifies the full name of an administrative, technical or zone
>> contact person for other objects in the database.
>> 
>> It should contain 2 to 10 words.
>> A word is made up of ASCII alphanumeric characters and additionally: .`'_-
>> The first word should begin with a letter.
>> At least one other word should also begin with a letter.
>> Max 64 characters can be used in each word.
>> 
>> role
>> ----
>> Specifies the full name of a role entity, e.g. RIPE DBM.
>> 
>> A list of 1 to 30 words separated by white space.
>> A word is made up of ASCII alphanumeric characters and additionally: ][)(._"*@,&:!'`+/-
>> A word may have up to 64 characters and is not case sensitive. 
>> Each word can have any combination of the above characters with no restriction on the start or end of a word.
>> 
>> 
>> -- 
>> 
>> To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://mailman.ripe.net/
>