[db-wg] Proposal to allow non-ASCII characters in "org-name:", "person:" and "role:" attributes

Dear colleagues,

There was a question about UTF-8 support by major Whois providers during last week's DB-WG session at RIPE88.

During the UTF-8 discussion in December I checked the other RIRs as follows:

LACNIC: only Latin-1 encoded characters are accepted in updates (UTF-8 is ignored) but UTF-8 is returned on port 43.
	Example: whois -h whois.lacnic.net PAP12
APNIC: only Latin-1 is returned
	Example: whois -h testwhois.apnic.net YYYYMMDD-MNT

Subsequently I tested the other RIRs to be sure:

ARIN: UTF-8 is supported in the RPSL object and UTF-8 is returned on port 43.
	Example: whois -h whois.arin.net POC SHRYA12-ARIN
AFRINIC: UTF-8 characters are accepted in updates and UTF-8 is returned on port 43.
	Example: whois -h whois.afrinic.net SHRYANE-MNT

RIPE stores Latin-1 and returns Latin-1 on port 43.

So in summary, 3 RIRs return UTF-8 and 2 RIRs return Latin-1 on port 43.

Regards
Ed Shryane
RIPE NCC



> On 2 May 2024, at 16:02, Edward Shryane <eshryane at ripe.net> wrote:
> 
> Dear colleagues,
> 
> To follow-up on the UTF-8 discusssion in January, the DB team plans to implement support for UTF-8 in 3 phases:
> 
> (1) Add a flag to allow a client to choose a character set
> 
> In the Whois release 1.112, we have added the "-Z / --charset" query flag to allow clients to specify which character set they expect. The server response will encode RPSL objects using that character set.
> 
> This new flag can already be tested in the RC environment, e.g. the SHRYANE-MNT object contains "remarks:" attributes with non-ASCII (but still latin-1) characters:
> 
>    $ whois -h whois-rc.ripe.net -r shryane-mnt
>    $ whois -h whois-rc.ripe.net -r -Z utf8 shryane-mnt
> 
> This flag has no impact on the default behaviour of the RIPE database. This change only affects port 43, and the default character set remains latin-1.
> 
> This flag will already be useful for example, to capture responses as UTF-8 to file or use UTF-8 encoding in your terminal. In future, if the default on port 43 changes to UTF-8, then clients can keep latin-1 by using "-Z/--charset latin1".
> 
> (2) Convert the database schema to UTF-8
> 
> In the following Whois release, the DB team plans to switch the RIPE database schema character set from latin-1 to UTF-8. This will allow Whois to store UTF-8 strings in the database index tables.
> 
> Switching the database schema character set will involve about 1 hour of downtime to Whois updates, and Whois queries will not be affected. We will announce this change in advance.
> 
> This change will have no impact on the default behaviour of the RIPE database. All interfaces will behave as before, and RPSL objects will remain latin-1 encoded internally.
> 
> (3) Allow UTF-8 to be used in RPSL objects
> 
> Once the RIPE database schema supports the UTF-8 character set, the DB team will create a further Whois release that will allow UTF-8 to be used in RPSL objects, in addition to the index tables.
> 
> The default behaviour of the RIPE database will remain the same. All interfaces will behave as before, but RPSL objects will use UTF-8 internally.
> 
> In future, if the DB-WG decides to allow UTF-8 characters in RPSL, the database will already support it.
> 
> Regards
> Ed Shryane
> RIPE NCC
> 
> 
>> On 18 Jan 2024, at 10:34, Edward Shryane <eshryane at ripe.net> wrote:
>> 
>> Dear colleagues,
>> 
>> Based on the discussion regarding UTF-8 in the RIPE database during the interim meeting yesterday, I suggest that we implement support for UTF-8 in the database (i.e. convert the schema and add a flag to allow a client to choose a character set), but we do not allow additional characters for now, pending further DB-WG discussion. Our intention is to lay the groundwork for future support, without breaking existing functionality. If you have any concerns or objections please let me know.
>> 
>> We will now prepare an implementation plan / impact analysis of these changes.
>> 
>> Regards
>> Ed Shryane
>> RIPE NCC
>> 
>> 
>>> On 24 Nov 2023, at 10:03, Edward Shryane via db-wg <db-wg at ripe.net> wrote:
>>> 
>>> Dear colleagues,
>>> 
>>> Currently the RIPE database only allows a subset of ASCII characters in the "org-name:", "person:" and "role:" attributes, for a few reasons including:
>>> 
>>> * These attributes are also a look-up key and the Whois protocol does not allow specifying character sets in queries.
>>> * RPSL names are ASCII according to RFC2622
>>> * Using a normalised name makes the object easier to query
>>> * Reading a normalised name is easier to interpret
>>> 
>>> However there are some drawbacks to forcing names to only use a subset of ASCII characters:
>>> 
>>> * Organisations, roles and persons cannot use their actual name if it includes characters outside this subset.
>>> * Normalisation is not standard, but is an interpretation done by each maintainer, e.g. characters could be excluded or converted in different ways.
>>> 
>>> Since we support the Latin-1 character set in the RIPE database, I propose we also allow non-ASCII Latin-1 characters in these attributes.
>>> 
>>> Querying for a name can be done either using the latin-1 characters (proposed) or a normalised, ASCII representation (currently). The normalised version will be generated by Whois and stored in a database index for querying. The primary key will also be generated from the normalised version.
>>> 
>>> Please let me know your feedback.
>>> 
>>> Regards
>>> Ed Shryane
>>> RIPE NCC
>>> 
>>> ---
>>> 
>>> Whois attribute verbose description (copied from the help text).
>>> 
>>> org-name
>>> --------
>>> Specifies the name of the organisation that this organisation object
>>> represents in the RIPE Database. This is an ASCII-only text attribute.
>>> The restriction is because this attribute is a look-up key and the
>>> whois protocol does not allow specifying character sets in queries.
>>> The user can put the name of the organisation in non-ASCII character
>>> sets in the "descr:" attribute if required.
>>> 
>>> A list of 1 to 30 words separated by white space. 
>>> A word is made up of ASCII alphanumeric characters and additionally: ][)(._"*@,&:!'`+/-
>>> A word may have up to 64 characters and is not case sensitive. 
>>> Each word can have any combination of the above characters with no restriction on the start or end of a word.
>>> 
>>> person
>>> ------
>>> Specifies the full name of an administrative, technical or zone
>>> contact person for other objects in the database.
>>> 
>>> It should contain 2 to 10 words.
>>> A word is made up of ASCII alphanumeric characters and additionally: .`'_-
>>> The first word should begin with a letter.
>>> At least one other word should also begin with a letter.
>>> Max 64 characters can be used in each word.
>>> 
>>> role
>>> ----
>>> Specifies the full name of a role entity, e.g. RIPE DBM.
>>> 
>>> A list of 1 to 30 words separated by white space.
>>> A word is made up of ASCII alphanumeric characters and additionally: ][)(._"*@,&:!'`+/-
>>> A word may have up to 64 characters and is not case sensitive. 
>>> Each word can have any combination of the above characters with no restriction on the start or end of a word.
>>> 
>>> 
>>> -- 
>>> 
>>> To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://mailman.ripe.net/
>> 
>