This is great feedback for a future updated scoring mechanism.
When assigning scores to the tests in the self-certification guide, we had a few challenges and tradeoffs:
- Distinguishing between components and systems which could host internationalized mailboxes and those which could
only send/receive to/from internationalized addresses
- Scoring the
individual components from which a system could be assembled versus scoring
complete end to end systems
- Checklist
pass/fail scoring versus weighed scoring
- Limiting the number of badges
The tradeoffs we selected resulted in scoring wherein even the very best system which can only send/receive can never get a score higher than Silver. There was a lot of debate about that!
In contrast, a more-complex 6-badge system might have awarded Gold or Platinum to Gmail even as a send/receive system (what we used to call “level 1”) if we had made the tradeoffs differently.
/marksv
From: Mutegeki Cliff via UA-discuss <ua-discuss@icann.org>
Sent: Monday, June 30, 2025 5:00 AM
To: anil Jain <asdj1990@gmail.com>
Cc: John Levine <john.levine@standcore.com>; ua-discuss@icann.org
Subject: [EXTERNAL] [UA-discuss] Re: Final report on the EAI self-hosting project
Dear John,
Thanks for sharing all of this—great work! It’s impressive to see the EAI self-hosting server project wrapped up so thoroughly. I went through the report and slides, and the level of detail is
excellent. The demo months ago already gave a strong impression of the system’s performance, so it’s interesting (and a bit surprising) to see the disparity between the real-world performance and the self-evaluation test results.
I completely agree with your point about considering an overall user experience score. Some of the failed tests do seem marginal in terms of actual impact. It might be helpful to weight the evaluation criteria accordingly or create a separate UX effectiveness
metric.
I'll review the scripts and updated documentation more closely over the next few days and get back to you with any specific comments or suggestions.
Also, I wanted to ask if you might be open to mentoring during a DNS Hackathon we’re organizing this August as part of the upcoming Uganda DNS Forum. Your experience with the EAI self-hosting project would be incredibly valuable to participants, especially
those working on internationalization and DNS infrastructure. The time commitment would be minimal — mainly offering guidance and feedback to teams — and your insights could make a real difference. Let me know if this is something you’d be interested in. We’d
truly appreciate your support!
Thanks again for pulling all of this together!
|
Mutegeki Cliff Agaba |
|
|
Digital Adoption Consultant |
|
|
+256 701 800 679 |
|
|
+256 776 800 679 |
|
|
|
|
On Sat, 28 Jun 2025 at 20:42, anil Jain via UA-discuss <ua-discuss@icann.org> wrote:
John,
I must congratulate you for exemplary work undertaken by you.
I hope UA community will get highly benefitted with your work.
Congratulations John.
Anil
Sent from my iPhone
> On 27 Jun 2025, at 23:19, John Levine via UA-discuss <ua-discuss@icann.org> wrote:
>
> I think I have finally finished the EAI self-hosting server project. The report on the resuts is here:
>
> https://docs.google.com/document/d/1ThjISfMkAn6bP0yTXP-uBHebvMkPwr3LxuG_B4YnxHo/edit?usp=sharing
>
> And a slide presentation is here:
>
> https://docs.google.com/document/d/1ThjISfMkAn6bP0yTXP-uBHebvMkPwr3LxuG_B4YnxHo/edit?usp=sharing
>
> One surprising result is that even though the server works quite well, as we saw in the demo I
> did a few months ago, it scores poorly on the self-evaluation tests. It looks to me like
> many of the tests it fails have little effect on the user's experience, so you might want to
> think about an overall experience score rather than just adding up individual tests.
>
> The scripts are in an ICANN github repo here:
>
> https://github.com/icann/eaiselfhost
>
> The installation instructions, slightly updated from last time are here:
>
> https://docs.google.com/document/d/1azNEZAdk6wyg0PNHBuXWWrKF1ZexQjv0Txyd1zaU8O8/edit?usp=sharing
>
> And the installation tutorial is here:
>
> https://docs.google.com/presentation/d/1dva_lS7nMJNx8mgRQExMy0jGONUGaLFMiM8Q5zKN4UA/edit?usp=sharing
>
> Take a look, let me know if you have any comments.
>
> R's,
> John
> _______________________________________________
> UA-discuss mailing list -- ua-discuss@icann.org
> To unsubscribe send an email to ua-discuss-leave@icann.org
> _______________________________________________
> By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
_______________________________________________
UA-discuss mailing list -- ua-discuss@icann.org
To unsubscribe send an email to ua-discuss-leave@icann.org
_______________________________________________
By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.