Digital accessibility monitoring in Luxembourg

1. Executive summary

The Web Accessibility Directive has been transposed to Luxembourg’s legislation in the law of 28 May 2019. It obliges the public sector to improve the accessibility of its websites and mobile applications in order to respect the norms stipulated by the EU directive.

The Government’s information and press service (SIP – Service information et presse du gouvernement) is commissioned to periodically verify the websites and mobile applications within the scope of the law and to make sure that public sector organisms apply the legal rules in this matter. SIP also performs general information and awareness-raising activities, and provides training to public sector officials.

The auditing task consists of three parts: in-depth audits of websites, simplified audits of websites, and audits of mobile applications.

This report covers the period from 2020 to 2021 and is the first to be delivered in the scope of the directive. It describes the initial measures to implement a national auditing system as well as national reference frameworks for the evaluation of the accessibility of websites and mobile applications. During this period, SIP has performed 15 in-depth audits of websites, 87 simplified audits of websites, and 2 audits of mobile applications. For the in-depth monitoring, the average compliance score is 44%, and 55% for the simplified monitoring.

SIP is also in charge of managing complaints regarding digital accessibility compliance problems. During 2020 and 2021, 18 complaints have been received.

Multiple awareness activities have been put into place, including numerous days of specific training for public servants.

The complete documents about the auditing activity as well as the accessibility declarations and the complaints are published on Luxembourg’s open data portal:

• Inventory of public sector websites

• Inventory of public sector applications

• In-depth audits of public sector websites and applications

• Simplified audits of websites

• Accessibility statements

• Complaints about accessibility problems

2. Description of the monitoring activities

2.1. General information

In the current period (2020-2021), the monitoring activities have been performed between 27 August 2020 and 26 October 2021.

As stipulated in the law of 28 May 2019, the body in charge of the monitoring is the “Information and Press Service”, more specifically its division “Open Data and access to information” (e-mail address: accessibilite@sip.etat.lu).

The in-depth and mobile audits have been mostly outsourced to an external company specialised in digital accessibility, whereas the simplified audits have been taken care of internally.

The number of audited websites and mobile apps is based on the objectives set by the EU according to the population size of each country. For Luxembourg, in the first control period, these objectives resulted in the following figures:

• 15 in-depth audits of websites,

• 87 simplified audits of websites,

• 2 audits of mobile apps.

For each type of audit, it was necessary to define the websites or applications that should be part of the respective samples. For this purpose, and in collaboration with other governmental actors, a first step was made to build inventories of websites and applications owned by public sector bodies. For complete transparency, these documents were published on the national data portal. Based on these lists, samples were subsequently proposed by SIP with the objective of representing as much as possible the different governmental and administrative sectors and levels, as required in the commission implementing decision (EU) 2018/1524. The stakeholders of the handicap sector in Luxembourg were involved in this process, and their feedback was taken into account during the process (cf. section 5.1 Description of the mechanisms for consulting with relevant stakeholders on the accessibility of websites and mobile applications).

It is worthwhile noting that the sample slightly evolved during the control period as some websites went offline in the meantime or were merged into other existing websites. The corresponding websites were in those cases replaced in the sample by other websites from the same category whenever possible. Due to the size of the country, this was not always evident, because of the small numbers of sites available.

Figure 1: Public service sector coverage in the website sample (see description below)

Figure 2: Administrative levels of the websites in the sample (see description below)

In our inventory, we have not found any website at the regional level, which reflects the fact that in Luxembourg, the regional (canton, district) subdivisions of the territory are less visible. The local level websites included in the sample are all owned by major cities.

2.2. Composition of the sample

Our sample is available as Open Data on Luxembourg’s Open Data Portal.

The composition of the sample can be described as follows:

• Total number of websites and mobile applications included in the sample: 104

• Number of websites monitored using the simplified monitoring method: 87

• Number of websites and mobile apps monitored using the in-depth monitoring method: 17

• Number of monitored websites per administrative levels

  1. State websites: 67

  2. Regional websites: 0

  3. Local websites: 16

  4. Websites of bodies governed by public law not belonging to categories (a) to (c): 19

• Number of websites from the respective sectors, as retained in the sample:

  • social protection: 9

  • health: 7

  • transport: 6

  • education: 10

  • employment and taxes: 8

  • environmental protection: 7

  • recreation and culture: 10

  • housing and community amenities: 7

  • public order and safety: 7

  • Other: 31

• Distribution of the sample for mobile applications across the different operating systems

  • iOS: 1

  • Android: 1

2.3. Correlation with the standards, technical specifications and tools used for monitoring

2.3.1. In-depth monitoring method for websites

Our in-depth monitoring method is based on the one defined in RGAA 4.1 (General Framework for the Improvement of Web-Accessibility, in French “Référentiel Général d’Amélioration de l’Accessibilité web”), the accessibility framework defined by the DINUM in France, which has been slightly adapted to the Luxembourgish law. It is important to note that no usability tests were involved in the in-depth monitoring, but only accessibility tests.

The RGAA technical method provides an operational framework for verifying compliance with accessibility requirements. It includes 106 control criteria including an average of 2.5 tests per criterion. Some tests refer to implementation techniques (HTML, CSS, JavaScript ...) and are very precisely defined in order to reduce the margin of interpretation regarding the compliance with the norm EN 301 549.

RGAA 4.1 fully supports the requirements identified in the EU norm EN 301 549 v 2.1.2 as described in the mapping table available in Annexe 1: Mapping RGAA 4.1 / EN 301 549 v2.1.2 / WCAG 2.1.

In this method, a sample of pages is evaluated against RGAA criteria. The respective website sample usually contains the following pages when they exist:

• the home page

• the legal notice

• the authentication page

• the contact page

• the "accessibility" page

• the "help" page

• the "site map" page

• at least one relevant page for each type of service provided and any other main intended use (e.g.: 1st level headings in the tree structure), including the search functionality

• at least one relevant downloadable document, if applicable, for each type of service provided and for any other primarily intended use

• all the pages constituting a process (for example, an entry form or a transaction on several pages)

• examples of pages having a distinct appearance or presenting a different type of content (e.g. page containing data tables, multimedia elements, illustrations, forms, etc.).

The selection of the audited pages as well as their number must be representative of the site. Finally, there are pages selected at random representing at least 10% of the elements of the sample described above.

Some of the criteria of RGAA, mainly linked to the evaluation of the accessibility of rich user interface components based on JavaScript, have been tested with different combinations of screen readers and web browsers, as described in the testing environment of RGAA. These tests have been performed on desktop and mobile devices with combinations among the following:

The (non-normative) test method of RGAA recommends a set of tools for the audits:

• Web Developer Toolbar

• Wave

• W3C Markup Validation Service

• WCAG Color Contrast Checker for Chrome or WCAG Color Contrast Checker for Firefox

• Colour Contrast Analyser

• HeadingsMap for Chrome or HeadingsMap for Firefox

• PAC: PDF Accessibility Checker

• PEAT: Photosensitive Epilepsy Analysis Tool

Note: Using these tools is not required by the framework.

2.3.2. Simplified monitoring method for websites

Our simplified monitoring method is directly built on top of RGAA and it uses semi-automated tests whenever possible, as proposed in the commission implementing decision (EU) 2018/1524. The mapping between the criteria used in this method, the norm EN 301 549 and the requirements from (EU) 2018/1524 is available in Annexe 2: Mapping simplified method criteria / EN 301 549 / Principles and accessibility needs.

In this setup, only three pages are tested, and the following are recommended:

• The home page

• One page containing a form (for example the contact page)

• One page with special content, like a video, a table, a complex image.

The recommended tools to execute this method are the following:

• The tools recommended in the testing method of RGAA (see previous sections)

• Axe

• “nu HTML Checker” combined with the bookmarklet called “check for WCAG 2.0 parsing compliance” available on the about page of this service.

Apart from this, we have developed a tool called “simplA11yMonit” which performs checks from Axe and nu HTML Checker and prepares an evaluation grid for manual testing. But for a few exceptions (ex: RGAA criteria 2.1, 8.1), all automated tests are manually verified to avoid false positives. The evaluation grid is then read by another tool, “simplA11yGenReport”, which automatically generates a report from the evaluation grid. Moreover, in order to sensitize site owners to the accessibility of the document provided on the site, our third tool “simplA11yPDFCrawler” enables to crawl a subset of all available PDF documents on a given site, and their accessibility is evaluated with 6 simple automated checks (tagged document, protected document, scanned document, availability of a title, of bookmarks, of a default language). These automated checks do not constitute tests for a criterion per se but they give a global idea of the general accessibility of PDF documents on a given website.

No usability tests were involved in the simplified monitoring, only accessibility tests.

In this method, 50% of the criteria defined in RGAA are evaluated. They cover the overall requirements of websites to be perceivable, operable, understandable and robust, as referred to in Article 4 of Directive (EU) 2016/2102, and the user accessibility needs, as defined in (EU) 2018/1524, except for “usage without vocal capability” for the reason that the inventory contains no website requiring vocal interaction.

Figure 3: Simplified monitoring method, distribution of tests by requirements (see description below)

Figure 4: Simplified monitoring method, number of tests per user accessibility need (see description below)

2.3.3. Monitoring method for mobile apps

Luxembourg developed its reference framework for the accessibility evaluation of mobile applications. This framework is named RAAM 1 (“Référentiel d’évaluation de l’Accessibilité des Applications Mobiles“) and is heavily inspired by RGAA. It has been published under the Creative Commons Attribution licence, which enables its free reuse by other member states, organisations or companies.

As such, it is an operational framework for verifying compliance with accessibility requirements. Therefore, it enables to verify that a mobile application complies with the criteria described in European norm EN 301 549 v.3.2.1 for the following sections:

• 5 Generic requirements

• 6 ICT (Information and Communications Technology) with two-way voice communication

• 7 ICT with video capabilities

• 10 Non-web documents

• 11 Software

• 12 Documentation and support services

This framework contains 107 control criteria. The mapping between RAAM criteria and the norm EN 301 549 is described in Annexe 3: Mapping EN 301 549 – RAAM 1. Each control criterion is associated with a practical evaluation method for iOS and Android devices.

The (non-normative) test method included in RAAM recommends the following tools on top of the previously mentioned screen readers:

• A physical keyboard

• Accessibility Inspector for iOS

• Android Accessibility Suite

• Accessibility Scanner on Android

• Colour Contrast Analyser

• SCRCPY

No usability tests were involved in the monitoring of mobile apps, only accessibility tests.

3. Outcome of the monitoring

3.1. Detailed outcome

3.1.1. In-depth monitoring of websites

In this monitoring program, 15 websites have been controlled, with an average of 14 pages per website. The detailed results of the control are available in Annexe 4: Results of the in-depth monitoring of websites. Every in-depth audit has a compliance score. The corresponding scoring method is defined in RGAA as follows: it is obtained from the formula C/ (C+NC), where C is the number of compliant criteria on the sample of pages and NC is the number of non-compliant criteria on this sample. It is important to note that if a criterion is non-compliant on one page, then it is considered as well as non-compliant on all pages of the sample.

Figure 5: in-depth monitoring, global compliance score distribution (see description below)

The minimum score is 24% and the maximum is 62%, with an average of 44% (standard deviation: 11%). A website is considered as non-compliant if it scores under 50%, partially compliant if the result is between 50% and 100%, and fully compliant only at 100 %. We thus have here 9 non-compliant and 6 partially compliant websites.

In RGAA, criteria are also linked to the WCAG levels A and AA. Although the criteria from both levels are mandatory, the A-level criteria have more impact on the overall accessibility of a website, they should hence be remediated with a higher priority level.

Figure 6: in-depth monitoring, compliance score of A-level criteria. (see description below)

Figure 7: In-depth monitoring, compliance score of AA-level criteria. (see description below)

The compliance scores are globally better at the AA-level than at the A-level. A focus on solving A-level issues should be the priority of the public sector bodies for the next monitoring period.

Regarding non-compliance issues, 1410 of them have been found in total on the 15 websites. The following table presents their distribution among RGAA topics.

The three topics with the most non-compliance issues are:

• “Mandatory elements”, which encompass technical criteria like HTML code validity, presence of a default language, tagging of language changes when present, and avoiding HTML tags used only for presentation purposes.

• “Presentation of information” with criteria like keyboard focus visibility, capacity to zoom without issues, capacity to adapt text properties.

• “Forms” with all criteria related to forms like field labelling, error prevention and management.

The non-compliance can also be analysed at the criterion level.

This table shows that all criteria in this Top 10 except two are A-level criteria. The most common criterion is still related to text colour contrasts. Development teams together with content creators can solve most of these issues. To solve contrast issues in particular (ranks #1 and #5), the focus could be put during the next monitoring period on awareness activities targeting designers.

3.1.2. In-depth monitoring of applications

The objective set by the European Commission was to monitor two apps in 2021. This objective has been met during this period (cf. Annexe 5: Results of the in-depth monitoring of mobile applications). Both apps were non-compliant (47% and 41% of compliance). The number of apps being quite limited, it is impossible to provide as many statistics as in the previous section.

The two topics with the most non-compliant issues are:

• “Interactive components”: the encountered problems in this topic are mostly about keyboard accessibility and compatibility with assistive technologies (e.g.: no accessible name, non-explicit accessible name, no state available to assistive technologies).

• “Navigation”: the main problems were related to focus management and navigation order.

It is worthwhile noting that these two mobile applications do not support the landscape orientation, which seems to be a common and recurring issue.

3.1.3. Simplified monitoring of websites

In this monitoring program, 87 websites and 3 pages per website have been controlled. The detailed results of the control are available in Annexe 6: Results of the simplified monitoring of websites. In this regard, a scale of five compliance levels has been defined: very weak, weak, average, good, very good. They are based on the obtained score and represent evenly distributed percentage intervals.

Figure 8: simplified monitoring, global compliance score distribution

The average compliance score is 55% (standard deviation: 13%). 36% of all websites monitored here obtained a score under 50%.

Figure 9: simplified monitoring, average compliance score per administrative level (see description below)

The good results of State websites compared to other administrative levels mainly reside in the fact that most of them have been developed by CTIE (“Centre des Technologies de l’Information de l’État”, the Government IT centre), which has more than 15 years of experience in digital accessibility, has developed its own best practices guide (“Renow”) and has set up processes, which include accessibility checks in the quality control circuit.

Regarding non-compliance issues, a total number of 1121 issues have been found on these 87 websites. The following table presents their distribution among RGAA topics:

The most non-compliant topics encompass “forms” and “mandatory elements” like in the results of the in-depth monitoring. The non-compliance can also be analysed at the criterion level.

The two most non-compliant criteria are related to forms. This over-representation of forms issues may be linked to the fact that for all sites, the sample of 3 tested pages contained at least one page with a form. As for the in-depth monitoring, the majority of criteria in this top 10 are related to A-level criteria.

3.2. Use of the disproportionate burden derogation

The context of disproportionate burden derogation is defined in Article 5 of the Directive (EU) 2016/2102 and has been transposed in article 4 of the Luxembourgish law of 28 May 2019.

When an organisation invokes this derogation for some contents or features, this should be mentioned in their accessibility statement. At the moment of finalizing this report, SIP was officially notified of the publication of 115 accessibility statements of websites and 3 of mobile applications among 637 websites and apps in the inventory. 19% of all websites and mobile applications in Luxembourg have an accessibility statement. Among these 118 accessibility statements, 67% have invoked a disproportionate burden derogation.

Figure 10: Most frequent types of disproportionate burden derogations in accessibility statements (see description below)

4. Use of the enforcement procedure and end-user feedback

SIP is in charge of proposing an online contact form for all persons wishing to submit a complaint regarding a non-compliance of a public sector body with the accessibility requirements, according to Article 7(2) of the Luxembourgish law of 28 May 2019 transposing the Web Accessibility Directive.

This accessibility complaint form is available since 26 June 2020 on the website of SIP.

When a complaint is submitted via this form, SIP acts as a mediator between the citizen and the public sector body by following this process:

1. Analysis of the complaint: is the complaint about digital accessibility? Is the object of the complaint within the scope of the law? Is the encountered issue reproducible?

2. Search for a contact person at the public sector body.

3. Send an e-mail to the public sector body. This e-mail informs about the legal framework, explains the complaint, confirms that the issue has been verified and reproduced, and informs about the legal timeframe of 1 month to react.

4. Exchange with the Public Sector Body, for example about possible solutions to the encountered issue.

5. Inform the user of the conclusions, as soon as a solution has been proposed.

In cases where the reported accessibility problems are out of the scope of the law of 28 May 2019, SIP informs the responsible organisation and gives feedback to the complainer, even if there is no legal obligation that has to be respected.

The Ombudsman can also act as a mediator according to the law and follow a similar procedure. At the moment of finalizing this report, no information was received from the Ombudsman concerning eventual complaints related to accessibility issues.

SIP has published as Open Data the anonymised list of complaints it managed during the period 2020-2021.

During this period, in total 18 complaints have been managed:

• 17 complaints were about 13 websites or mobile applications within the scope of the law, which represent 2 % of the inventory. 1 complaint was not within the scope of the law.

• 6 complaints led to an accessible solution: the citizen was provided with a solution to his precise problem, which could be realised on a short-term basis. This could either be the concrete correction of the problem on the site or an alternative functionality ensuring an equal service level. In case no solution could be provided to the citizen because it was considered this solution would imply a disproportionate financial burden or amount of work, the complainer still received feedback and help, even if the latter cannot be qualified as a “solution”.

• 4 complaints were considered by the public sector body as a disproportionate burden without any short-term solution.

• 2 complaints did not get any answer from the public sector body.

The majority of complaints are about State online services.

The majority of complaints do not match the proposed list of sectors as defined in EU 2018/1524. The second most important sector regarding the number of complaints is the transport sector.

9 complaints have been submitted against 5 mobile apps. Inaccessible PDF forms are the second most frequent reason for complaint, with 5 complaints. It would be interesting here to address this point more globally by progressively replacing PDF forms with Web forms. A paper-based process was object of a complaint. Although this issue was not within the scope of the law, the competent public sector body has been informed.

5. Additional measures

5.1. Description of the mechanisms for consulting with relevant stakeholders on the accessibility of websites and mobile applications

The most important aspect in our approach to the stakeholders’ involvement was their consultation during the process to establish the test samples for the sites and applications to be included in the monitoring program. Hereafter is a summary of the process:

• SIP created (and continuously updates) inventories of existing public sector websites and applications, and published these documents on the national open data portal to ensure transparency and to induce feedback.

• The existence of these (draft and evolving) documents was announced via the common communication channels of the open data portal (article, Twitter).

• The feedbacks by the stakeholders’ community were taken into account and led to several modifications of the inventories.

• A first sample for the three monitoring programs was drafted based on these inventories taking into account the criteria specified by the implementing decision (EU) 2018/1524 and aiming at representing as well as possible all administrative sectors and levels. Other aspects taken into consideration while building the sample were criteria like statistics of use, importance level of the site/app for the general public or eventually planned updates.

• The monitoring sample was published on the open data portal as well and underwent scrutiny by the stakeholders’ community. In an interactive process, the sample lists were subsequently adapted, without omitting the basic principles (and exclusions) stated by the law and the related execution rules.

5.2. Procedures to make public any developments in accessibility policy relating to websites and mobile applications

Regarding Luxembourg’s awareness-raising and publicity efforts, the following can be mentioned:

• SIP’s accessibility staff is part of the interministerial committee on digitalisation, as well as of the interministerial working group on digital inclusion.

• In SIP’s team, a legal expert is available to help with the legal aspects and corresponding popularisation and communication efforts.

• Regularly scheduled training sessions and workshops are organised for the employees of the public sector, where also policy-related issues are communicated and discussed.

• A considerable communication effort essentially happens formally via the dedicated accessibility website and in a continuous and more informal way via the social networks. It is also considered to eventually organise awareness-raising events, dedicated to audiences beyond the borders of public sector employees.

5.3. Experiences and findings from the implementation of the rules on conformity with the accessibility requirements

Void.

5.4. Information on training and awareness-raising activities

5.4.1. Training

SIP has developed an offer of training courses available at the National Institute of Public Administration (INAP). These training courses target the employees of the entire public sector of Luxembourg.

Here is the list of training courses, as they actually exist:

1. Introduction to digital accessibility: taking disability into account in the digitalisation of services

2. Accessibility of Office and PDF documents

3. Digital accessibility at all stages of a project

Moreover, every public servant beginning a career at Luxembourg’s Government has to follow a mandatory training called “formation spéciale”. This training also contains a module about digital accessibility, which serves as a first introduction and advertises the other in-depth training offer.

During 2020-2021, 259 hours of training were given about digital accessibility. 106 persons followed the in-depth courses and 343 assisted to the 30 minutes introduction sessions.

5.4.2. Awareness actions

With regard to the communication channels, SIP’s accessibility team has set up a Twitter account in April 2020. In December 2021, this account had 267 followers. It has since proven to be the preferred way to communicate with the public on this topic. The message frequency is about one tweet per week.

Furthermore, a dedicated article on the Government’s website explains its commitment to digital accessibility in four languages.

A dedicated website, accessibilite.lu, has been set up to publish Luxembourg’s accessibility frameworks, general information about Luxembourg’s simplified evaluation method, as well as some tools like for example a form to help in writing an accessibility declaration. Between 1 August 2020 and 1 December 2021, this website registered 12691 page views, 7901 visits and 4091 unique visitors. Most of the visitors are coming from Luxembourg and France, which may be linked to the fact that the content is mainly written in French.

Another tool to be mentioned is the form for reporting complaints about digital accessibility issues, as required by Luxembourgish law.

In September 2021, a circular letter has been sent to all local governments (municipalities) and the communal syndicates in Luxembourg, informing them about the legal requirements concerning the digital accessibility of their websites.

Annexes

Annexe 1: Mapping RGAA 4.1 / EN 301 549 v2.1.2 / WCAG 2.1

Annexe 2: Mapping simplified method criteria / EN 301 549 / Principles and accessibility needs

Annexe 3: Mapping EN 301 549 – RAAM 1

Annexe 4: Results of the in-depth monitoring of websites

Annexe 5: Results of the in-depth monitoring of mobile applications

Annexe 6: Results of the simplified monitoring of websites

Annexe 7: Key figures of the monitoring period 2020-2021

Inventory

• 598 websites
• 95 mobile applications

Sample

• 15 in-depth audits
• 87 simplified audits
• 2 audits of mobile applications

Monitoring

• In-depth
  • Average compliance score: 44%
  • Minimum score: 24%
  • Maximum score: 62%
  • 60% of all websites conformity score under 50% (i.e. non-conform)
• Mobile
  • Compliance scores: 47% and 41%
• Simplified
  • Average compliance score: 55%
  • 36% of all websites conformity score under 50%

Accessibility statements

• 118 accessibility statements received by SIP
• 19% of all websites and mobile applications with an accessibility statement

Disproportionate burden derogation

• 67% of all accessibility statements invoked a disproportionate burden derogation

Enforcement procedure and end-user feedback

• 18 filed complaints, 17 within the scope of the law
• Complaints about 13 websites and apps (2% of the inventory)
• 6 complaints led to an accessible solution
• 4 complaints considered as a disproportionate burden

Training courses

• 25 sessions
• 449 participants
• In-depth courses: 106 participants
• Introduction sessions: 343 participants