Towards Automated eGovernment Monitoring

Paper H - Global web accessibility analysis of national government portals and ministry web sites.

Author: Morten Goodwin

This chapter was originally published in the Journal of Information Technology & Politics . Please see the original source for the complete paper.1

  

Original paper authors: Morten Goodwin, Deniz Susar, Annika Nietzio Mikael Snaprud, and Christian S. Jensen

Morten Goodwin and Mikael Snaprud is with Tingtun AS, Kirkekleiva 1, 4790 Lillesand, Norway, morten.goodwin@tingtun.no,mikael.snaprud@tingtun.no.

Deniz Susar is with United Nations Department of Economic and Social Affairs susar@un.org

Annika Nietzio is with Forschungsinstitut Technologie und Behinderung (FTB) der Evangelischen Stiftung Volmarstein, Grundschötteler Str. 40 58300 Wetter (Ruhr), Germany. egovmon@ftb-net.de

Christian S. Jensen is with Aarhus University (AU) csj@cs.au.dk

Abstract

Equal access to public information and services for all is an essential part of the United Nations (UN) Declaration of Human Rights. Today, the Web plays an important role in providing information and services to citizens. Unfortunately, many government Web sites are poorly designed and have accessibility barriers that prevent people with disabilities from using them. This article combines current Web accessibility benchmarking methodologies with a sound strategy for comparing Web accessibility among countries and continents. Furthermore, the article presents the first global analysis of the Web accessibility of 192 United Nation Member States made publically available. The article also identifies common properties of Member States that have accessible and inaccessible Web sites and shows that implementing antidisability discrimination laws is highly beneficial for the accessibility of Web sites, while signing the UN Rights and Dignity of Persons with Disabilities has had no such effect yet. The article demonstrates that, despite the commonly held assumption to the contrary, mature, high-quality Web sites are more accessible than lower quality ones. Moreover, Web accessibility conformance claims by Web site owners are generally exaggerated.

Keywords: Antidisability discrimination, BenchmarkingWeb Sites, e-government, e-participation, Web accessibility

Introduction

Web accessibility barriers are problems that prevent people with special needs or disabilities from using Web sites. For example, a poorly designed Web site may contain images without alternative textual descriptions. Information in these images will be lost to anyone who cannot see them, such as visually impaired users relying on screen reader technology. As more and more government services are moved online, the accessibility of government Web sites plays an increasingly important role. Inaccessible Web sites can prevent a visually impaired person from paying his or her taxes online; a Web site that requires the use of a mouse can prevent a physically impaired person from applying for social benefits. Web sites without such obstacles are barrier-free and accessible, while Web sites containing barriers that prevent some people from using them are inaccessible.

Several laws and agreements signify that people with disabilities should not be hindered from using government Web sites or portals. However, no global analysis of the accessibility of government Web sites exists. Thus, to date, no empirical evidence exists to show which efforts have had a practical, positive influence on the accessibility of public Web sites.

This article presents the first publicly available results from a global Web accessibility analysis of government Web sites from all UN Member States.2 It also provides an analysis of common properties of Member States having accessible and inaccessible Web sites —providing strong indications of what influences Web accessibility and which policies work well, as well as an analysis of how Web site quality relates to accessibility. This article also extends existing Web accessibility evaluation methodologies with sound methods for comparing Web accessibility on the national and international levels.

Equal access to public information and public services for all is a universal human right [1]. In particular, one trend in recent years has been to make public services available on the Internet [2]. As nations promote e-government and make government services available online, persons with disabilities stand to benefit greatly. For the first time in history, people who used to be dependent on help from others have the opportunity to use public services without assistance.

Unfortunately, many public Web sites are designed without accessibility in mind. This omission creates significant barriers for people with special needs, and it prevents some people from gaining access to both the information

available online and government services on the Web. Often, simple adjustments to Web sites can remove such barriers and thus increase the quality of life for disabled citizens.

Studies have shown a correlation between Web usability and Web accessibility [3], as well as between accessibility and user satisfaction [4]. But the accessibility of a Web site does not by itself say anything about its services or how useful its content is to citizens. For example, from a theoretical point of view, it is possible for a site to be almost content-free but still score well in most accessibility tests. Similarly, it is possible to have a well-developed Web site that is full of accessibility barriers, making it inaccessible to persons with disabilities. In fact, it is a common belief that making Web sites accessible is equivalent to providing simplified Web pages [5] and that accessible Web sites are less mature and are generally of lower quality than inaccessible Web sites. Many Web site owners also claim that their Web sites are accessible, by explicitly adding accessibility logos on the sites. It is assumed in the literature that these logos are not exaggerated [6], but little empirical data are available to support this assumption.

Web accessibility measurements have been carried out on local, national, and international levels. Even though most of the evaluations are based on the Web Content Accessibility Guidelines (WCAG) [7],[8], they are mostly one off studies using specialized methodologies. To deal with this problem, the development of a Unified Web Evaluation Methodology (UWEM) [9],[10] was initiated by the European Commission.

This article is organized as follows. First, we review the current state of Web accessibility and assessments of e-government. The Methodology section presents a brief description of the methodology used for carrying out the article's comparative analysis. Furthermore, the Web Accessibility Measurement section extends UWEM to include a sound Web accessibility comparison on the ministerial, national, and continental levels, as well as a description of the methodology used for measuring Web accessibility of national portals and ministerial Web sites. The Results section presents a ranking of the 192 UN Member States3 based on the first publicly available global evaluation of Web accessibility of national portals and ministerial Web sites. The Findings and Comparison section (including, for the first time, empirical evidence) analyzes a set of common hypotheses in the Web accessibility literature. Examples include Member States with antidisability discrimination laws have more accessible Web sites, Web sites that conform to accessibility standards are less mature, and Web sites that claim conformance to accessibility standards using accessibility logos are more accessible.

Current State

This section outlines the current state of Web accessibility including large-scale Web quality assessment and political initiatives.

Equal Rights for All Citizens

The right to public information and services is stated in the Universal Declaration of Human Rights [11]:

Naturally, the Universal Declaration of Human Rights, signed in 1948, long before the invention of the Web, does not explicitly address Web accessibility. However, it addresses the notion of everyone having equal rights to public information and services. Thus, whenever public services are made available through inaccessible Web sites, some groups are prevented from accessing the public information and services, which is in violation of the Declaration of Human Rights.

More recently, the UN Convention on the Rights of Persons with Disabilities [12],[13] presented measures for promoting access for all, including the following:

The convention introduces Web accessibility as a right for persons with disabilities and strongly encourages governments to make their Web sites accessible to all.

The ambition to make Web sites barrier-free is not limited to conventions and laws, but has been on the agendas of governments and public agencies for some time. For example, the European Commission introduced e-inclusion as part of the Lisbon Strategy in 2000 [14], which states that governments must "Ensure that every citizen should have the appropriate skills needed to live and work in a new Information Society for all." The European Union also has, as part of its i2010 strategy, a goal of making all public Web sites barrier-free by the year 2010 [15].

United Nations Global E-Government Survey

Every second year the UN conducts an e-government survey (previously e-readiness) [16],[17] that aims to indicate which governments are progressive pioneers in relation to e-government technology.4 The resulting publication receives substantial attention, both as a representation of how information and communication technology (ICT) is used for e-government, and as a way of acknowledging the Member States that perform well and thus promote good practices.

The survey has been in operation since 2002, and the most recent e-government report was published in 2010. In this article, all references to the UN e-government survey refer to 2010, unless otherwise stated. This includes references to the UN e-government benchmarking methodology [18], as well as the Web site URLs5 to the national and ministerial Web sites. It should be noted that all information related to the UN e-government survey, including the URLs for the evaluated Web sites, is available through the UNDESA [17],[16],[18].

The UN survey is based on a questionnaire that assigns a binary value to each indicator based on the presence or absence of specific online services. These values are further combined into an e-government index that ranges from 0 to 1. Thus, a Member State offering developed e-government services receives a high e-government score, whereas a Member State offering few government services online receives a low score. In 2010, the Republic of Korea (South Korea), the United States of America, and Canada received the highest scores, while San Marino, Somalia, and Tuvalu had "most improvement potential" scores.

The e-government survey assesses the same questions on the same or similar sites in each country to ensure consistency. The primary site is the national portal or the official government home page of the UN Member State. The remaining sites are the five ministries of Health, Education, Social Welfare, Labor, and Finance, which are representative of the public services required the most by citizens. Each ministerial site is assessed according to the same set of questions.

The questions from the survey are divided into four sections:

  1. Enhanced presence, intended to give an indication of how present a Member State is on the Internet. An example of a question is whether the state's Web sites have been updated within the last three months.
  2. Interactive presence, indicating the online services' degree of usability and interactivity. For example, it poses a question regarding whether the Web site meets the Web content accessibility guidelines level WAI-A. (See the Web Content Accessibility Guidelines section for details on accessibility levels.)
  3. Transactional presence, indicating the Web site's service delivery level. An example of a question is whether there is any online tracking system for typical government requests like permits.
  4. Networked presence, indicating the government's communication level. For example, the question is posed as to whether or not statements encouraging citizen participation are present.

The UN Member States are ranked according to their e-government score. This ranking can be seen as both an acknowledgement to the Member States focusing on e-government and a motivation for countries having improvement potential.

Web Content Accessibility Guidelines

 

The Web Content Accessibility Guidelines (WCAG) developed by the World Wide Web Consortium (W3C) constitute the de facto standard for Web accessibility. Version 1.0 (WCAG 1.0) [7] was superseded by version 2.0 [8] at the end of 2008. The intention is that Web sites conforming to the guidelines are accessible and thus can be used by all people, including those with special needs and disabilities.

The WCAG are categorized into 14 main guidelines that are further organized into 65 checkpoints. The main guidelines cover issues such as textual alternatives of visual content and clear navigation. Furthermore, each of the 65 checkpoints is assigned as priority level 1, 2, or 3 based on its impact on Web accessibility.

The W3C Web Accessibility Initiative (WAI) presents three possible conformance claims for the WCAG 1.0 at the Web site level depending on which checkpoints are satisfied:

All checkpoints, main guidelines, and priority levels can be found on the W3C [7]. Any Web site owner claiming conformance to the above checkpoints can express this by placing a logo on its Web site. These WAI logos represent accessibility conformance claims, but are not verified by any third party [6].

It should be noted that the WCAG are guidelines for creating Web sites and do not by themselves provide the tools for measuring accessibility. Despite this, measurements according to the WCAG are carried out on global and national levels, as well as on local government Web sites. One consequence of this approach is that the measurement methodologies differ in subtle, but important ways, such as which pages within a Web site may be sampled and how an accessibility score of a Web site is calculated. This leads to difficulties in comparing accessibility results from different surveys (for example comparing two countries' Web accessibility levels).

Unified Web Evaluation Methodology To deal with the problem of having many different methodologies for measuring Web accessibility, a Unified Web Evaluation Methodology (UWEM) was commissioned by the European Commission and developed by 24 partners in the Web Accessibility Benchmarking Cluster [9]. The UWEM is based on the WCAG 1.0 and intends to provide a unified and repeatable methodology that is independent of implementation and software. It also takes into account that certain accessibility tests are more efficient when applied automatically, while others require human judgement. In total, the UWEM includes 144 accessibility tests (of which 26 can be run automatically). The UWEM tests are all designed in such a way that all elements that have accessibility problems are counted as barriers.

Essentially, the UWEM can be used in four ways [19]:

  1. Conformance claim: The intention is to verify that a Web site is accessible. A Web site owner can claim that a Web site is accessible by passing all the UWEM tests. This is a claim similar to having a W3C WAI logo on a Web site.
  2. Evaluation of a single Web site: Evaluating a single Web site is less strict than conformance claims. The evaluation can be carried out manually, automatically, or as a combination of both. Depending on the procedure, either a manual sample of representative Web pages or an automatic, uniform set of Web pages is selected. After an evaluation has been carried out, a report with potential barriers within the Web site can be presented. Based on this type of report, Web developers and content providers can identify and fix the accessibility barriers on their Web site.
  3. Evaluation of a single Web page: Evaluating an entire Web site is time consuming and cannot be done directly on demand even when performed automatically. However, barriers within Web sites are often replicated across several pages [20]. Because of this, it is helpful for Web developers to assess the barriers on single Web pages. Several online tools for assessing Web accessibility on Web pages have emerged. These services evaluate a single Web page and return a list of barriers [21],[22].
  4. Monitoring and comparison of Web accessibility: The UWEM also includes large scale monitoring of Web sites. This approach allows Web accessibility evaluation of many Web sites and thus introduces the possibility of determining which Web sites are more and less accessible than others. Similarly, the Web accessibility of geographical regions such as countries and continents can be compared using the Web sites within the respective regions. Evaluating large numbers of Web sites is time consuming and can therefore only be carried out automatically. The complete approach is presented in the Web Accessibility Measurement section.

Measuring progress of eAccessibility in Europe (MeAC)

 

The "Assessment of the Status of eAccessibility in Europe" [23] and a follow up report [24] surveys the e-accessibility of the European Union (EU) Member States, Australia, Canada, and the USA. The studies include a survey of e-accessibility policies in the evaluated countries. Furthermore, the studies include measurements based on key indicators and results from questionnaires sent to stakeholder groups.

MeAC covers accessibility evaluations of both hardware (such as televisions and telephones) and software (such as public and private Web sites). The Web accessibility evaluation was carried out as a combination of automatic and manual evaluations to test for WCAG 1.0 WAI-A conformance. In total, 314 Web sites were evaluated.

The surveys show that there is in fact an increase in the number of failure results (Web accessibility barriers), and a decrease in the number of passing results, from 2007 to 2008. MeAC plans to provide annual surveys in the future.

Other global Web accessibility evaluations

Some smaller, international, one-off surveys on Web accessibility exist.

In 2009, Kuzma, Yen, and Oestreicher carried out an accessibility evaluation on a global scale using Tests de Accessibilidad Web (TAW) [25]. Results are presented as barriers on WAI conformance levels A, AA, and AAA. The comparative study also considers whether a country has strong or weak Web accessibility regulations, and whether the country has signed the UN Rights and Dignity of Persons with Disabilities Convention and Optional Protocol [1]. Furthermore, the study presents the number of errors (barriers) as a measurement. Unfortunately, this survey was limited to only 12 countries, and thus cannot be considered globally representative.

Additionally, Chodrie, Ghinea, and Weerakkody conducted a global evaluation of five countries assessing Web accessibility, HTML validation, and broken links using the tools WebXact and Vischeck [26]. Their survey presents the accessibility results as the number of errors for the WCAG 1.0 WAI-A, WAI-AA, and WAI-AAA.

The above two surveys use the number of errors as a measurement, which has a significant bias because the number of barriers is directly related to the Web page size (number of bytes). Thus, the results of Kuzma et al. [25] and Choudrie et al. [26] are biased, and the conclusions based on these results are consequently unsound. Details on why the number of failed tests is significantly biased are presented in the Web Site section.

Methodology

This article uses both quantitative and qualitative research methods. Measurements of Web accessibility of public Web sites of all UN Member States are carried out, as well as a comparative study of the accessibility results among the Member States. The aim is to identify properties of Member States with accessible and inaccessible Web sites in order to identify policies that work well. Such properties include the existence of antidiscrimination laws, the signing of the UN Rights and Dignity of Persons with Disabilities Convention, the high-quality Web sites, and accessibility claims on public Web sites. The methodology for measuring Web accessibility is addressed in the next section, Web Accessibility Measurement.

Web Accessibility Measurement

This article aims to measure the accessibility of public Web sites of all UN Member States. There exists no directly applicable benchmarking methodology, but the UWEM is close to being applicable [9],[27],[19]. This article also extends the UWEM to support a sound aggregation on the ministerial, national, and continental levels.

Each step of the measurement, from a single test and evaluation of a Web site to aggregating the results on UN Member State levels, is covered here. Each place where the methodology in this paper differs from the UWEM is clearly mentioned, and details of the differences are provided.

An overview of the formal notations is provided in Table 1.

Formal DefinitionDescription
Tests
$T$All tests
$t$A single test with the possible result of fail-pass
Web Page
$p$Web page
$N_p$Number of tests for page $p$
$B_p$Number of fail test results (barriers) for page $p$
Web Site
$s$Web Site
$N_s$Number of tests for site $s$
$B_s$Number of fail test results (barriers) for site $s$
$F(s)$Web Site Accessibility Score (lower score is better)
Ministry
$M$Ministry Web sites.
$N_M$Number of tests for a ministry $M$
$B_M$Number of fail test results (barriers) for a ministry $M$
$F(M)$Ministry Accessibility Score (lower score is better)
UN Member State
$c$UN Member State.
$N_c$Number of tests for a UN Member state $c$
$B_c$Number of fail test results (barriers) for a
UN Member state $c$
$F(c)$UN Member State Accessibility Score (lower score is better).
For presentation purposes, $F(c)$ can further be discretized
into the letters A,B,C,D, and E, where A is the best score,
B second best,and so on. The cut-off values are defined as
following [27]:
A: $F(c)=0%$
B: $0%
C: $12.5%
D: $25%
E: $50%
Continent
$C$Continent of UN member states
$N_C$Number of tests for a continent $C$
$B_C$Number of fail test results (barriers) for a continent $C$
$F(C)$Continent Accessibility Score (lower score is better)
Table 1: Formal definitions
 

Tests

The tests detect single Web accessibility barriers, such as an image without alternative text. In total, 23 automated tests are applied, of which 21 tests address (X)HTML,6 and two tests address CSS.6 These tests are all defined in the UWEM.

For each Web page $p$, each test $t$ is applied to each element to which it is applicable. Most tests are defined on the element level, an example of which is a test detecting whether an image has an alternative description. Images without alternative text cause challenges for, for example, people who rely on screen readers. Thus, if a test finds an image without an alternative text, a potential barrier exists, and the test reports a failing result; otherwise, the test reports a passing result. Formally, this test detects the presence of the alt attribute in the <img /> tag. A Web page may contain many images (many <img /> tags).

Other tests are defined on the Web page level, for instance the test for a valid document type. For each Web page, a test can only detect whether it has a valid document type or whether no valid document type is present.

The list of all tests applied can be found in the Appendix H.1 — Accessibility Tests. The way that tests are aggregated into a Web site accessibility score is described in the Web page and Web site sections. Note that there is no weighting of the tests. This is in contrast to some research aiming at finding the severity of barriers [28],[29],[30]. However, most current research does not weigh barriers based on severity. In our approach, any feature of a Web page that is likely to cause accessibility problems for users with any kind of disability is counted as a barrier. It is a subject of great controversy to determine severity weights for accessibility barriers, because the weights could be interpreted as an importance ranking of the disabilities themselves. For instance, are barriers for blind users more severe than barriers affecting people with motor impairments?

Web Page

 

A Web page is a resource identifiable by a single URL, each typically consisting of multiple elements such as images, headings, and text paragraphs.

The number of times a test $t$ is applied on a page $p$ is denoted as $N_{pt}$, while the total number of applied tests for all types of tests is denoted as $N_p$.

Similarly, the number of barriers detected by a given test $t$ for page $p$ is denoted as $B_{pt}$. The total number of barriers found on page $p$ independent of tests is denoted as $B_p$.

Formally,:

For example, if a Web page consists of only ten images, and two of these are lacking alternative textual descriptions, the number of tests is ten ($N_p = 10$) and the number of barriers is two ($B_p = 2$).

Web Site

 

A Web site is a collection of Web pages that naturally belong together, such as Web pages from one organization. The Web site definition in the UWEM and the definition from the UN e-government report are not compatible. The UWEM defines a Web site as a set of Web pages, typically identified by a domain name such as www.example.com. In contrast, the UN e-government report defines a Web site as an official URL where a ministerial or national Web site can be reached. In the UN e-government report, a tester will use the URL as an entry point. Answers for any survey question should be attainable from this URL.7

This means that certain Web sites will be identified by a domain name, just as in the UWEM. However, for other Member States, several ministries share the same domain name. In these situations, the domain name will itself not be representative of an individual ministry.

An example of a Web site compatible with both the UWEM and the UN e-government report is the United States Ministry of Finance (Department of the Treasury) available at: http://www.treasury.gov. All URLs starting with http://www.treasury.gov are part of the Department of the Treasury.

An example of a Web site from the UN e-government report that is not compatible with the UWEM is the Web site of the Swedish ministries, which are all available under the domain name http://www.regeringen.se. The Ministry of Finance (Finansdepartementet) is available at http://www.regeringen.se/sb/d/1468. Similarly, the Ministry of Education (Utbildningsdepartementet) is available at http://www.regeringen.se/sb/d/1454. To include the entire domain http://www.regeringen.se would be incorrect and lead to biased results. All ministries that are a part of the Swedish government would be included, while only five selected ministries are to be part of the e-government report.

Since this article uses the same Web sites as the UN e-government report, the sites are defined the same way—as a single URL.

The score for an individual Web site is calculated as the fraction of the number of detected barriers over the number of applied tests. The number of tests applied ($N_s$) for a Web site $s$ is the sum of all tests applied for every evaluated Web page within the site. Similarly, the number of barriers detected ($B_s$) for Web site $s$ is the sum of all detected barriers for every evaluated Web page within the site.

Formally:

Figure 1: (a) Correlation between the size in bytes of the Web pages and number of barriers detected. (b) Percentage of Web accessibility barriers in member states versus percentage of population with access to the Internet.(c) Percentage of Web accessibility barriers in member states versus GNI per Capita. (d) Percentage of Web accessibility barriers versus the ICT Development index.
 
Figure 2: (a) Percentage of Web accessibility barriers in member states versus UN eGovernment score. (b) Percentage of Web accessibility barriers in member states versus eParticiaption score. (c) Percentage of Web accessibility barriers in member states versus infrastructure. (d) Percentage of Web accessibility barriers in member states versus Human Capital Component index. (e) Percentage of Web accessibility barriers in member states versus Online Services.
 
Figure 3: (a) Percentage of Web accessibility barriers in member states with developing and emerging economies. (b) Percentage of Web accessibility barriers versus member states with and without anti-disability discrimination Laws. (c) Percentage of Web accessibility barriers versus member states who have signed and not signed the United Nations Rights and Dignity of Persons with Disabilities Convention
(d) Percentage of Web accessibility barriers versus detected WAI conformance claims. 

Note that other studies use only the number of barriers ($B_p$) as a measure of accessibility [25],[26]. However, the number of tests and the number of barriers are directly dependent on the Web page size (number of bytes). For example, concerning our data, the Pearson correlation coefficient between Web page size and number of barriers detected is 0.86.8 A graphical representation of this can be seen in Figure 1(a). The Web page size should not by itself have such a discriminatory effect on the results, since the size of the Web page alone does not necessarily say anything about the accessibility of the page.

In contrast, the Web site accessibility score ($F(s)$) has no such correlation (Pearson correlation coefficient of only 0.038). Further statistical justification for the score is offered by Nietzio et al. [27].

Ministry

For this study, ministries with an online presence are identified by their Web sites. The sites considered in this study are those in the 2010 UN e-government survey, which includes the national Web sites for the Member States as well as five ministries: Education, Labor, Social Affairs, Health, and Finance. For most Member States, there are six Web sites, one national site, and one site for each of the ministries.

In the UN e-government survey, each Web site is either a ministerial or a national site, although for certain Member States, not all ministries exist or have independent Web sites, while in other states, ministries have several sites. Finally, not all ministries of the UN Member States fit into the five ministries defined by the UN e-government survey. For example, several Member States have their Ministry of Labor and Ministry of Social Affairs combined into one ministry.

The UWEM is not designed for aggregating national accessibility scores based on ministry and national Web sites. Thus, the UWEM's simple average of the Web sites for a country is not sufficient, as a simple arithmetic average would not weigh all ministries equally. Using the UWEM's simple arithmetic average means that adding Web sites to a ministry may significantly change the accessibility score—even if the Web sites added are simple mock ups. It also means that ministries that have one inaccessible, but otherwise mature, Web site and several simple and accessible Web sites will receive an undeservedly good score.

To address the above variations, we have chosen to weight each ministry equally, independent of the number of Web sites the ministry has. Thus, the number of ministerial Web sites does not have any positive or negative impact on the country accessibility score. Because of this, the ministry accessibility score F(M) is defined as the percentage of barriers over applied tests, which is independent of the number of Web sites evaluated.

UN Member state

In this study, UN Member States with an online presence are identified by their national and ministerial Web sites. Using the same approach for Member States as for ministries, the Member State accessibility score $F(c)$ is defined as the percentage of barriers over the applied tests. Note that not all ministries in all countries have Web sites. If a ministry does not have a Web site, this ministry is not included in the score calculations. This is formally noted in the following manner:

This is formally noted in the following manner:

For presentation purposes, the UN Member accessibility state score can further be discretized into the letters A—E as outlined in Table 1.

Continent

Continents are the UN's macro-geographical regions.9 The continent accessibility score is also calculated as the percentage of barriers over the applied tests. This means there is no explicit weighting for the size of a Member State (population or similar). All Member States—for example the United Kingdom and Monaco—contribute equally to the European continent score. In contrast, pages with a larger number of tests applied implicitly contribute more.

Formally, the continent score is calculated in the following manner:

R.Country$F(c)$#R.Country$F(c)$#
1Germany1.28%533Saint Kitts and Nevis18.26%6
2Portugal2.43%634Bulgaria18.61%5
3Spain2.59%635Switzerland18.75%6
4Netherlands3.6%636Denmark18.97%6
5Palau4.06%437Dominican Republic20.0%4
6Hungary4.85%638Ethiopia20.18%4
7Equatorial Guinea5.51%139Slovenia20.18%5
8Czech Republic5.85%640Libyan Arab Jamahiriya20.42%4
9Italy6.16%641Congo20.63%2
10Japan6.73%642Mozambique20.88%5
11Australia8.55%743Bangladesh22.14%6
12Belgium9.95%544Nauru22.22%1
13Colombia10.03%445Sweden22.22%7
14Guinea11.11%146Burundi22.22%1
15Togo11.11%147UK and Northern Ireland22.22%5
16Iceland11.11%548South Africa22.27%7
17Cape Verde12.23%549Turkey22.92%5
18France13.29%650Antigua and Barbuda23.21%6
19Canada14.81%551Republic of Korea24.14%5
20Vanuatu14.89%152Ecuador24.56%6
21Ireland15.03%653Latvia24.76%5
22USA15.22%554Azerbaijan24.78%5
23Estonia15.44%755Philippines24.96%6
24Brazil15.51%656Guatemala25.0%3
25Ukraine15.55%357Tajikistan25.0%4
26Grenada15.61%558Gabon25.23%2
27New Zealand15.77%659Malaysia25.32%6
28Slovakia15.96%660Poland25.48%6
29Viet Nam16.28%461Mongolia25.93%5
30Norway16.67%762Malta26.48%5
31Lao16.67%263Micronesia26.49%3
32Central African Republic18.0%164Croatia26.61%6
Table 3(1-64) UN Member States ranked (R.) with Member State score ($F(c)$) and Number of Web Sites (#).
 
R.Country$F(c)$#R.Country$F(c)$#
65Greece26.61%697Bosnia and Herzegovina31.65%4
66Kiribati26.63%298Mali31.88%4
67Finland26.98%699United Arab Emirates31.95%5
68Mexico27.02%5100Venezuela32.46%6
69Tonga27.2%4101Kuwait32.53%4
70Austria27.45%7102Cambodia32.64%4
71Lithuania27.62%5103Chad32.91%2
72Syrian Arab Republic27.66%3104Sierra Leone32.97%3
73Romania27.68%5105Iraq33.04%4
74Liberia28.01%2106Haiti33.33%1
75Botswana28.02%4107Belize33.33%4
76Yemen28.16%1108Dominica33.33%1
77Monaco28.18%2109Trinidad and Tobago34.55%6
78Luxembourg28.21%5110Bahamas34.62%5
79Uzbekistan28.38%7111El Salvador34.87%4
80Armenia28.4%5112Bolivia35.13%5
81Panama28.83%6113Belarus35.69%6
82Chile29.13%5114Barbados35.73%6
83Swaziland29.27%5115Sao Tome and Principe36.06%5
84Seychelles29.3%5116Pakistan36.17%4
85Benin29.32%2117Peru36.18%5
86Mauritania29.4%4118Kazakhstan36.21%5
87Uruguay30.0%3119Guyana36.61%5
88Maldives30.08%6120Myanmar37.03%7
89Bhutan30.15%5121Russian Federation37.04%6
90Jordan30.16%5122Samoa37.2%5
91Burkina Faso30.75%5123Côte d'Ivoire37.26%5
92Algeria30.77%5124Republic of Moldova37.65%6
93Saint Vincent and the Grenadines31.19%5125Papua New Guinea37.8%4
94Tuvalu31.43%2126Cyprus37.86%5
95Madagascar31.51%4127Andorra37.93%4
96Kenya31.64%6128Morocco38.33%7
Table 4: (65-128) UN Member States ranked (R.) with Member State score ($F(c)$) and Number of Web Sites (#).
 
R.Country$F(c)$#R.Country$F(c)$#
129Singapore38.45%6161Rwanda46.63%5
130Nepal38.62%5162Cuba46.75%4
131Cameroon38.93%3163Georgia46.96%4
132Nigeria39.65%2164Timor-Leste47.25%6
133Malawi39.78%7165Zambia47.84%1
134Qatar39.8%6166Somalia48.97%5
135Israel40.11%7167Lebanon49.32%7
136Gambia40.14%5168Suriname50.0%1
137Nicaragua40.16%5169Honduras50.08%5
138Costa Rica40.93%7170China50.48%5
139Zimbabwe41.03%5171Turkmenistan50.88%2
140Thailand41.31%6172Uganda51.56%4
141Kyrgyzstan41.53%3173San Marino51.67%6
142Solomon Islands41.61%3174Oman52.4%9
143Eritrea41.67%1175Fiji53.44%6
144Egypt41.81%6176Sudan54.17%5
145Argentina42.06%6177Serbia54.54%5
146Jamaica42.31%4178India54.57%7
147Afghanistan42.34%6179Sri Lanka54.61%6
148Tunisia42.62%9180Angola56.25%4
149Guinea-Bissau42.86%1181Niger56.31%1
150Namibia43.75%6182Ghana56.52%5
151Bahrain44.03%9183Mauritius57.45%10
152Comoros44.04%1184Saudi Arabia57.87%5
153The former Yugoslav Republic of Macedonia44.3%5185Liechtenstein58.1%3
154Iran 45.29%6186Marshall Islands59.42%1
155Democratic Republic of the Congo45.33%3187Djibouti59.43%4
156Montenegro45.39%3188Brunei Darussalam59.81%6
157Paraguay45.43%6189Saint Lucia60.65%6
158Albania45.74%6190Lesotho66.17%5
159Indonesia45.98%6191United Republic of Tanzania66.75%6
160Senegal46.62%5-Democratic People's Republic of Korea-0
Table 5: (128-191) UN Member States ranked (R.) with Member State score ($F(c)$) and Number of Web Sites (#).
 

Results

 

Member state ranking list

Tables 3, 4 and 5 rank the member states according to their country Web accessibility score ($F(c)$) as described in the Web Accessibility Measurement section.

The only UN Member State missing is the Democratic People's Republic of Korea (North Korea) because it has no online presence—no real national or ministerial Web sites exist.

It should be noted that the Web site accessibility score does not by itself say anything about usefulness of the content of the Web sites. For example, the Palau Web sites are more accessible than most other Web sites in this survey. However, these Web sites are also simpler than most. Thus, we cannot in any way suggest that the Palau Web sites are more useful or better to its citizens than, for example, the Web sites of the United States of America; we can only conclude that they contain lower percentages of barriers, which will cause fewer problems for people with special needs.

The Member State ranking is also presented as an intensity map in Figure 4. Each UN Member State is colored according to the Web site accessibility score in the corresponding country, ranging from close to 0% barriers detected (light color) to more than 60% of the tests detecting barriers (dark color).10

Continent Ranking List

Certain strategies for dealing with Web accessibility have been devised, which hold the potential for influencing Web accessibility on the continent level. One example of this is the European Union i2010 goal of making all public Web sites barrier free within the year 2010 [15] and the commissioning of the UWEM [9]. Similar goals regarding the other continents do, to the best of our knowledge, not exist. A comparison on the continent level shows what impact the European Union i2010 goal has had on Web accessibility in Europe and if it has had positive effects that are not present in the Web accessibility of the other continents.

Figure 5 indicates that there is a difference in accessibility of the Web sites found in the different continents. This section presents the barriers per continent. The accessibility score F(C) is calculated as described in the Web Accessibility Measurement section. Figure 5 shows that Europe has the most accessible Web sites, with 24.9% of the tests detecting barriers. Following this are Oceania and America, with 32% ${}-$ 35% of the tests detecting barriers. Asia and Africa come in last with 39% ${}-$ 42% of the tests detecting barriers.

It is worth noticing that 24.9% of the tests detecting barriers is a significant amount of barriers. This means that, on average, about one fourth of all checked elements represent a potential barrier. This suggests that many users are prevented from accessing public services. Even though Europe has better accessibility scores than the other continents, it is also an undeniable fact that the goal of making all public European Web sites accessible within 2010 has failed.

Figure 4: Intensity Map Representing the Web Accessibility of UN member states. The colors range from close to 0% barrier detected (light color) to more than 60% of tests detected barriers (dark color).
 

Findings

 

Most common Web accessibility barriers

This section presents the most common Web accessibility barriers found in the survey. The UWEM tests are designed so that all accessibility problems are counted as barriers. It should be noted that the UWEM has not been designed explicitly to present statistics at the barrier level, but rather at the Web page and Web site levels. Different tests have different properties. For instance, there is only one document type (doctype) per Web page, and it can only be valid or invalid. Thus, this test can at the most provide a failing result once per Web page. In contrast, a Web page may have several images that can have missing alternative textual descriptions. Consequently, this type of barrier may occur several times within a Web page. Further details on test applicability can be found in the UWEM [9].

This leads us to three different ways of presenting

the most common barriers:

  1. Percentage of test detecting barriers: The number of barriers divided by the number of tests applied. This is derived directly from the UWEM score and is intended as a representation of a Web site's accessibility. It is, however, not intended to be presented for individual tests, as it does not address the comparison between tests.
  2. Average barriers on each page: The number of barriers divided by the number of pages. This approach does not distinguish well between barriers that can only occur once per page and barriers that may occur many times. For example, images without alternative textual description would be over represented compared to missing document types.
  3. Percentage of Web pages with barrier: The number of Web pages in which the barrier occurs at least once. A disadvantage of this is, however, that multiple barriers detected from the same test are not taken into account. There is noway to distinguish between a page with only one image and a page with five images lacking alternative texts.

Table 2 shows the most common barriers detected in the evaluated UN Member States. To provide completeness, all three numbers are presented. In addition, the table also shows also how the tests are related to the WCAG 1.0 [7] and WCAG 2.0 [8]. It should be noted that the UWEM is based on the WCAG 1.0. Therefore, all tests can be mapped to the WCAG 1.0 checkpoints. In contrast, the mappings from the UWEM IDs to the WCAG 2.0 only contain information on how the existing UWEM IDs match the WCAG 2.0 guidelines. The degree to which the WCAG 2.0 can be automated has not been considered, since automating the WCAG 2.0 testing is beyond the scope of this article.

UWEMShort Description(a)(b)(c)WCAG 1.0WCAG 2.0
11.1.HTML.01Latest W3C technology not used92.9%5.392.9%11.1-
3.2.HTML.02Invalid (X)HTML52.0%166.776.2%3.24.1.1
11.2.HTML.02Deprecated elements45.1%283.3100%11.2-
1.1.HTML.01Non-text without text alternative36.3%98.255.2%1.11.1.1
13.1.HTML.01Links with the same title but different target33.2%56.921.7%13.12.4.9
3.2.HTML.01Invalid Doctype33.1%1.937.3%3.24.1.1
3.2.CSS.01Invalid CSS32.6%37.644.9%3.24.1.1
11.2.HTML.01Deprecated (X)HTML attributes29.2%85.1100%11.2-
6.4.HTML.01Mouse required17.1%26.273.1%6.42.1.1, 2.1.3
12.4.HTML.01Form without legend16.5%3.662.4%12.44.1.2
1.1.HTML.06Non-text without embed12.7%1.417.2%1.11.1.1
12.1.HTML.01Frames without description9.1%3.995.3%12.14.1.2
12.4.HTML.02Form without label8.7%2.269.5%12.44.1.2
7.3.HTML.01Marquee element used6.6%0.69.0%7.32.2.2
7.5.HTML.01Page redirect5.6%1.35.6%7.52.2.1, 3.2.5
3.5.HTML.03Levels skipped in heading1.9%0.714.3%3.52.4.10
3.6.HTML.03Number list simulated0.7%0.11.5%3.61.3.1
12.3.HTML.01Fieldset without description0.4%0.543.8%12.32.4.10
7.2.CSS.02Blink element used in CSS0.4%$\sim$0.00.4%7.22.2.2
7.2.HTML.01Blink element used in HTML0.3%0.10.6%7.22.2.2
7.4.HTML.01Refresh used$\sim$0.0%$\sim$0.0$\sim$0.0%7.42.2.1, 2.2.4
12.3.HTML.03Optgroup without legends$\sim$0.0%$\sim$0.0$\sim$0.0%12.32.4.10
9.1.HTML.01Server side image maps$\sim$0.0%$\sim$0.0$\sim$0.0%9.1-
Table 2: Most common barriers represented with the UWEM IDs. The tests are ranked according to (a) Percentage of Web pages with at least one barrier, (b) Average barriers on each page and (c) Percentage of fail test results, as well as how the tests relate to the WCAG 1.0 and the WCAG 2.0 Checkpoints
 

Figure 5 presents the most common barriers (percentage of Web pages with barriers) grouped by continent. It is evident that the trends between continents are similar. However, there are some noticeable differences. For instance, images without alternative text (1.1.HTML.01) comprise a much more common barrier in Africa and Asia when compared to America, Europe, and Oceania. A possible explanation for this might be that missing alt attributes are often used as a typical example of Web accessibility and have been discussed at length in Europe and America [31].

Figure 5 also shows that Africa, Asia, and Oceania are using more outdated technologies than Europe and America. The barriers 11.2.HTML.02 (deprecated elements) and 11.2.HTML.01 (deprecated attributes) are much more common in Africa, Asia, and Oceania when compared to Europe and America. In contrast, there are significantly fewer European Web sites using the latest W3C technologies (11.1.HTML.01) when compared to the other continents, while invalid use of these technologies is less represented. This indicates that while Africa, America, Asia, and Oceania declare that they use the most recent W3C technologies more often than Europe, they use these technologies incorrectly.

Figure 5: The most common Web accessibility barriers per continent as well as the continent accessibility score $F(C)$.
 

Comparisons

 

Many hypotheses and assumptions regarding Web accessibility have been made in the literature. Unfortunately, very few of these are properly supported by empirical evidence. Similarly, there exist several common intuitions regarding Web accessibility where no empirical evidence has been presented in literature.

The comparisons are graphically represented with box-plots in Figures 1, 2, and 3. In these plots, the median is represented by a wide black line, the box is drawn between the quartiles (from the 25th percentile to the 75th percentile), and the whiskers (dashed line) extend to the minimum and maximum values (not including outliers). For presentation purposes, the country accessibility score ($F(c)$) is presented with the letters A—E as explained in Table 1.

Hypothesis (1): Developed member states have more accessible Web sites then member states with developing and emerging economies.

 

It is commonly believed that when resources are sparse, such as during a financial crisis, accessibility for all is prioritized less than other commitments [32]. We would therefore expect Member States with developing and emerging economies, which often have limited resources, to focus less on accessibility compared to wealthier Member States. Based on this expectation, we assume this will be reflected in the accessibility of the public Web sites. Thus, we can expect that the wealthy Member States have better Web site scores than Member States with developing and emerging economies.11

Figure 3(a presents a box-plot of the Web accessibility score ($F(c)$) in developed and developing countries. The figure supports this hypothesis, and it shows that developed countries have on average a significantly lower Web accessibility score. On average, a developed country has 18.8% of the tests detecting barriers on its Web sites, while a developing country has on average 33.3% of the tests detecting barriers. The Pearson correlation coefficient of this data is 0.38. The only outlier is Liechtenstein: While even though being a developed country, it

has many accessibility barriers on its Web site.

Hypothesis (1) conclusion: Confirmed.

Hypothesis (2): The wealthier (GNI per capita) a country is, the less barriers will be present on its Web sites.

The cost of making a Web site accessible is addressed in the literature. On the one hand, it is clear that having a Web site accessible for all users is cost effective [33]. On the other hand, fixing usability problems after a Web site is launched is up to 100 times more expensive than addressing the problems in the design and development phases [34].

Based on this fact, we would expect to see a correlation between wealth of a UN Member State and the Web accessibility scores of its public Web sites. This is also in line with the obvious intuition that wealthy countries can spend more on Web accessibility when compared to poor countries and thus develop more accessible Web sites. A commonly used measure of country wealth is gross national income (GNI) per capita.

The accessibility intensity map in Figure 4 reveals the difference in the accessibility of the Web sites of wealthy and poor Member States. The figure shows that both North America and Western Europe have more accessible Web sites because these regions are drawn in lighter colors. In comparison, the figure also shows that Africa, Middle America, and parts of the Middle East have less accessible Web sites, as these regions are darker.

Figure 1(c) shows that there is a correlation between the GNI per capita and Web accessibility ($F(c)$). The Pearson correlation coefficient between these data sets is ${}-$ 0.42, which indicates that there is a correlation between country wealth and Web site accessibility.12

Based on this data, we can confirm that the hypothesis is true and that richer Member States have fewer barriers on their Web sites.

The GNI per capita data is extracted from the World Bank national accounts data [35] and OECD National Accounts data files 2008 [36],[37]. Note that the UN Member States and World Bank Countries do not match completely. Therefore, only the 186 countries that were in both data sets are compared.

Hypothesis (2) conclusion: Confirmed.

Hypothesis (3): The higher the percentage of the population who have access to the Internet in a member state, the less barriers there are on its Web sites.

The literature shows that technology can be used to fuel social inclusion [38],[39]. Furthermore, technology such as public Web sites can be used to include disabled people into society. Thus, in Member States where most people have access to the Internet, it is important that the government Web sites are promoting social inclusion rather than preventing people with disabilities from participating. From this assertion, we can derive the hypothesis that Member States in which more people have access to the Internet have better Web site accessibility scores.

Figure Figure 1(b) shows a correlation between the UN Member State Web accessibility score and percentage of people with access to the Internet. Based on this, we can confirm that the hypothesis is correct and that the higher the percentage of the population with access to the Internet in a country, the more accessible its Web sites tend to be. The data13 has a Pearson correlation coefficient of ${}-$ 0.43. Note that, in line with the natural intuition, there is a very strong correlation (0.83) between GNI per capita and percentage of Internet users.

The number of Internet users is taken from the World Bank national accounts data. It originates from the International Telecommunication Union, World Telecommunication Development Report and database, and World Bank estimates. Note that the United Nation Member States and World Bank Countries do not match completely,

so only the 186 countries that were in both data sets are compared.

Hypothesis (3) conclusion: Confirmed.

Hypothesis (4): High eGovernment quality means low accessibility.

Even though we are starting to see mature, accessible Web sites, it is often understood that making Web pages accessible is equivalent to providing overly simplified versions of the content [5]. The UN e-government evaluation creates benchmarks for the maturity of Web sites [16],[17]. If it is the case that accessible Web sites are overly simplified, this should be seen when measuring the correlation between Member States' accessibility scores and e-government scores. Thus, countries that have a high UN e-government score (mature e-government) would be expected to have a high accessibility score (many accessibility barriers).

However, Figure 2(a) shows that the opposite is evident. The Pearson correlation coefficient between the e-government score14 and Web accessibility of UN Member States is $-0.36$.

It should be noted that checking for conformance to the WCAG 1.0 level WAI-A is already part of the UN e-government evaluation. Because the UWEM includes tests for WAI-A and WAI-AA conformance of the WCAG 1.0, there are overlapping tests between the e-government survey and the UWEM scores. However, Web accessibility is only one of 40 questions in the UN e-government survey. The correlation is stronger than what would be expected from this factor alone.

More insights on how the quality of e-government is related to accessibility can be gained by examining the e-government index in more detail. The UN e-government index is a combination of three components: telecommunication infrastructure, human capital, and online service. The correlations between the Web accessibility score and the three individual metrics are graphically represented in Figures 2(c-e), and they have Pearson correlations of $-0.39$, $-0.24$, and $-0.32$, respectively.

The correlation between the accessibility score and the telecommunication infrastructure index ($-0.39$), which represents the citizen access to the Internet, surprisingly is not similar to the correlation between the accessibility score and the percentage of population with access to the Internet ($-0.43$). The Pearson correlation coefficient between the telecommunication infrastructure index and the percentage of population with access to the Internet is 0.95.

The human capital component index captures a combination of literacy and education opportunities in the Member States. The Pearson correlation coefficient between the human capital component index and the Web accessibility score is only $-0.24$. Regarding this factor, we can conclude that literacy and education opportunities have less impact on Web accessibility than the other areas of the UN e-government score.

The Online Service index aims to measure the sophistication of the online government (for example Web sites). There is a stronger correlation between theWeb accessibility score and the online service index ($-0.32$) than between the Web accessibility score and the human capital component index ($-0.24$). This strongly suggests that sophisticated government Web sites are more accessible than government Web sites that are not that advanced.

Finally, the UN Global e-government Survey 2010 also includes e-parti- cipation, which targets the quality and usefulness of online government services and information. This indicator is strongly correlated with the Web accessibility results, having a Pearson correlation coefficient of $-0.40$. A graphical representation of the correlation can be seen in Figure 2(b). Again, this strongly suggests that Web accessibility goes hand-in-hand with the usefulness and quality of services and information: E-government Web sites with useful content and high quality are more likely to be accessible than e-government Web sites without useful content or high quality.

Hypothesis (4) conclusion: Disproved.

Hypothesis (5): High ICT quality produces accessible Web sites.

Since mature Web sites (Web sites with high UN e-government score) have a better accessibility score, we may expect to see similar correlations between Web accessibility and other e-government and ICT quality measurements.

A well known ICT quality measurement tool is the UN International Telecommunication Union (ITU) digital opportunity index (DOI [40]. The DOI combines 11 indicators to represent ICT capabilities, affordability, and quality of the Member States. In 2008, the DOI was superseded by the ICT Development Index (IDI). The IDI extends the DOI as well as incorporates a digital access index15 and ICT opportunity index [41].

Figure 1(d) shows that the Web accessibility score of the UN Member States and IDI are highly correlated. The Pearson correlation coefficient between Web accessibility score and IDI is $-0.53$. Thus, Member States with high ICT quality have more accessible Web sites.

Hypothesis (5) conclusion: Confirmed.

Hypothesis (6): Countries with anti-disability discrimination laws have more accessible Web sites.

Some Member States explicitly focus on reducing discrimination against people with disabilities by introducing national laws on antidisability discrimination or disability discrimination acts [42],[43],[44]. In other countries, no such explicit focus exists.

A natural hypothesis from this is that countries with such focus on antidiscrimination are, in general, more accessible in most levels of society—including when it comes to online public services such as official governmentWeb sites. Although this has been expressed in the literature [25], it has not been examined without real, unbiased empirical evidence.

By comparing the countries with explicit antidisability discrimination laws, we obtain a high Pearson correlation coefficient of 0.48. A graphical representation can also be seen in Figure 3(b). The figure shows that Member States with explicit accessibility laws have on average a score of 16.0%, while countries without explicit laws have in average an accessibility score of 35.7%. Seventy-five percent of the Member States with explicit laws have a country accessibility score lower than 26.5%. In contrast, 75% of the countries without explicit laws have a score larger than 26.5%. It is interesting to note that the data indicate that explicit accessibility laws have a stronger influence on the Web accessibility of a country than its wealth (developed/developing or GNI per capita).

Hypothesis (6) conclusion: Confirmed.

Hypothesis (7): Countries that have signed the United Nations Rights and Dignity of Persons with Disabilities Convention have more accessible Web sites.

As in the case of existence of national antidiscrimination legislation, it would be expected that countries that have signed the UN Rights and Dignity of Persons with Disabilities Convention and the Optional Protocol [1] would have more accessible Web sites.

However, in contrast to national antidisability discrimination legislation, there is no high level of correlation between Member States that have signed the convention and those that have not: the correlation coefficient is only 0.10. A graphical representation can be seen in Figure 3(c).

This is consistent with findings in the literature [25]. It should be noted that the UN convention is relatively new, and it may take several more years before the effects on Web accessibility can be detected by measurements.

Hypothesis (7) conclusion: Disproved.

Hypothesis (8): Web sites with Web Accessibility Initiative logos have better Web site accessibility scores.

Web site owners and content providers can claim that their sites are accessible by putting WAI logos on them. The logos represent the priority level of the WCAG that a Web site has met. A Web site with a WAI-A logo should conform to the WCAG priority level 1, while a Web site with logo WAI-AA should conform to both Priorities 1 and 2 of the WCAG, and so on. Details on the conformance claims and priority levels are presented in the Web Content Accessibility Guidelines section.

There is no verification—by WAI or other independent organizations—that a Web site is actually meeting the conformance claim of the WAI logos. This means that the Web site users need to trust the Web site owners regarding the assertions that the accessibility of the Web site corresponds to the logo's claim. The claims are often exaggerated and disputed [6],[45]. Because of this dispute, it is interesting to see whether the presence of WAI logos actually coincides with a better Web site accessibility score.

Figure 3(d) presents the Web site accessibility score versus WAI conformance claim logos and shows that there is a trend that Web sites with the WAI logos have fewer barriers. Furthermore, Web sites with the WAI-AA conformance claim have lower percentages of barriers than Web sites with only the WAI-A conformance claim. Note that only two Web sites with WAI-AAA conformance logo were detected. The tests carried out in this study all correspond to Levels 1 and 2 of the WCAG 1.0 checkpoints. This means that if the WAI conformance claims were correct, the Web sites with the WAI-AA and WAI-AAA logos would have 0 barriers. However, Figure 3(d) shows that this is far from being the case.

From this finding, we see, in line with the literature, that the accessibility claims indicated by the logos are highly exaggerated [6]. However, there is a greater chance that a Web site is accessible if a WAI logo is present. In addition to the three WAI conformance logos, we also detected the presence of the "Valid(X)HTML" logo, because valid HTML is also on of the accessibility criteria defined in the UWEM. However, as can be seen in Figure 3(d), there is no correlation between the Web site accessibility score and the "Valid (X)HTML" logo. It is worth mentioning that 35% of the Web pages that had a "Valid (X)HTML" logo in fact did not have valid HTML.

Hypothesis (8) conclusion: Confirmed.

Conclusion and further research

This article presents the first global Web accessibility overview of national government portals and ministry Web sites from all UN Member States. The evaluation methodology is based on the Unified Web Evaluation Methodology, but has also been extended to enable a sound representation of Web accessibility on the ministerial, national, and continental levels. The Web accessibility score of each Member State is calculated as a percentage of accessibility barriers found among all the tests applied.

Web accessibility barriers exist for all evaluated Member States. This causes significant problems for people with special needs, including people with disabilities. The Member States with the fewest accessibility barriers are Germany, Portugal, Spain, Netherlands, and the simple official Web sites in Palau.

The most commonly detected Web accessibility barriers are incorrect use of the HTML standard, including the uses of deprecated HTML and missing alternative descriptions for images. Such barriers cause significant problems for people with disabilities, for example blind or dyslexic users who rely on assistive technologies (such as screen readers) to use public Web sites. Furthermore, the results indicate that Web sites in Africa, America, Asia, and Oceania state use the most recent Web technologies more often than Web sites in Europe, but they use these technologies incorrectly.

The results also show that there is a great difference between the accessibility of the evaluated Web sites. Public Web sites in Western Europe and North America are more likely to have fewer accessibility barriers on their Web sites. The analysis further shows that accessible Web sites are more common in countries with developed economies, high GNI per capita and where a large percentage of the population has access to the Internet. However, even though there is a strong correlation between the economy of a country and the accessibility of its Web sites, exceptions to this rule were found.

Furthermore, the results show that, despite popular assumptions, accessible Web sites are not in any way less mature or have lesser quality than inaccessible Web sites.

The analysis shows that there are correlations between the obtained accessibility scores and all the addressed quality measures. Most significantly, there is a strong correlation between the UN e-government index, which aims to measure the quality of the e-government services, and the accessibility score. Thus, e-government Web sites with high quality and useful content are more likely to be accessible than Web sites without useful content or high quality. Web sites that claim to be accessible using WAI logos generally have fewer accessibility barriers than those where no such logo is present. However, the claims are exaggerated and do not represent the actual accessibility of the Web sites.

This article shows that significant work remains to be done to make governmental Web sites accessible, but all hope is not lost. Even though the financial status dictates the accessibility of national portal and ministerial Web sites, some countries are able to attain good accessibility scores even with limited resources.

The results further indicate that there is a substantial correlation between antidisability discrimination laws and the reached accessibility scores. This strongly suggests that introducing antidisability discrimination laws and policies has a significant positive effect on Web accessibility in a country. In contrast, there are no differences between Member States that have or have not signed the UN Rights and Dignity of Persons with Disabilities Convention.

Future research includes examining in more detail the properties of the results' outliers. This may answer questions such as the following: Why is it that Palau has accessible Web sites even though it is a country with a developing and emerging economy and without any antidisability discrimination laws?

The authors plan to extend the accessibility testing beyond the UWEM. This allows for more elaborate test results at the Web page and Web site levels. This is helpful for developers and content providers who want to improve the accessibility of their Web sites. Such tests include, for example, checking for the validity of alternative texts in addition to merely detecting their presence. Furthermore, the tests and methodology are planned to be updated to the WCAG 2.0.

Finally, the authors plan to extend the measurement methodology and implementation and to include other important e-government areas such as transparency.

Acknowledgements

The eGovMon project (http://www.egovmon.no) is co-funded by the Research Council of Norway through the VERDIKT Program. Project no.: Verdikt183392/S10.

C. S. Jensen is an Adjunct Professor at University of Agder.

Much of the work presented this paper has been done during an internship at the United Nations Department of Economic and Social Affairs (UNDESA). This work would not have been possible without the colleagues at UNDESA.

The statements, findings, interpretations, and conclusions expressed in this publication and in all contents herein are those of the authors and do not necessarily reflect the view of, or are endorsements from the United Nations. The United Nations is not responsible for and hereby expressly disclaims any and all liability for any and all damages or loss resulting from the use of any results, or third party Web sites accessed via links from this publication.

Author Note

Morten Goodwin

Tingtun AS

Morten Goodwin is currently involved in Tingtun AS and as a Ph.D. student at Aalborg University in the field of eGovernment benchmarking. His field of expertise is automatic assessment of eGoverment services. He is also involved in the project entitle "eGoverment Monitoring" where his main focus is applying learning algorithms and Web-mining techniques on eGovernment indicators which are difficulty to measure automatically. He has previously worked as a scientific developer in the European Internet Accessibility Observatory, including contributing to the Unified Web Evaluation Methodology. He has been a member of the expert group working on the United Nations eGovernment Survey and has done an internship at the United Nations Department of Public Administration working with eGovernment assessment.

Deniz Susar

United Nations Department of Economic and Social Affairs

Deniz Susar is a staff member at the United Nations Department of Economic and Social Affairs (UNDESA). He is currently serving as the Coordinator of the UNPAN Management Unit (UMU) at the Division for Public Administration and Development Management (DPADM). He holds a Master's Degree in International Political Economy and Development from Fordham University, New York, USA and a Computer Engineering degree from the Bosphorus University of Istanbul, Turkey. He has worked in several areas in both the private and public sector: he assumed senior project manager positions in Information Communication Technology (ICT) companies serving the social and private sector in Dublin, Ireland, and he was a Senior Consultant and Project Manager for various technology companies in Istanbul, Turkey.

Annika Nietzio

Forschungsinstitut Technologie und Behinderung (FTB) der Evangelischen Stiftung Volmarstein

Annika Nietzio holds a degree in mathematics from the University of Bochum. She joined FTB in 2005. Since then she has been involved in several eAccessibility projects. As a research scientist on the EIAO project, she worked on test procedures for automated evaluation of Web accessibility. In this context she also contributed to the Unified Web Evaluation Methodology (UWEM) developed by the WAB Cluster. Recently she has extended her research interest to eGovernment and is currently working on the eGovMon project, focusing on the accessibility challenges of eGovernment applications.

Mikael Snaprud

Tingtun AS \& University of Agder

Mikael Snaprud, Dr. Techn, has a specialization in Human-Computer Interaction, knowledge-based systems, and Control Engineering from Vienna University of Technology, Austria. Currently, Dr. Snaprud is the CEO of Tingtun AS, and he has a position as Associate Professor of Computer Science at the University of Agder in Grimstad, Norway. In both capacities he is active in ICT research, and project management in the field of eGovernment and eInclusion. He co-ordinates the eGovMon project, co-funded by the Research Council of Norway, and the eGovMoNet thematic network, supported by the European Commission. He initiated and co-ordinated the EU-funded EIAO project which delivered a demonstrator for a large-scale Internet accessibility observatory and participated in the preparations of the Unified Web Evaluation Methodology (UWEM).

Christian S. Jensen

Aalborg University

Christian S. Jensen, Ph.D., Dr.Techn., is a professor of computer science at Aarhus University and is an Adjunct Professor at University of Agder, Norway. His research concerns data management and spans issues of semantics, modeling, and performance. He is an IEEE Fellow and a member of the Danish Academy of Technical Sciences and the Royal Danish Academy of Sciences and letters. He is vice president of ACM SIGMOD. He is an editor-in-chief of The VLDB Journal, and has served on the editorial boards of ACM TODS, IEEE TKDE, and the IEEE Data Engineering Bulletin. He was PC chair or co-chair for STDM 1999, SSTD 2001, EDBT 2002, VLDB 2005, MobiDE 2006, MDM 2007, TIME 2008, and DMSN 2008.

Please address correspondence to Morten Goodwin, Tingtun AS, Kirkekleiva 1, N-4790 Lillesand, Norway. E-mail: morten.goodwin@tingtun.no.

Appendix H.1 — Accessibility Tests

 

This section presents the list of applied tests. These are all part of the fully automatable tests of the UWEM. A full description of the tests, including implementation details, can be found in Web Accessibility Benchmarking Cluster (2007).\nocite{UWEM12}

This section extends the UWEM descriptions by adding an explanation of why a test with a failing result can represent a barrier. It should be noted that barriers are highly individual. What may be perceived as a barrier for one person may not be a barrier for another, even if they have similar disabilities [46]. Whether a barrier is encountered or not depends on the expertise of the user, the assistive technology available, and whether the user is familiar with the Web site. It should be noted that our examples are in no way intended to represent all possible barriers that can be encountered by failing the tests—additional scenarios/barriers exist.

Missing alternative textual descriptions

Using graphical elements without textual alternative is a very common barrier. The most typical example is using images without alternative text (no alt or longdesc attribute present).

Tests part of this barrier:

Why this may be a barrier

This causes problems for people who are visually impaired who are unable to see the images. Any information conveyed in the image is lost to these users whenever the textual alternative is missing.

Use of invalid or outdated W3C technologies

The most common barrier is invalid or deprecated (X)HTML and/or CSS. The recent versions of (X)HTML and CSS have some built in accessibility features, which makes it easier for assistive technologies to successfully present the content. By not using the latest version, or using it wrongly, the accessibility features are not utilized and there is no guarantee that the Web page will be presented equally in different Web browsers.

Tests part of this barrier:

Why this may be a barrier

A typical example is the use of the font tag which is an outdated HTML element. The font tag provides design information such as font type, font size, and similar features. Such information is now part of CSS. Users with customized CSS, such as users who want to increase the font size, may encounter problems if the font element is part of the (X)HTML.

Furthermore, some user agents might not be able to present the content of the document as it is intended.

Non Descriptive links

Having links with the same text and title but with different targets is a common barrier. Quite often, links do not describe the target pages well.

Tests part of this barrier:

Why this may be a barrier

A typical example is the link text "read more", which will only make sense when used in context. For fast and efficient navigation, some accessibility tools present all links within a Web page to the user in a list. However, if all links have the same text "read more", such a list will become useless for navigation. More descriptive links such as "read more about the Nobel Peace Prize" will solve this problem.

Mouse required

Another typical barrier is Web pages implemented in such a way that it requires the use of mouse.

Tests part of this barrier:

Why this may be a barrier

For people with motor impairments, or people using devices without a mouse such as mobile phones, this causes a severe challenge. One example is a Web site implemented with menu items that can only be accessed by clicking with a mouse. People with motor impairments are often not able to use such Web sites at all.

Blinking or moving content

Web pages are implemented in such a way that they cause elements to move or blink.

Tests part of this barrier:

Why this may be a barrier

Blinking content causes problems for slow readers as well as people with photosensitive epilepsy.

Wrong use of structural elements

Structural elements are used to increase the navigability similar to chapters in a book. Not using these properly can lead to confusion regarding the structure of a page.

Tests part of this barrier:

Why this may be a barrier

If headings are skipped or otherwise used wrongly, it becomes more challenging to navigate a Web page. This is especially true for people who have the Web page read out loud, such as visually impaired users.

Missing labels or legends in form elements

Another common barrier is form elements without labels.

Tests part of this barrier:

Why this may be a barrier

An example is not correctly marking a search button as "search". The fact that a Web site is searchable is often understood by the context of the search field, such as showing a magnifying glass nearby. However, people using screen readers are unable to see this magnifying glass. If a button is not clearly marked as a search button, there is no way of knowing that it is intended for searching the Web site.

Refresh and redirection

Automatic refreshing may disorient users and can disrupt a browser's history of visited pages.

Tests part of this barrier:

Why this may be a barrier

Refreshing and redirecting can be a challenge for people with limited reading abilities who have problems concentrating.

Numbered list simulated

Ordered lists help non-visual users navigate. Non-visual users may get lost in lists not formally marked as such.

Tests part of this barrier:

Why this may be a barrier

For people who use screen readers ordered lists are of great navigational assistance. When such lists are not formally marked as such, these users may get lost in the navigational process.

Footnotes

1. [This paper is published in the Journal of Information Technology and Politics 8(1): 41 - 67 (2011). Taylor & Francis Group, LLC© 2011]

2. [The methodology is based on the UWEM with focus on automatic monitoring and comparing Web accessibility. The tests are carried out using the eGovMon tool. The eGovMon tool is Open Source and can be downloaded from http://www.egovmon.no. All results presented in this paper can be replicated using the eGovMon tool. Results on single Web pages can also be replicated on the Web interface http://accessibility.egovmon.no. All results and the URLs to the Web sites can be found at: http://mortengoodwin.net/GlobalWebAccessibility. Note that the Web sites subject the benchmarking are likely to have been updated since the evaluation was carried out which influences the results.]

3. [A list of all 192 UN member states can be found at http://www.un.org/en/members/index.shtml]

4. [E-government and eGovernment are equivalent terms used in the literature. For readability purposes, this article consistently uses the term e-government.]

5. [A Uniform Resource Locator (URL) is an address which identifies a Web page such as http://www.example.com/index.html]

6. [(X)HTML is a common abbreviation for both Extensible Hyper Text Markup Language (XHTML) and Hyper Text Markup Language (HTML). These technologies are most often used for Web pages.]

7. [Attainable is in the UN e-Government report defined as available within two clicks from the home page. This does not mean that testers are limited to two clicks, but that the testers are confident that they have found what they are looking for within two clicks, even though they may have to click further.]

8. [A Pearson correlation (Pearson product-moment correlation coefficient) is a standard statistical measure of correlations between two data sets. Independent data sets will have a Pearson correlation close to 0, while two data sets with a very strong correlation will have a measure close to 1 or -1. What determines a strong correlation depends on properties of the data sets. However, in general, any correlation less that -0.5 or larger than 0.5 indicates a large correlation between the data sets. Any measure between 0.5 and 0.3 (and between -0.5 and -0.3) is considered a medium correlation, while anything larger than 0.1 and smaller -0.1 is considered significant (yet small) correlation. It should be noted that correlation can be used as evidence of a possible relationship between the data sets, but cannot by itself indicate what the relationship is.]

9. [The definitions of the continents can be found at: http://unstats.un.org/unsd/methods/m49/m49regin.htm]

10. [The map has been generated using Google Fusion Tables and is not intended to be geographically accurate. In fact, it has deliberately been modified in order to reduce space.]

11. [In this paper all 135 member states part of G77, but not considering the group of 24, are defined as having developing and emerging economies. More information can be found at: http://www.g77.org/]

12. [The data are reciprocal: The more accessible Web site, the lower percentage of barriers (best accessibility score 0%). In contrast, a wealthy country will have a higher GNI per capita then a poor country.

13. [The data are reciprocal: The more accessible Web site, the lower percentage of barriers (best accessibility score 0%). In contrast, the more percentage of people who have access to Internet the higher the number.

14. [The data are reciprocal: The more accessible Web site, the lower percentage of barriers (best accessibility score 0%). In contrast, the best eGovernment score is 1.

15. [The digital access index focuses on to what extent individuals in a country are able to access and use ICT. Accessibility is not part of this indicator.]

16. [A doctype is a document declaration specifying which (X)HTML type used in the document.]

Bibliography

[1] Convention and Optional Protocol Signatures and Ratifications,United Nations Rights and Dignity of Persons with Disabilities,Retrieved November 4th, 2009, from http://www.un.org/disabilities/countries.asp?navid=17&pid=166

[2] 'E-government': is it the next big public sector trend?,Gauld, R.,International Handbook of Public Management Reform,105,2009

[3] The relationship between accessibility and usability of websites,Petrie, Helen and Kheir, Omar,397—406,2007,CHI '07: Proceedings of the SIGCHI conference on Human factors in computing systems,978-1-59593-593-9,http://doi.acm.org/10.1145/1240624.1240688,New York, NY, USA,San Jose, California, USA

[4] Measuring usability: are effectiveness, efficiency, and satisfaction really correlated?,,345—352,2000,CHI '00: Proceedings of the SIGCHI conference on Human factors in computing systems,1-58113-216-6,http://doi.acm.org/10.1145/332040.332455,New York, NY, USA,The Hague, The Netherlands

[5] Modeling Web Accessibility for Rich Document Production,Lopes, R. and Carrico, L.,Journal of Access Services,237—260,2009,6,1

[6] Sex, lies and Web accessibility: the use of accessibility logos and statements on e-commerce and financial Web sites,Petrie, H. and Badani, A. and Bhalla, A.,23—25,2005,Proceedings of Accessible Design in the Digital World Conference

[7] Web Content Accessibility Guidelines 1.0. W3C Recommendation 5 May 1999,World Wide Web Consortium,Retrieved November 4th, 2009, from http://www.w3.org/TR/WCAG10/

[8] Web Content Accessibility Guidelines (WCAG) 2.0,World Wide Web Consortium,Retrieved November 4th, 2009, from http://www.w3.org/TR/REC-WCAG20-20081211/

[9] Web Accessibility Benchmarking Cluster,2007,Retrieved November 4th, 2009, from http://www.wabcluster.org/uwem1_2/

[10] A Unified Web Evaluation Methodology Using WCAG,Velleman, E. and Strobbe, C. and Koch, J. and Velasco, C.A. and Snaprud, M.,Lecture Notes in Computer Science,177,2007,4556

[11] The Universal Declaration of Human Rights: origins, drafting, and intent,Morsink, J.,2000

[12] Convention on the Rights of Persons with Disabilities,United Nations,Retrieved November 4th, 2009, from http://www.un.org/disabilities/convention/conventionfull.shtml

[13] The Accessibility Imperative,Global Initiative for Inclusive Information and Communication Technologies,Retrieved December 18th, 2009, from http://g3ict.com/download/p/fileId_774/productId_118

[14] eInclusion: Participation for all in the Knowledge-based Society,Silver Economy Network of European Regions,2000,Retrieved November 4th, 2009, from http://www.silvereconomy-europe.org/EC_policies/einclusion_en.htm

[15] i2010 eGovernment Action Plan: Accelerating eGovernment in Europe for the Benefit of All,European Commission,Retrieved March 18th, 2010, from http://ec.europa.eu/information_society/activities/egovernment/docs/highlights/comm_pdf_com_2006_0173_f_en_acte.pdf

[16] Global E-Government Survey 2008, From E-Government to Connected Governance,United Nations Department of Economic and Social Affairs,Retrieved March 8th, 2010, from http://www2.unpan.org/egovkb/global_reports/08report.htm,December

[17] Global E-Government Survey 2010, Leveraging E-government at a Time of Financial and Economic Crisis,United Nations Department of Economic and Social Affairs,Retrieved May 11th, 2010, from http://www2.unpan.org/egovkb/global_reports/10report.htm,February

[18] Report of the Expert Group Meeting E-Government Survey:Getting to the Next Level,United Nations Department of Economic and Social Affairs,2009,Retrieved November 4th, 2009, from http://unpan1.un.org/intradoc/groups/public/documents/un/unpan035535.pdf

[19] The Unified Web Evaluation Methodology (UWEM) 1.2 for WCAG 1.0,Annika Nietzio and Christophe Strobbe and Eric Velleman,394-401,2008,ICCHP

[20] Research on Web accessibility in higher education,Thompson, T. and Burgstahler, S. and Comden, D.,Information technology and disabilities,2003,9,2

[21] Benchmarking and improving the quality of Norwegian municipality web sites,Morten Goodwin Olsen and Annika Nietzio and Mikael Snaprud and Frank Fardal,5th Int'l Workshop on Automated Specification and Verification of Web Systems,June

[22] eGovernment: New chance or new barrier people with disabilities?,Annika Nietzio and Morten Goodwin Olsen and Mikael Snaprud and Rudolph Brynn,8th international EGOV conference,September

[23] Kevin Cullen and Lutz Kubitschke and Ingo Meyer,Retrieved March 19th, 2010, from http://ec.europa.eu/information_society/activities/einclusion/library/studies/meac_study/index_en.htm

[24] Lutz Kubitschke and Ingo Meyer Feliccital Lull and Marten Haesner and Kevin Cullen and Shari McDaid and Ciaran Dolphin and Luke Grossey and Brian Grover and Chris Bowden and Carol Pollit and Siobonne Brewster and Tilia Boussios,Retrieved November 4th, 2009, from http://ec.europa.eu/information_society/activities/einclusion/docs/meac_study/meac_follow-up_2008.pdf

[25] Global e-government Web Accessibility: An Empirical Examination of EU, Asian and African Sites,Kuzma, J.M. and Yen, D. and Oestreicher, K.,The Second International Conference on Information and Communication Technology and Accessibility, Hammamet, Tunisia,May

[26] Evaluating global e-government sites: a view using web diagnostic tools,Choudrie, J. and Ghinea, G. and Weerakkody, V.,Electronic Journal of E-government,105—114,2004,2,2

[27] Unified Web Evaluation Methodology Indicator Refinement,,2007,Retrieved November 4th, 2009, from http://www.eiao.net/resources

[28] Measuring web accessibility by estimating severity of barriers,Brajnik, G.,Lecture Notes in Computer Science,112—121,2008,5176

[29] Examining the Relationship Between Visual Aesthetics and Web Accessibility: A Formative Study,Grace Mbipom,2008,Retrieved November 4th, 2009, from http://hcw-eprints.cs.man.ac.uk/68/,August

[30] The impact of accessibility assessment in macro scale universal usability studies of the web,,5—14,2008,W4A '08: Proceedings of the 2008 international cross-disciplinary conference on Web accessibility (W4A),978-1-60558-153-8,http://doi.acm.org/10.1145/1368044.1368048,New York, NY, USA,Beijing, China

[31] The art of ALT: toward a more accessible Web,Slatin, J.M.,Computers and Composition,73—81,2001,18,1

[32] Human Rights and Disability: The current use and future potential of United Nations human rights instruments in the context of disability,Quinn, G. and Degener, T. and Bruce, A.,2002

[33] Achieving web accessibility,Sierkowski, Brian,288—291,2002,SIGUCCS '02: Proceedings of the 30th annual ACM SIGUCCS conference on User services,1-58113-564-5,http://doi.acm.org/10.1145/588646.588725,New York, NY, USA,Providence, Rhode Island, USA

[34] Maximum accessibility: making your web site more usable for everyone,Slatin, J.M. and Rush, S.,2003

[35] Data and Statistics,The World Bank,Retrieved November 4th, 2009, from http://go.worldbank.org/SI5SSGAVZ0

[36] Statistics Portal,Organisation for Economic Co-operation and Development,Retrieved November 4th, 2009, from http://www.oecd.org/statsportal/0,3352,en_2825_293564_1_1_1_1_1,00.html

[37] Government at Glance,Organisation for Economic Co-operationd and Development ,Retrieved November 4th, 2009, from http://www.oecd.org/gov/indicators/govataglance,October

[38] Web Accessibility: A Digital Divide for Disabled People?,Adam, A. and Kreps, D.,217,2006,Social Inclusion: Societal and Organizational Implications for Information Systems: IFIP TC8 WG8. 2 International Working Conference, July 12-15, 2006,Springer

[39] Handbook of Research on Overcoming Digital Divides: Constructing an Equitable and Competitive Information Society,Ferro, E.,2009

[40] World Information Society Report,United Nations International Telecommunication Union,Retrieved December 21st, 2009, from http://www.itu.int/osg/spu/publications/worldinformationsociety/2007/WISR07_full-free.pdf

[41] Measuring the Information Society - The ICT Development Index,United Nations International Telecommunication Union,Retrieved February 1st, 2010, from http://www.itu.int/ITU-D/ict/publications/idi/2009/index.html

[42] Government Laws, Regulations, Policies, Standards, and Support,Shawn Lawton Henry,Retrieved March 18th, 2010, from http://www.uiaccess.com/access_links.html#government_laws_regs_policies

[43] World Laws,WebAIM,Retrieved March 18th, 2010, from http://www.webaim.org/articles/laws/world/

[44] Policies Relating to Web Accessibility,,Retrieved March 18th, 2010, from http://www.w3.org/WAI/Policy/

[45] Conforming to Web Site Accessibility: By Law or by Choice,McConnell, K.,2002

[46] Evaluating web site accessibility: validating the WAI guidelines through usability testing with disabled users,,535—538,2008,NordiCHI '08: Proceedings of the 5th Nordic conference on Human-computer interaction,978-1-59593-704-9,http://doi.acm.org/10.1145/1463160.1463238,New York, NY, USA,Lund, Sweden

The author of this document is:
Morten Goodwin
E-mail address is:
morten.goodwin ASCII 64 uia.no
Phone is:
+47 95 24 86 79