The Usability of Healthcare Websites - How they were Assessed? A Systematic Literature Review on the Usability Evaluation | Open Access Journals

Yakışıklı erkek tatil için bir beldeye gidiyor burada kendisine türk Porno güzel bir seksi kadın ayarlıyor Onunla beraber otel odasına gidiyorlar Otel odasına rokettube giren kadın ilk önce erkekle sohbet ederek işi yavaş halletmeye çalışıyor sex hikayeleri Kocası fabrikatör olan sarışın Rus hatun şehirden biraz uzak olan bir türk porno kasabaya son derece lüks bir villa yaptırıp yerleşiyor Kocasını işe gönderip mobil porno istediği erkeği eve atan Rus hatun son olarak fotoğraf çekimi yapmak üzere türk porno evine gelen genç adamı bahçede azdırıyor Güzel hatun zengin bir iş adamının porno indir dostu olmayı kabul ediyor Adamın kendisine aldığı yazlık evde sikiş kalmaya başlayan hatun bir süre sonra kendi erkek arkadaşlarını bir bir çağırarak onlarla porno izle yapıyor Son olarak çağırdığı arkadaşını kapıda üzerinde beyaz gömleğin açık sikiş düğmelerinden fışkıran dik memeleri ile karşılayıp içeri girer girmez sikiş dudaklarına yapışarak sevişiyor Evin her köşesine yayılan inleme seslerinin eşliğinde yorgun düşerek orgazm oluyor

The Usability of Healthcare Websites - How they were Assessed? A Systematic Literature Review on the Usability Evaluation

Muhammad Usman1*, Mahmood Ashraf1 and Masitah Ghazali2

1Department of Computer Science, Federal Urdu University of Arts, Science and Technology, Islamabad, India

2Faculty of Computing, University Technology Malaysia, 81310 Johor Bahru, Johor, Malaysia

*Corresponding Author:
Muhammad Usman
Department of Computer Science
Federal Urdu University of Arts
Science and Technology, Islamabad, India
Tel: 22-429-765-2080

Received date: 02/06/2017; Accepted date: 24/06/2017; Published date: 30/06/2017

Visit for more related articles at Research & Reviews: Journal of Educational Studies


Background: The people demand for online services as they access hospital websites for health information and services before actually moving towards them due to the increased usage of Internet. Therefore, the health-related websites should be user-centered. Aims: This study systemically reviews the literature of usability evaluation of hospitals websites. It has been tried to highlight the preferred web usability evaluations methods applied by researchers to evaluate hospital’s websites since 2009. In addition, the vital parameters are also identified to evaluate hospital websites. Method: The standard systematic literature review method is adapted in the study and four digital libraries are searched for primary studies published during 2009 to 2014. The primary studies are thoroughly compared in accordance with the four categories described in 2012. Results: Twenty primary studies are collected and data extracted from these on the basis of research questions. It is observed that 50% of the researchers prefer questionnaire method for evaluation of hospital websites. Cross sectional method was the preference of another 15% researchers. In as many as 30% studies, researchers evaluated contents and usage of the hospital websites as basis for usability. Another 25% of the researchers evaluated on the basis of existing guidelines. 23% primary studies fell into physical presentation and information architecture categories. However, 33% research studies fell into contents category and 21% in interactivity. Conclusion: Increased usage of internet demand better online services by hospital on their websites. Researchers preferred questionnaire method and content and usage of websites was the basis for evaluation in most of the cases.


Active learning, Argumentation, Feedback, Formative assessment, Information technology, Reflection


Hospitals and health institutions are primary source of treatment and satisfaction for general public when they are in need of medical attention. Health institutions can play a vital role in the society by providing medical services as well as preventive measures against disease. Web is the fastest growing technology nowadays [1-4]. A website is considered as gateway towards information, products and services for an organization. Due to increased usage of web, hospitals need to provide satisfactory web serveries as well. Hospitals are turning towards web in order to provide enhanced services to the patients. User centred design has proven to be the answer to most problematic healthcare and medical systems, which includes the health-related websites that should also be user-centred. Large number of people access hospital websites for medical services before actually moving towards them. The increase in seeking of online health information has been observed in recent years [5-9]. Hospitals can provide online services and doctors can discuss health issues using Internet. The users demand for online services such as where to find health facilities, what services are available there and operating hours for the facility. By designing better interfaces and mended contents, hospital websites could play better role in providing services [1]. The need and expectations of the citizens become greater due to advancement of information and communication (ICT) technologies. The user’s expectation for website contents is an important area of interest these days. Moreover, there exists another aspect; when a website is accessed in case of emergency, and to what extent they are useful. Health websites can be equipped with usability standards to provide accurate and timely information [10-14].

Evaluation is an important component for developing interactive systems. Heuristic evaluation approach proposed has been adapted by researches for evaluation of websites and known as best way of evaluation. Some researchers applied these with little modification and some others adapted few of these heuristics. The underlying study used the categories in their research emphasized that evaluation based on heuristics is mostly applied by researchers by proposing their own heuristics on the basis of evidences collected through evaluation performed by users and experts. The researchers recorded large usability issues and wisely emerged them using grounded theory approach into four categories namely physical interaction, content, information architecture and interactivity [15-19]. Colour contrast, page layout and interactive elements were mainly discussed in the category “physical interaction”. Whereas, the category contents comprised of duplicated contents, contents not defined and too much contents. Information architecture category was devised with page structure, headings and title, purpose of the structure. At the end, “interactivity” category mainly included input/output formats, sequence of interaction, error messages and lack of feedback on user action. These set of categories were formulated to support the design and evaluation of interactive websites [20-23].

The paper is organized such that Section 2 contains method in which research questions, search process and inclusion/ exclusion criteria is explained. In Section 3 results are narrated. The discussion on the results is reported in the next section and conclusion is described in the last section of the paper [24-26].

This systematic literature review (SLR) is based on original guidelines proposed afterwards systematic literature reviews in software engineering [27-29].

Research Questions

The focus of the study is to answer following questions:

RQ1: What are the web usability evaluations methods used by researchers to evaluate hospital’s websites since 2009?

RQ2: What parameters are considered vital for the evaluation of hospital websites?

RQ3: Which found vital parameters could be placed under the categories proposed by Petrie and Power (2012)?

The answers to these research questions will allow us to summarize the current usability evaluation methods for hospital’s websites and help in identifying known usability issues raised by the researchers.

Search Process

The primary studies were manually searched for conference proceedings and papers published in journals from the selected digital libraries shown in Table 1.

Table 1. Selected digital libraries. Digital library Website
3 Springer
4 Science Direct

The studies from 2009 to 2014 have been searched in the above listed digital libraries. The search strings used were:

i. Usability AND Evaluation AND Hospital AND Websites

ii. Evaluation AND Methods AND Usability AND Hospital AND Websites

iii. Usability AND Evaluation AND Hospital OR Health Center AND Websites

iv. Hospital AND Website AND Evaluation

Inclusion and Exclusion Criteria

Each identified study was carefully evaluated for inclusion in the systematic literature review (SLR). The study that met the following criteria was included for SLR:

• Studies presenting usability evaluation methods.

• Studies evaluating websites of either hospital or health center for usability.

• Studies having usability component/heuristic evaluation of hospital’s websites.

• Studies which are conference proceedings or full research papers.

The studies with following shortcomings were not included:

• The studies falling outside the threshold year limit.

• The studies which only proposed the framework for evaluation.

• The papers with other than English language.

Data Collection

The data extracted from each study were:

• The author and full reference

• Abstract and conclusion to find the relevance to the subject matter

• Year of publication

The above relevance was observed and studies which met the criteria were downloaded. Most of the downloaded studies belong to ACM (37%) and least to Science Direct (10%).

The detail of downloaded papers and access date is elaborated in the Table 2 below:

Table 2. Detail of searching primary studies. Digital Library Date accessed Papers Downloaded Papers Included
1 ACM 11 and 22 June, 2015 30 06
2 IEEE 22 and 23 June, 2015 20 00
3 Springer 24 and 26 June, 2015 23 09
4 Science Direct 17 and 18 July, 2015 08 05
  Total   81 20

Data Analysis

The data was tabulated to show:

• The publishing year of primary study.

• The source of primary study from where it is downloaded.

• Evaluation method adopted by the researcher (RQ-1).

• Usability parameters were used during the experiment (RQ-2).

• The primary studies have been explored to categorize it (RQ-3).


Table 2 portrays the results of the search procedure. The total of 81 research studies was downloaded from the sources. Twenty studies were found relevant to the subject despite the large number of available articles in the digital libraries.

Table 3 shows the extract of primary research studies included in the SLR. The categories proposed by Petrie and Power are numbered as (I) Physical Representation, (II) Content, (III) Information Architecture and (IV) Interactivity. The studies are sorted in ascending order on year of publication, followed by authors and source:

Table 3. Systematic review studies. Author Source Key Contribution Evaluation Method Usability Parameters Category
1 Gwetu [6] ACM Investigation South African health institutions Manual assessment Self-designed framework on content and interactivity II and IV
2 Zufferey and Schulz [1] Science Direct Impact of a patient-centered website on Low-Back Pain Patients in Switzerland (qualitative) In-depth interviews Modalities and effect of using II and III
3 Choi and Bakken [5] Science Direct Heuristic evaluation of Neonatal ICU website for low literacy parents Task based user study and questionnaire Heuristic Evaluation I, II, III and IV
4 Dima et al. [3] ACM Hospital webpages in Romania, a case study Questionnaire Webpage structure, information quality I and III
5 Maifredi et al. [14] Springer Italian hospitals on the web Cross-sectional analysis Usage II
6 Moreno et al. [15] Springer Quality evaluation methodology for health related websites Focus group with 2-tuple fuzzy linguistic Framework of 30 criteria grouped in five dimensions I, II, III and IV
7 Fang-Fang et al. [12] Springer Evaluation criteria can be applied to Chinese language websites Literature survey Accuracy, readability, interactivity etc. I, III and IV
8 Liu et al. [13] Springer The quality and characteristics of general hospitals’ website in China Cross-sectional descriptive study Content, function, design, management and usage I, II and III
9 Rocha et al. [21] Springer Quality evaluation of health websites Literature survey ISO standard: content, services, technical I, II and IV
10 Sharit et al. [24] ACM Health problem solving by older persons using complex a government website Tasked based user study and questionnaire Interactivity, navigation II, III and IV
11 Petrie and Power [16] ACM Usability problems found by users and experts Think aloud protocol, Task based evaluation and questionnaire Heuristic Evaluation I, II, III and IV
12 Rafe and Monfaredzadeh [5] Springer Qualitative framework to evaluate hospital websites Questionnaire based user study Self-designed quality framework I, II, III and IV
13 Saeed et al. [22] ACM Usability evaluation of hospital websites in Pakistan Questionnaire bases user study Usage II
14 Selig et al. [23] Science Direct Evaluation of the online-presence of burn centers in Germany, Austria and Switzerland Multicenter, observational, cross-sectional study Previously published criteria to assess online performance II
15 Dias et al. [2] ACM Accessibility of existing web system with presented ACCESSA tool Representative tasks based user study Selected WCAG Guidelines 2.0 I, II, III and IV
16 Knijnenburg et al. [7] Springer Evaluation of a patient information website for childhood cancer survivors Online questionnaire System Usability Scale and questionnaire II
17 Lee et al. [9] Springer Website color guidelines for universal access in Korea and United States Doctor 2.1 by Fujitsu Color of website I
18 Raji et al. [19] Science Direct Evaluation of University Teaching Hospital Websites in Nigeria Task based user study and questionnaire Heuristic Evaluation I, II, III and IV
19 Raji et al. [20] Springer Usability Evaluation of Hospital Websites in Nigeria User testing and questionnaire based Usability, content, aesthetic design, information quality I, II, III and IV
20 Venkatesh et al. [27] Science Direct Usability evaluation of the Obamacare website Questionnaire based online survey Existing website usability guidelines available at II


The SLR was planned with the intent to explore the researchers preferred usability evaluation methods for evaluating hospital websites. The supplementary plan was to investigation the parameters vital for evaluating these websites. These parameters were further categorized on the basis of categories prepared to highlight the highly and least addressed areas.

In total, twenty (20) relevant studies have been found in the digital sources (Table 3). The studies are used to find the answers to the research questions. Table 4 reflects the answer to research question “web usability evaluations methods used by researchers to evaluate hospital’s websites since 2009”. Two of the selected primary studies (10%) were survey papers on evaluation of hospital websites. It has been observed that most of the researchers (50%) in the relevant studies used the questionnaire based method to evaluate the hospital websites. These questionnaires were either filled after performing the representative tasks or filled online as feedback. 15% researchers adopted cross sectional study for website evaluation. 5% of the researchers used interview method for usability evaluation. Researched used think aloud method in 5% cases and focus group was used in another 5% studies. In two cases (10%), the researcher used other methods for evaluation like manual assessment and Doctor 2.1 by Fujitsu.

Table 4. Web usability evaluations methods used by researchers to evaluate hospital’s websites. Evaluation Methods No. of Papers Percentage
1 Questionnaire 10 50%
2 Interview 01 5%
3 Think aloud 01 5%
4 Focus group 01 5%
5 Cross sectional study 03 15%
6 Literature survey 02 10%
7 Others 02 10%

Table 5 represents the answer to the research question “parameters considered vital for the evaluation of hospital websites”. In five research studies (25%), researchers used existing guidelines either by the ISO or by the government of relevant region. Evaluation on the basis of contents and information quality was also basis for researchers in six primary studies (30%). Self-designed framework remained the focus of 15% researchers. Another 15% used heuristic evaluation. 10% researchers either did not recorded the parameters or the approach was not based on parameters. Grounded theory approach was used in one study (5%).

Table 5. Parameters, considered vital for the evaluation of hospital websites. Usability Parameters No. of Papers Percentage
1 Self-designed framework 03 15%
2 Existing guidelines 05 25%
3 Heuristic evaluation 03 15%
4 Contents, usage and information quality 06 30%
5 Not recorded/applicable 02 10%
6 Grounded theory approach 01 5%

The answer to the final research question “categorization of vital parameters in the light of Petrie and Power categories” has been elaborated in Table 6. The research studies have carefully been analyzed to match with the categories proposed. Most Researchers addressed more than one category in their paper. In seven cases, all four categories have been addressed during the evaluation. Content and usage remained the primary focus in 33% of cases like in their paper focused contents for the evaluation of websites. Similarly, studies focused usability, content, aesthetic design, information quality (all four categories). Studies only focused usage for website evaluation. “Contents” is the highly addressed parameter for evaluation. Interactivity (21%), however, remained the least concern for evaluation in primary studies. Physical presentation and information architecture has been focused by 23% researchers each. Recent studies address contents and interactivity (II and IV) in their research paper. Studies evaluated on the basis of contents and information architecture.

Table 6. Categories of parameters in light of patrie and power categories. Categories No. of Papers Percentage
1 Physical Presentation 12 23%
2 Content 17 33%
3 Information Architecture 12 23%
4 Interactivity 11 21%


The increased usage of internet requires better support and services in the area of health as well. People demand equipped hospital websites having online health services and information. A hospital website designed and developed with user-centered approach can play vital role in society for improving health standards. Website evaluation, however, come into play with a purpose. Twenty research studies presenting hospital website evaluation were collected and analyzed for evaluation methods, parameters for evaluation and categorization.

As many as 50% of the research studies have been evaluated on the basis of questionnaire either by user after performing representative tasks or online feedback. Content and usage of websites with 30% among other remained the primary focus as usability parameter. However, existing guidelines was the priority of 25% of the researchers. Usability parameters were categories on the basis of proposal and it reflected that 33% papers addressed the category of “contents”. 35% of the research studies under consideration have addressed all the four categories.

The evaluation methods described and vital parameters highlighted are big picture for researchers who intend to focus websites for usability evaluation particularly in healthcare domain. These methods and parameters can lead in particular direction of research either by establishing connection with mostly addressed areas or by focusing least addressed factors.