1Department of Computer Science, Federal Urdu University of Arts, Science and Technology, Islamabad, India
2Faculty of Computing, University Technology Malaysia, 81310 Johor Bahru, Johor, Malaysia
Received date: 02/06/2017; Accepted date: 24/06/2017; Published date: 30/06/2017
Visit for more related articles at Research & Reviews: Journal of Educational Studies
Background: The people demand for online services as they access hospital websites for health information and services before actually moving towards them due to the increased usage of Internet. Therefore, the health-related websites should be user-centered. Aims: This study systemically reviews the literature of usability evaluation of hospitals websites. It has been tried to highlight the preferred web usability evaluations methods applied by researchers to evaluate hospital’s websites since 2009. In addition, the vital parameters are also identified to evaluate hospital websites. Method: The standard systematic literature review method is adapted in the study and four digital libraries are searched for primary studies published during 2009 to 2014. The primary studies are thoroughly compared in accordance with the four categories described in 2012. Results: Twenty primary studies are collected and data extracted from these on the basis of research questions. It is observed that 50% of the researchers prefer questionnaire method for evaluation of hospital websites. Cross sectional method was the preference of another 15% researchers. In as many as 30% studies, researchers evaluated contents and usage of the hospital websites as basis for usability. Another 25% of the researchers evaluated on the basis of existing guidelines. 23% primary studies fell into physical presentation and information architecture categories. However, 33% research studies fell into contents category and 21% in interactivity. Conclusion: Increased usage of internet demand better online services by hospital on their websites. Researchers preferred questionnaire method and content and usage of websites was the basis for evaluation in most of the cases.
Active learning, Argumentation, Feedback, Formative assessment, Information technology, Reflection
Hospitals and health institutions are primary source of treatment and satisfaction for general public when they are in need of medical attention. Health institutions can play a vital role in the society by providing medical services as well as preventive measures against disease. Web is the fastest growing technology nowadays [1-4]. A website is considered as gateway towards information, products and services for an organization. Due to increased usage of web, hospitals need to provide satisfactory web serveries as well. Hospitals are turning towards web in order to provide enhanced services to the patients. User centred design has proven to be the answer to most problematic healthcare and medical systems, which includes the health-related websites that should also be user-centred. Large number of people access hospital websites for medical services before actually moving towards them. The increase in seeking of online health information has been observed in recent years [5-9]. Hospitals can provide online services and doctors can discuss health issues using Internet. The users demand for online services such as where to find health facilities, what services are available there and operating hours for the facility. By designing better interfaces and mended contents, hospital websites could play better role in providing services [1]. The need and expectations of the citizens become greater due to advancement of information and communication (ICT) technologies. The user’s expectation for website contents is an important area of interest these days. Moreover, there exists another aspect; when a website is accessed in case of emergency, and to what extent they are useful. Health websites can be equipped with usability standards to provide accurate and timely information [10-14].
Evaluation is an important component for developing interactive systems. Heuristic evaluation approach proposed has been adapted by researches for evaluation of websites and known as best way of evaluation. Some researchers applied these with little modification and some others adapted few of these heuristics. The underlying study used the categories in their research emphasized that evaluation based on heuristics is mostly applied by researchers by proposing their own heuristics on the basis of evidences collected through evaluation performed by users and experts. The researchers recorded large usability issues and wisely emerged them using grounded theory approach into four categories namely physical interaction, content, information architecture and interactivity [15-19]. Colour contrast, page layout and interactive elements were mainly discussed in the category “physical interaction”. Whereas, the category contents comprised of duplicated contents, contents not defined and too much contents. Information architecture category was devised with page structure, headings and title, purpose of the structure. At the end, “interactivity” category mainly included input/output formats, sequence of interaction, error messages and lack of feedback on user action. These set of categories were formulated to support the design and evaluation of interactive websites [20-23].
The paper is organized such that Section 2 contains method in which research questions, search process and inclusion/ exclusion criteria is explained. In Section 3 results are narrated. The discussion on the results is reported in the next section and conclusion is described in the last section of the paper [24-26].
This systematic literature review (SLR) is based on original guidelines proposed afterwards systematic literature reviews in software engineering [27-29].
The focus of the study is to answer following questions:
RQ1: What are the web usability evaluations methods used by researchers to evaluate hospital’s websites since 2009?
RQ2: What parameters are considered vital for the evaluation of hospital websites?
RQ3: Which found vital parameters could be placed under the categories proposed by Petrie and Power (2012)?
The answers to these research questions will allow us to summarize the current usability evaluation methods for hospital’s websites and help in identifying known usability issues raised by the researchers.
Search Process
The primary studies were manually searched for conference proceedings and papers published in journals from the selected digital libraries shown in Table 1.
Table 1. Selected digital libraries.
| S.no | Digital library | Website |
|---|---|---|
| 1 | ACM | http://dl.acm.org |
| 2 | IEEE | http://ieeexplore.ieee.org |
| 3 | Springer | www.link.springer.com |
| 4 | Science Direct | www.sciencedirect.com |
The studies from 2009 to 2014 have been searched in the above listed digital libraries. The search strings used were:
i. Usability AND Evaluation AND Hospital AND Websitesii. Evaluation AND Methods AND Usability AND Hospital AND Websites
iii. Usability AND Evaluation AND Hospital OR Health Center AND Websites
iv. Hospital AND Website AND Evaluation
Inclusion and Exclusion Criteria
Each identified study was carefully evaluated for inclusion in the systematic literature review (SLR). The study that met the following criteria was included for SLR:
• Studies presenting usability evaluation methods.
• Studies evaluating websites of either hospital or health center for usability.
• Studies having usability component/heuristic evaluation of hospital’s websites.
• Studies which are conference proceedings or full research papers.
The studies with following shortcomings were not included:
• The studies falling outside the threshold year limit.
• The studies which only proposed the framework for evaluation.
• The papers with other than English language.
Data Collection
The data extracted from each study were:
• The author and full reference
• Abstract and conclusion to find the relevance to the subject matter
• Year of publication
The above relevance was observed and studies which met the criteria were downloaded. Most of the downloaded studies belong to ACM (37%) and least to Science Direct (10%).
The detail of downloaded papers and access date is elaborated in the Table 2 below:
Table 2. Detail of searching primary studies.
| S.no | Digital Library | Date accessed | Papers Downloaded | Papers Included |
|---|---|---|---|---|
| 1 | ACM | 11 and 22 June, 2015 | 30 | 06 |
| 2 | IEEE | 22 and 23 June, 2015 | 20 | 00 |
| 3 | Springer | 24 and 26 June, 2015 | 23 | 09 |
| 4 | Science Direct | 17 and 18 July, 2015 | 08 | 05 |
| Total | 81 | 20 |
Data Analysis
The data was tabulated to show:
• The publishing year of primary study.
• The source of primary study from where it is downloaded.
• Evaluation method adopted by the researcher (RQ-1).
• Usability parameters were used during the experiment (RQ-2).
• The primary studies have been explored to categorize it (RQ-3).
Table 2 portrays the results of the search procedure. The total of 81 research studies was downloaded from the sources. Twenty studies were found relevant to the subject despite the large number of available articles in the digital libraries.
Table 3 shows the extract of primary research studies included in the SLR. The categories proposed by Petrie and Power are numbered as (I) Physical Representation, (II) Content, (III) Information Architecture and (IV) Interactivity. The studies are sorted in ascending order on year of publication, followed by authors and source:
Table 3. Systematic review studies.
| S.no | Author | Source | Key Contribution | Evaluation Method | Usability Parameters | Category |
|---|---|---|---|---|---|---|
| 1 | Gwetu [6] | ACM | Investigation South African health institutions | Manual assessment | Self-designed framework on content and interactivity | II and IV |
| 2 | Zufferey and Schulz [1] | Science Direct | Impact of a patient-centered website on Low-Back Pain Patients in Switzerland (qualitative) | In-depth interviews | Modalities and effect of using | II and III |
| 3 | Choi and Bakken [5] | Science Direct | Heuristic evaluation of Neonatal ICU website for low literacy parents | Task based user study and questionnaire | Heuristic Evaluation | I, II, III and IV |
| 4 | Dima et al. [3] | ACM | Hospital webpages in Romania, a case study | Questionnaire | Webpage structure, information quality | I and III |
| 5 | Maifredi et al. [14] | Springer | Italian hospitals on the web | Cross-sectional analysis | Usage | II |
| 6 | Moreno et al. [15] | Springer | Quality evaluation methodology for health related websites | Focus group with 2-tuple fuzzy linguistic | Framework of 30 criteria grouped in five dimensions | I, II, III and IV |
| 7 | Fang-Fang et al. [12] | Springer | Evaluation criteria can be applied to Chinese language websites | Literature survey | Accuracy, readability, interactivity etc. | I, III and IV |
| 8 | Liu et al. [13] | Springer | The quality and characteristics of general hospitals’ website in China | Cross-sectional descriptive study | Content, function, design, management and usage | I, II and III |
| 9 | Rocha et al. [21] | Springer | Quality evaluation of health websites | Literature survey | ISO standard: content, services, technical | I, II and IV |
| 10 | Sharit et al. [24] | ACM | Health problem solving by older persons using complex a government website | Tasked based user study and questionnaire | Interactivity, navigation | II, III and IV |
| 11 | Petrie and Power [16] | ACM | Usability problems found by users and experts | Think aloud protocol, Task based evaluation and questionnaire | Heuristic Evaluation | I, II, III and IV |
| 12 | Rafe and Monfaredzadeh [5] | Springer | Qualitative framework to evaluate hospital websites | Questionnaire based user study | Self-designed quality framework | I, II, III and IV |
| 13 | Saeed et al. [22] | ACM | Usability evaluation of hospital websites in Pakistan | Questionnaire bases user study | Usage | II |
| 14 | Selig et al. [23] | Science Direct | Evaluation of the online-presence of burn centers in Germany, Austria and Switzerland | Multicenter, observational, cross-sectional study | Previously published criteria to assess online performance | II |
| 15 | Dias et al. [2] | ACM | Accessibility of existing web system with presented ACCESSA tool | Representative tasks based user study | Selected WCAG Guidelines 2.0 | I, II, III and IV |
| 16 | Knijnenburg et al. [7] | Springer | Evaluation of a patient information website for childhood cancer survivors | Online questionnaire | System Usability Scale and questionnaire | II |
| 17 | Lee et al. [9] | Springer | Website color guidelines for universal access in Korea and United States | Doctor 2.1 by Fujitsu | Color of website | I |
| 18 | Raji et al. [19] | Science Direct | Evaluation of University Teaching Hospital Websites in Nigeria | Task based user study and questionnaire | Heuristic Evaluation | I, II, III and IV |
| 19 | Raji et al. [20] | Springer | Usability Evaluation of Hospital Websites in Nigeria | User testing and questionnaire based | Usability, content, aesthetic design, information quality | I, II, III and IV |
| 20 | Venkatesh et al. [27] | Science Direct | Usability evaluation of the Obamacare website | Questionnaire based online survey | Existing website usability guidelines available at usability.gov | II |
The SLR was planned with the intent to explore the researchers preferred usability evaluation methods for evaluating hospital websites. The supplementary plan was to investigation the parameters vital for evaluating these websites. These parameters were further categorized on the basis of categories prepared to highlight the highly and least addressed areas.
In total, twenty (20) relevant studies have been found in the digital sources (Table 3). The studies are used to find the answers to the research questions. Table 4 reflects the answer to research question “web usability evaluations methods used by researchers to evaluate hospital’s websites since 2009”. Two of the selected primary studies (10%) were survey papers on evaluation of hospital websites. It has been observed that most of the researchers (50%) in the relevant studies used the questionnaire based method to evaluate the hospital websites. These questionnaires were either filled after performing the representative tasks or filled online as feedback. 15% researchers adopted cross sectional study for website evaluation. 5% of the researchers used interview method for usability evaluation. Researched used think aloud method in 5% cases and focus group was used in another 5% studies. In two cases (10%), the researcher used other methods for evaluation like manual assessment and Doctor 2.1 by Fujitsu.
Table 4. Web usability evaluations methods used by researchers to evaluate hospital’s websites.
| S.no | Evaluation Methods | No. of Papers | Percentage |
|---|---|---|---|
| 1 | Questionnaire | 10 | 50% |
| 2 | Interview | 01 | 5% |
| 3 | Think aloud | 01 | 5% |
| 4 | Focus group | 01 | 5% |
| 5 | Cross sectional study | 03 | 15% |
| 6 | Literature survey | 02 | 10% |
| 7 | Others | 02 | 10% |
Table 5 represents the answer to the research question “parameters considered vital for the evaluation of hospital websites”. In five research studies (25%), researchers used existing guidelines either by the ISO or by the government of relevant region. Evaluation on the basis of contents and information quality was also basis for researchers in six primary studies (30%). Self-designed framework remained the focus of 15% researchers. Another 15% used heuristic evaluation. 10% researchers either did not recorded the parameters or the approach was not based on parameters. Grounded theory approach was used in one study (5%).
Table 5. Parameters, considered vital for the evaluation of hospital websites.
| S.no | Usability Parameters | No. of Papers | Percentage |
|---|---|---|---|
| 1 | Self-designed framework | 03 | 15% |
| 2 | Existing guidelines | 05 | 25% |
| 3 | Heuristic evaluation | 03 | 15% |
| 4 | Contents, usage and information quality | 06 | 30% |
| 5 | Not recorded/applicable | 02 | 10% |
| 6 | Grounded theory approach | 01 | 5% |
The answer to the final research question “categorization of vital parameters in the light of Petrie and Power categories” has been elaborated in Table 6. The research studies have carefully been analyzed to match with the categories proposed. Most Researchers addressed more than one category in their paper. In seven cases, all four categories have been addressed during the evaluation. Content and usage remained the primary focus in 33% of cases like in their paper focused contents for the evaluation of websites. Similarly, studies focused usability, content, aesthetic design, information quality (all four categories). Studies only focused usage for website evaluation. “Contents” is the highly addressed parameter for evaluation. Interactivity (21%), however, remained the least concern for evaluation in primary studies. Physical presentation and information architecture has been focused by 23% researchers each. Recent studies address contents and interactivity (II and IV) in their research paper. Studies evaluated on the basis of contents and information architecture.
Table 6. Categories of parameters in light of patrie and power categories.
| S.no | Categories | No. of Papers | Percentage |
|---|---|---|---|
| 1 | Physical Presentation | 12 | 23% |
| 2 | Content | 17 | 33% |
| 3 | Information Architecture | 12 | 23% |
| 4 | Interactivity | 11 | 21% |
The increased usage of internet requires better support and services in the area of health as well. People demand equipped hospital websites having online health services and information. A hospital website designed and developed with user-centered approach can play vital role in society for improving health standards. Website evaluation, however, come into play with a purpose. Twenty research studies presenting hospital website evaluation were collected and analyzed for evaluation methods, parameters for evaluation and categorization.
As many as 50% of the research studies have been evaluated on the basis of questionnaire either by user after performing representative tasks or online feedback. Content and usage of websites with 30% among other remained the primary focus as usability parameter. However, existing guidelines was the priority of 25% of the researchers. Usability parameters were categories on the basis of proposal and it reflected that 33% papers addressed the category of “contents”. 35% of the research studies under consideration have addressed all the four categories.
The evaluation methods described and vital parameters highlighted are big picture for researchers who intend to focus websites for usability evaluation particularly in healthcare domain. These methods and parameters can lead in particular direction of research either by establishing connection with mostly addressed areas or by focusing least addressed factors.
Menu