What is going on? A Systematic Review on QoS Scenarios

Abstract

Quality of Service (QoS) is a set of non-functional requirements that could enhance the adoption rate of web services. It is essential to consider the QoS requirements for web services in order to guarantee for quality of web services and attract more users. There are many QoS scenarios such as QoS computation, QoS evaluation, QoS specification, QoS prediction and QoS measurement. In this paper, we prepare a systematic review study to provide a broad vision of primary studies to researchers in QoS scenarios to find the gaps and give the overview to perform researches to fill in the gap. The study focuses on these five most usable digital libraries including the IEEE Explorer, Springer link, ISI web of Knowledge, Emerald, and Science Direct between 2001 and 2012. The result shows that there is an urgent need to do research especially in QoS prediction and QoS computation

Keywords

QoS requirement, quality of service, web service, QoS scenarios, QoS specification, QoS evaluation, QoS measurement, QoS computation, QoS prediction.

INTRODUCTION

The fulfilment of functional requirement without incorporating QoS requirement during development could not guarantee the successful implementation of a web service. moreover, since WSDL only describes functional requirements of the system and is not able to describe non-functional properties, we need to consider QoS to satisfy customer [chen et al., 2006]. Our research has determined five QoS scenarios relate to QoS for web service research field. these include (1) QoS specification which specify the requirement for QoS aware web service, (2) QoS computation which compute the QoS parameters and metrics either in quantitative or qualitative manner, (3) QoS evaluation which evaluate the QoS models and techniques in the system to identify the problem, (4) QoS prediction to forecast the QoS requirement in the system and (5) QoS measurement that measure the metrics of QoS for the system. The first systematic review regarding the QoS matter has been performed in 2003. The study is about QoS performance and regulation. as the result they have identified a set of opportunities for further and future regulation and practice of QoS which cover QoS goal, measurement and performance [niederprum et al., 2003].

In our research, we clarify the important and wide gap in QoS matters such as QoS specification, QoS computation, QoS evaluation, QoS prediction and QoS measurement. The outcome indicates that there is a huge lack of work in terms of QoS specification, computation and evaluation for web services. This research provides a background to suitably position the future work and further research activities. the rest of the paper is structured as follows, section2 explains about research protocol consists of research strategy and study selection, section3 displays the result of the search, section4 discusses the result and finally section 5 concludes our work.

IDENTIFICATION OF THE STUDY

The main research question in this study is defined as:

How is the trend of QoS scenarios? And what is going on to the future of QoS scenarios for web services?

According to Barbara Kitchenham paper [Kitchenham, 2007], if the population, intervention, outcomes and experimental design is clearly defined then the related primary study will be defined better [Afzal et al., 2008]. Therefore, the population of this study has been limited to QoS for web service application and network area. The intervention consists of five QoS scenarios which are QoS specification, QoS computation, QoS measurement, QoS evaluation and QoS prediction. The outcome of this study presents the situation of each QoS scenario with regards to clarifying the gap and lack of work on that specific area both in the network and application level.

GENERATING A SEARCH STUDY

The following search terms have been used to find the relevant papers:

i. Population: web services, web applications, network

ii. Intervention: Quality of service (QoS) specification, QoS computation, QoS evaluation and QoS measurement.

iii. Outcomes: non functional, QoS quality properties. Since this research is related to quality, the over all aspects of web services quality such as reliability, performance, availability, accessibility, speedy, integrity, usability, reusability, flexibility, interoperability, network security, maintainability, dependability, exception handling should be considered.

We have also used this Boolean search string to find the relevant papers: “QoS specification” AND “QoS computation” AND “QoS evaluation”.

The digital libraries applied to search the papers are as follows:

i. IEEE Explore

ii. Springer link

iii. ISI web of knowledge

iv. Science direct

v. Scopus

vi. Emerald

The search was conducted by the time span 2003-2012 and in both area of journals and conferences.

STUDY SELECTION

QoS scenarios play very important role and they are necessary to be considered in order to improve the quality of web services and the Internet communication over the network. In this study we have selected the papers in computer science area and excluded the QoS in another area such as electronic and communication. In some libraries, we mentioned only journal as the contribution of the journals on those libraries were more important rather than conference proceedings. Besides, the number of conference papers in some libraries for examples Springer-link, ISI web of knowledge, Science direct and Emerald were quite few (only 1-2). However, in other libraries such as IEEE Explore and Scopus we consider the number of conference papers as well. The search was in all the field not only title, neither abstract nor key word. Only in Springer link digital libraries we have mentioned chapter book as well. We also have repeated the search that combine of each scenario with the others such as “QoS specification” And “QoS computation”. Since the result for another combination of the scenarios like triple or even four or five combination of them, was not useful and as there was almost zero in each library. Therefore, we ignore that part. We have repeated the search for each digital library and for each scenario two times.

RESULT

The result illustrates that in overall the total number of papers in QoS prediction and QoS computation has the minimum rate of published papers compare to another scenarios. However, from the following tables we can see that in Emerald digital library we do not have even one paper regarding QoS prediction. In summary, the results clearly indicate the lack of work in QoS prediction and computation and totally in all the five QoS scenarios as well. Table 1 shows the number of papers in IEEE explore and Scopus digital libraries for journals and conferences.

Table 2 indicates the number of journal papers extracted from three digital libraries such as Science direct, ISI web of knowledge, and Emerald journals regard to five QoS scenarios. As the table shows the number of journal papers for QoS prediction in both Science direct and Emerald digital libraries is less than other Scenarios while in ISI web of knowledge this minimum rate belongs to QoS computation with only one paper in that area. However, the maximum rate is for QoS measurement for both Science direct and ISI web of knowledge digital libraries while this maximum rate for emerald journal belongs to QOS specification.

Table 3 illustrates the number of journal and book chapter in Springer Link digital library. As it is obvious from the table, the minimum rate of paper belongs to QoS computation scenarios in journals while for the eBook the minimum is for QoS prediction. However the maximum rate for both journal and eBook chapter belongs to QOS specification.

In any way from the papers is clear that we have big gap and lack of work in terms of prediction. Even in the next repetition of search we see the triples with prediction becomes minimum although they merged with measurement that is maximum work in single search. As it is clear from table 6, there is only one conference paper in IEEE Explore digital library that included four scenarios such as specification, computation, evaluation and measurement.

DISCUSSION

This section gives a broad overview of research and highlights the gap regards to the five QoS scenarios. The QoS scenarios include QoS computation, QoS measurement, QoS prediction, QoS specification and QoS evaluation.

A. QoS computation

QoS computation plays a vital role in web service selection. It consists of user satisfaction calculation, property value pre-processing and aggregation of multiple QoS properties aspects. The user requirements can be submitted during the computation process of QoS, and the degree of satisfaction to meets the requirements through the service can be calculated [ Shi et al., 2012]. The authors examined QoS computation from the experienced and novice users’ perspectives and proposed a user-cantered computation model for QoS to provide a new choice of normalization in attribute value pre-processing terms, an approach of approximation in user satisfaction calculation aspect, and a weight suggestion way in aggregation of multiple properties aspect. In [Liu, et al., 2004] they talked about QoS computation regarding three generic criteria of QoS including reputation, execution duration and price. Besides, they mentioned about usability as business related criteria of QoS. The authors have also stated that execution price can be provided by service providers and execution duration can be computed based on service invocation instances, while service reputation is based on service requesters’ feedback.

In [Syed Muhammad, 2006] the author presented a fair QoS computation framework for web service selection. The author mentioned that the QoS model should be quite intelligent to predict the trend of QoS from certain providers at run time and also to notify the provider when the QoS of its service is below a certain threshold. In another research [Moussa and Kohout, 2001] the authors proposed an approach to build a new paradigm for modelling and solving QoS computation problems by providing a foundation and overview of the possible utilization of fuzzy logic in the area of QoS data communication. Since user satisfaction is influenced by optimizing service QoS computation, therefore QoS computation techniques are used to return the real QoS user experience during execution of service. The authors in [Wagner et al., 2012] proposed a new QoS technique considering QoS dependencies and can be used for multiple workflows at the same time.

B. QoS measurement

QoS measurement techniques are essential for estimating the prior of performance metrics to confirm its QoS. There are two main categories for measurement scheme: (1) active (obtaining direct delay performance through transmitting search by sender and then receive a response by receiver) and (2) passive measurement (indirect estimation of delay performance by suing equipment) [Ryoki, et al., 2002]. In the design part of web service development, the measurement details for the definition of metrics should be separated. Measurement partner which is a group of performing measurement tasks should fill in the metric measurement details, according to the service execution environment [Zhou et al., 2005].

The authors in another paper mentioned that it is essential to support a measurement system as well as QoS-aware discovery system to improve provider’s competition benefits. The authors presented a framework for QoS measurement based on DAML ontology along with measurement code generator to help generating measurement code based on DAML specification [Zhou et al., 2005]. In [Saito et al., 2007] the authors proposed a rapid and efficient scheme for measurement of end-to-end QoS systems. The framework gathers the information of QoS and autonomously measured at each network layer and then estimates the attributes of end-to-end QoS system from the gathered data.

The authors in [Yang et al., 2009] presented a scalable, distributed and cooperative framework for QoS measurement using multi agent technique to provide an independent service for network application which measure QoS characteristics because currently the overlay network application fulfil measurement themselves. Another paper creates a new technique to gather and analyse the behaviour of user service selection and set the characteristics of measurement for QoS automatically. The technique is based on QoS constraint and clarifies the QoS measurement and their weight in different area [Liu et al., 2009]. A survey has been performed concentrating on QoS measurement and semantic service retrieval in digital ecosystem environment through a case study to explore and formulate the existence issues in that domain by presenting a solution [Dong et al., 2010].

C. QoS prediction

QoS prediction is important for both users and providers. In terms of users because of choosing providers for optimal provision regarding QoS of their application, and for providers they are willing to perform QoS forecasting to plan resource assignment to applications and users, to provide capacity planning and network dimensioning considering short-, medium- and long QoS forecasts based on real network measurements [Miloucheva et al., 2003]. Predicting of QoS changes in discovery phase can reduce the web services thrash [Li et al., 2009].

In [Yang et al., 2003] they proposed a QoS prediction approach regarding to dynamic, real time and distributed benchmark employed as Test bed. The authors introduced in the paper a strong algorithm to reject outliers automatically and achieve predicting of path latency. They have experimentally evaluated that their technique has higher accuracy compare to previous methods. In [Dong et al., 2009] the authors proposed a technique for performance prediction of composition in web services based on BPEL which translate specification of web service composition into Stochastic Petri Nets (SPN) model and then drive the analytical prediction of performance in process completion time. The following papers talked about the QoS attributes prediction such as reliability prediction [Ren et al., 2009; Zheng and Lyu, 2010], performance prediction [Gao and Wu, 2005; Dong et al., 2009].

We have also some researches performed to forecast QoS by some techniques such as QoS Ontology and BP Neutral Networks [Hai and Li, 2012], Collaborative Filtering [Shao et al., 2007], ARIMA and GARCH Models [Amin etal., 2012, Neighbourhood Integrated Matrix Factorization [Zheng et al., 2012], Multi-Dimension QoS [Li et al., 2011], Collaborative Filtering [Zhang et al., 2010], service using information [Zhang et al., 2010], improved collaborative filtering [Li et al. 2010], Multi Agents [Malak et al., 2009] which most of them are focusing on user perspective.

D. QoS specification

QoS specification can be clarified by a specification language or a set of pre-defined APIs [Zhou et al., 2006]. The authors in [Zhou et al., 2006] proposed a semantic quality of service specification model that is ontology based technique to facilitates information sharing and negotiation among QoS entities along the end-to-end path.

In [Jin and Nahrstedt, 2004] they mentioned that QoS specification should describe qualitative and quantitative QoS parameters, has to be naturally declarative and must specify the requirement only. They categorized the QoS specification languages by providing taxonomy. The authors mentioned five criteria for having good QoS specification such as expressiveness, declaratively, independence, extensibility and reusability. In [Tondello and Siqueira 2008] the authors presented an ontology to enable QoS specification for semantic web service which can be used on both development stage and publishing and discovery stages of web services.

In Ambient Intelligence (AMI) systems [Schneider et al., 2007] they elaborated QoS specification of AmI systems and described an initial technique towards a solution. They argued that current QoS specification approaches are more focusing on communication systems and do not address the issues such as high dynamics aspects. The authors stated that the service specification includes syntactic specification and semantic specification. Semantic service specification is needed when the composition should be done automatically while the syntactic specification consists of functional aspects of the services included the information of the quality of data. It seems that syntactic specification captures both functional and non-functional (QoS) information.

In [Wang et al., 2007] they provided a specification language with a tool to define QoS requirements, offers and contracts. In their framework, they consider three types of services: task-oriented services, message-oriented and streaming media services. In 2004, a research survey has been done to compare the QoS specification languages for distributed multimedia application to show how to evaluate QoS specification languages, and for designing new language what aspects should be considered [Jin and Nahrstedt, 2004].

E. QoS evaluation

QoS evaluation is necessary and crucial specially in network issue and from user’s perspective due to identifying the problem may occur in the subscribe [Cotanis, 2005]. In [Shiping and Greenfield, 2004] the authors presented an empirical study to evaluate the implementation of QoS for a JMS product (an API specification to define a standard to access messaging services for java applications). In [Lo and Wang, 2007] the authors presented a QoS evaluation framework called (CosmosQoS) for QoS-required Web services to monitor, store and evaluate the physical usage and post-condition of requested web services in a requester-located network . The authors believe that the proposed model support users to properly select QoS based web services and also to effect of network connection and configuration between user and providers. The authors in another research [Guo et al., 2008] have attempted to solve the QoS ranking problem in web service by proposing an algorithm for QoS evaluation based on neural network artificial intelligent.

OWL-s is a language to describe semantic web services. However, it does not consist of QoS evaluation details. For that purpose a research has been done to propose a new QoS evaluation model in semantic web services to enhance the precision, effectiveness and smartness of web service selection and auto discovery mechanism [Wu and Xia, 2008]. Another research [Cristofaro et al., 2009] proposed a QoS evaluation model for VOIP with video conference and examined the impact of various discipline and codecs on VOIP. The authors in another article [Li and Lu, 2009] proposed a new network QoS evaluation model based on customer satisfaction indices such as user behaviour and also technical factors to make it reasonable and comprehensive.

Regarding to network area again, the authors in another article [Liang et al., 2009] based on middleware presented a mechanism for QoS guarantee to meet the requirements of Wireless Sensor Networks (WSNs). They have also proposed a QoS evaluation model based on multi dimension technique to solve the QoS evaluation problems of network. It is quite obvious that in QoS evaluation part, most of the papers are related to network area such as [Audah etal., 2011; Kumar et al., 2011; Audah etal., 2011;Rejeb etal., 2011;Tymchenko and Kolodij 2008; Zhang et al., 2011; Qiang et al., 2009; Li et al., 2009; Umbert et al., 2007] not application level and service itself. From the paper it is obvious that there is no comprehensive reference for QoS concepts and their relationship to show what are QoS parameters exactly, how they can be predicted, how they can be computed and measured, how to evaluate the QoS parameters and how can we use them for our purpose specially regarding to application level and service itself to develop a desired web service.

CONCLUSION

The idea of creating this paper came from huge lack of work regarding QoS matters especially in application level. Therefore, in this paper we have decided to show the lack of work systematically and reasonably. We have considered five major QoS scenarios such as QoS specification, computation, measurement, evaluation and prediction. The result clearly shows that there is a huge gap in predicting computing and specifying QoS especially in application level of web service development. It is quite obvious that for the future work there are still lots of work need to be done to fulfil the gap of QoS scenarios especially in the application level and particularly in the initial stage of web service development phase.

References

1. Amin, A., Colman, A., & Grunske, L.. An Approach to Forecasting QoS Attributes of Web Services Based on ARIMA and GARCH Models. In the Proceedings of IEEE 19th International Conference on Web Services (pp. 74-81) , 2012.
2. Audah, L., Sun, Z, & Cruickshank. H.. QoS Evaluation of HTTP over Satellites. In the Proceedings of IEEE International Conference on Cyber- Enabled Distributed Computing and Knowledge Discovery (pp. 177-182) ,2011.
3. Audah, L., Sun, Z, & Cruickshank. H.. QoS evaluation of multiservice applications over integrated satellite-terrestrial networks. In the 3rd International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT) (pp. 1-8), 2011.
4. Rejeb, S.B. and Tabbane, S.. A novel cross-layer design and QoS evaluation for wireless multiservice networks. In the Proceedings of the 7th International Wireless Communications and Mobile Computing Conference (IWCMC)(pp. 320-325) , 2011.
5. Wang, C., Wang, G., Wang, H., Chen, A., & Santiago, R.. Quality of Service Contract Specification, Establishment, and Monitoring for Service Level Management. Journal of Object Technology, 6(11), Special Issue on Advances in Quality of Service Management, 25-44, 2007.
6. Zhou, C., Chia, L.T., & Lee, B.S.. Semantics in service discovery and QoS measurement. IT Professional, 7(2): 29-34, 2005.
7. Shi, C., Lin, D., & Ishida, T.. User-Centered QoS Computation for Web Service Selection. In the proceedings of IEEE 19th International Conference on Web Services (pp. 456-463) , 2012.
8. Cotanis, I.. A statistical measurement model for the 3G networks QoS evaluation. In the Proceedings of IEEE Wireless Communications and Networking Conference (pp. 1231-1235) , 2005.
9. Schneider, D., Anastasopoulos, M., Bayer, J., Becker, M. & Webel, C.. QoS Specification in Ambient Intelligence Systems. In the Proceedings of IEEE International Conference Pervasive Services (pp. 295-300) ,2007.
10. Cristofaro, N.D., McGill, G., Sallahi, A., Davis, M., Alsibai, A., & St-Hilaire, M.. QoS evaluation of a voice over IP network with video: A case study. In the Proceedings of the Canadian Conference on Electrical and Computer Engineering (pp. 288-292) ,2009.
11. Qiang, D., Xie, D.L., & Chen, S.Z.. A Fuzzy Logic Based QoS Evaluation Model for Wireless Sensor Network. In the Proceedings of the 5th International Conference on Wireless Communications, Networking and Mobile Computing (pp. 1-4) ,2009.
12. Niederprum, A., Hense, A., Kulenkampff, G., & Dieke, A.K.. Quality of Service Objectives, Performance and Measurement in Relation to Community Universal Postal Service, Final Report, wik-Consult, 2003.
13. Liu, G.Q., Zhu, Z.L., Li, D.C., & Liu, Y.. A Method of QoS Measurement Based on User Behavior Analysis. In the Proceedings of IEEE International Conference on e-Business Engineering, (pp. 383-387) ,2009.
14. Dong, H., Hussain, F.K., and Chang, E.. Semantic Service Retrieval and QoS Measurement in the Digital Ecosystem Environment. In the Proceedings of the International Conference on Complex, Intelligent and Software Intensive Systems, (pp. 153-160), 2010.
15. Zhang, H., Zhimin, A., & Tian, Z. QoS evaluation based on extend E-Model in VoIP. In the Proceedings of the 13th International Conference on Advanced Communication Technology, (pp. 852-854), 2011.
16. Miloucheva, I., Muller, E., & Anzaloni, A.. A practical approach to forecast Quality of Service parameters considering outliers. Retrieved March 20, 2013, from http://www.ist-intermon.org/workshop/papers/06_03_end_arima_corrected.pdf ,2003.
17. Jin, J., & Nahrstedt, K.. QoS specification languages for distributed multimedia applications: a survey and taxonomy. MultiMedia, IEEE, 11(3): 74- 87. 2004.
18. Liang, J.B., Chen, N.J., & Yu, M.M.. A cloud model based multi-dimension QoS evaluation mechanism for Wireless Sensor Networks. In the Proceedings of the 4th International Conference on Computer Science & Education, (pp. 348-352) ,2009.
19. Kitchenham, B.. Guidelines for Performing Systematic Literature Reviews in Software Engineering. (Technical Report EBSE-2007-01), Keele University and University of Durham, UK, 2007.
20. Kumar, W., Muhtar, A., Qazi, B.R., & Elmirghani, J.M.H.. Energy and QoS Evaluation for a V2R Network. In the Proceedings of IEEE Global Telecommunications Conference (GLOBECOM 2011), (pp. 1-5) ,2011.
21. Li, C.C., Cui, L.Q., Yue, D., & Feng, W.X.. A QoS prediction approach based on improved collaborative filtering. In the Proceedings of the 2nd International Conference on Advanced Computer Control (ICACC), (pp. 519-522) ,2010.
22. Li, M., Huai, J.P., & Guo, H.P. An Adaptive Web Services Selection Method Based on the QoS Prediction Mechanism. In the Proceedings of IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology - Volume 01, IEEE Computer Society, (pp. 395-402) , 2009.
23. Hai, Y., & Li, X.. A Web Service QoS Predicting Algorithm based on QoS Ontology and BP Neutral Networks. International Journal of Advancements in Computing Technology, 4(4), 199, 2012.
24. Zhou, L., Pung, H.K., & Ngoh, L.H.. Towards Semantic Modeling for QoS Specification. In the Proceedings of the 31st IEEE Conference on Local Computer Networks, (pp. 361-368) , 2006.
25. Shao, L., Zhang, J., Wei, Y., Zhao, J., Xie, B., & Mei, H.. Personalized QoS Prediction forWeb Services via Collaborative Filtering. In the Proceedings of IEEE International Conference on Web Services, (pp. 439-446) ,2007.
26. Liu, Y., Ngu, A.H.H., & Zeng, L. QoS computation and policing in dynamic web service selection. In the Proceedings of the 13th International World Wide Web Conference on Alternate track papers & posters. New York, NY, USA, ACM, (66-73) , 2004.
27. Lo, N.W. and Wang, C.H. Web services QoS evaluation and service selection framework - a proxy-oriented approach. In the Proceedings of IEEE Region 10 Conference TENCON 2007, (pp. 1-5) , 2007.
28. Guo, L.Y., Chen, H.P., Yang, G., & Fei, R.Y.. A QoS Evaluation Algorithm for Web Service Ranking Based on Artificial Neural Network. In the Proceedings of the International Conference on Computer Science and Software Engineering, (pp. 318-384) , 2008.
29. Li, L., Rong, M., & Zhang, G.. A web service QoS prediction approach based on multi-dimension QoS. In the Proceedings of the 6th International Conference on Computer Science & Education (ICCSE), (pp.1319-1322) , 2011.
30. Malak, J. S., Mohsenzadeh, M., & Seyyedi, M.A. Web Service QoS Prediction Based on Multi Agents. In the Proceedings of the International Conference on Computer Technology and Development, (pp. 265-269) , 2009.
31. Moussa, A. S., & Kohout, L.J. Using BK-products of fuzzy relations in quality of service adaptive communication. In the Proceedings of the Joint 9th IFSA World Congress and 20th NAFIPS International Conference, (pp. 681-686) , 2001.
32. Ren, Y., Gu, Q., Qi, J., & Chen, D. Reliability Prediction of Web Service Composition Based on DTMC. In the Proceedings of the 3rd IEEE International Conference on Secure Software Integration and Reliability Improvement, (pp. 369-375) , 2009.
33. Ryoki, N., Kawhara, K., Ikenaga, T., & Oie, Y. Performance analysis of queue length distribution of tandem routers for QoS measurement. In the Proceedings of Symposium on Applications and the Internet (SAINT) Workshops, (pp. 82-87) , 2002.
34. Saito, Y., Hasegawa, M., & Murata, Y. A Fast and Efficient End-to-End QOS Measurement Scheme for Real Seamless Handover. In the Proceedings of IEEE 18th International Symposium on Personal, Indoor and Mobile Radio Communications, (pp. 1-5) , 2007.
35. Shiping, C., & Greenfield, P. QoS evaluation of JMS: an empirical approach. System Sciences. In the Proceedings of the 37th Annual Hawaii International Conference on System Sciences, (pp. 1-10) ,2004.
36. Syed Muhammad, A. A framework for QoS computation in web service and technology selection. Computer Standards & Interfaces, 28(6): 714-720, 2006.
37. Tondello, G. F., & Siqueira, F. The QoS-MO ontology for semantic QoS modeling. In the Proceedings of the 2008 ACM Symposium on Applied Computing. Fortaleza, Ceara, Brazil, ACM, (pp. 2336-2340, 2008.
38. Tymchenko, O., & Kolodij, R. Analysis of QoS evaluation methods for VoIP applications. In the Proceedings of International Conference on Modern Problems of Radio Engineering, Telecommunications and Computer Science, (pp. 482) ,2008.
39. Umbert, A., Budzisz, L., Vucevic, N., & Bernardo, F. An all-IP heterogeneous wireless testbed for RAT selection and e2e QoS evaluation. In the proceedings of the International Conference on Next Generation Mobile Applications, Services and Technologies, (pp. 310-315) , 2007.
40. Wagner, F., Klein, A., Klopper, B., Ishikawa, F., & Honiden, S. Multi-objective Service Composition with Time- and Input-Dependent QoS. In the Proceedings of IEEE 19th International Conference on Web Services, (pp. 234-241) , 2012.
41. Afzal, W., Torkar, R., & Feldt, R. A Systematic Mapping Study on Non-Functional Search-Based Software Testing. In the Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE'08), San Francisco, CA, USA, Knowledge Systems Institute Graduate School, (pp. 42-46), 2008.
42. Wu, J. & Xia, H. Study of Adaptive Qos Evaluation Model for Semantic Web Service. In the Proceedings of the International Conference on Information Management, Innovation Management and Industrial Engineering, ICIII '08, (pp. 480-483) ,2008.
43. Chen, Y.P., Li, Z.Z., Jin, Q.X., & Wang, C. Study on QoS Driven Web Services Composition. Springer-Verlag Berlin Heidelberg, 3841: 702-707, 2006.
44. [Yang, J.R., Hu, R.M., Chen, J., & Zhong, J.B. A Service-Oriented Framework of Distributed QoS Measurement Based on Multi-Agent for Overlay Network. In the Proceedings of the International Conference on Communication Software and Networks, ICCSN '09, (pp. 158-162) , 2009.
45. Yang, L., Welch, L. R., Liu, J., & Cavanaugh, C.D. (2003). A robust QoS forecasting technique for a dynamic, distributed real-time testbed. In the Proceedings of IEEE International Workshop on Computer Architectures for Machine Perception, (pp. 100).
46. Dong, Y., Xia, Y., Zhu, Q., & Yang, R. (2009). A Stochastic Approach to Predicting Performance of Web Service Composition. Journal of Computers, 4(6): 485-493.
47. Zhang, L., Zhang, B., Liu, Y., Gao, Y., & Zhu, Z.L. A Web Service QoS Prediction Approach Based on Collaborative Filtering. In the Proceedings of Services Computing Conference (APSCC), IEEE Asia-Pacific, (pp. 725-731) , 2010.
48. Zhang, L., Zhang, B., Na, J., Huang, L.P., & Zhang, M. An Approach for Web Service QoS Prediction Based on Service Using Information. In the Proceedings of the International Conference on Service Sciences (ICSS), (pp. 324-328), 2010.
49. Li, Z.J., & Lu, Y.T. A network QOS evaluation method based on customer satisfaction indices. In the Proceedings of the 8th International Conference on Machine Learning and Cybernetics, (PP.1328-1332) , 2009.
50. Zheng, Z., & Lyu, M.R. Collaborative reliability prediction of service-oriented systems. In the Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1. Cape Town, South Africa, (pp. 35-44) , 2010.
51. Zheng, Z., Ma, H., Lyu, M.R., & King, I. Collaborative Web Service QoS Prediction via Neighborhood Integrated Matrix Factorization. IEEE Transactions on Services Computing, (99): 1-1, 2012.
52. [Gao, Z., & Wu, G. Combining QoS-based service selection with performance prediction. In the Proceedings of IEEE International Conference on e- Business Engineering, (pp. 611-614) , 2005.
53. Li, Z., Lu, Y., & Feng, S. Study on Network QoS Evaluation Method Based on Nondimensionalization. In the Proceedings of the International Conference on Multimedia Information Networking and Security, (pp. 304-307) , 2009.
54. Zhou, C., Chia, L.T., & Lee, B.S. QoS Measurement Issues with DAML-QoS Ontology. In the Proceedings of the IEEE International Conference on e-Business Engineering, IEEE Computer Society, (pp. 395-402) , 2005.