The relation between quality of service and quality of experience

Factors influencing the quality of experience

Quality of experience (QoE) is now taken to be “the degree of delight or annoyance of the user of an application or service,” following ITU-T Recommendation P.10/G.100 (ITU-T 2017). Conventionally it is regarded as being determined by “influencing factors” of the following kinds, which are discussed in more detail in Qualinet White Paper (Qualinet 2013):

• Human-related influencing factors: properties of users, such as gender, age, education, background, expertise, feeling, expectation, hearing, and sight.
• System-related influencing factors: properties of ICT used, such as power consumption, usability of input and output devices, accuracy of encoding and decoding techniques, speed of operation, and correctness of transmission.
• Context-related influencing factors: properties of environments of users, such as location, mobility, observers, participants, costs, and benefits.

The range of factors here is wide enough to suggest that QoE would include accessibility, availability, and affordability. These are human-related, system-related, and context-related factors (respectively) and are also the normal concerns of universal service. Yet, in practice work on QoE does not deal much with them: it deals more with new services for the affluent than with old services for the disadvantaged.

Nonetheless a service that is unavailable can hardly be said to be offer its users much “delight.” Indeed coverage and reliability are relevant to quality of service (QoS), let alone QoE; here QoS is “the totality of characteristics of a telecommunications service that bear on its ability to satisfy stated and implied needs of the user of the service” (ITU-T 2017). Coverage is often assessed fairly easily, through checking the presence of cables or the strengths of radio signals. As users are interested in calls, messages, data streams, and data files, not signal strengths, it should be defined in terms of these interests of users and included in QoS. Likewise, reliability can be included in QoS.

Some other topics might appear to be excluded from QoE and QoS because they are not mentioned among the influencing factors. Among these topics are the correctness of applications (whether systems would produce the intended results in the absence of transmission errors) and the security of systems (whether systems would provide the results to, and only to, the intended people). QoE is perhaps concerned with these topics only to the extent that user feelings might be affected by knowledge, beliefs, or suspicions about them. For instance, for online language translation systems the speed of translation might influence the QoE but the accuracy of translation would not. Yet there are exceptions to these apparent exclusions from QoE: money transfer services would not offer acceptable experiences if they debited accounts wrongly or admitted unauthorized access to accounts. Here, as in some other areas, regulation of the application is closer to user concerns than regulation of the network.

Assessments of the quality of experience

QoE assessments can be subjective or objective. Subjective assessments determine quality by asking human assessors (but not by asking all end users at the point of use). Objective assessments estimate quality from values obtained by making measurements of the systems being assessed.

Subjective assessments require human assessors, each of whom rates each experience, usually on a five-point scale; the average (or “mean”) of the ratings is then taken to be the mean opinion score (MOS). The assessments might be performed in the laboratory or in the field, where they have the following advantages and disadvantages:

• Subjective assessments in the laboratory can to some degree imitate the circumstances of normal use but can never represent normal use perfectly. However, they can be controlled carefully so that they can focus on exactly what is of interest; in particular, the systems being assessed can be varied to find how the ratings of the experience by the assessors vary. Typically, there are fairly few assessors (and fairly few variations in the systems) because of the cost and effort involved in getting people to spend time in the laboratory.
• Subjective assessments in the field can be performed in some circumstances of normal use; for instance, wireless services can be tested where coverage is marginal. However, they might not be performed in all such circumstances of normal use; for instance, residential multimedia streaming services might not be tested in homes at peak times. There can be many assessors if the assessments use crowdsourcing, to get people to rate their experiences wherever and whenever they choose, but those people might not be representative of users in general.

Even the fullest possible subjective assessments of systems ignore the feelings and environments of people in places and at times other than those of the assessors.

Objective assessments do not require human assessors, so they are generally cheaper and faster than subjective ones. They usually involve making measurements of the systems being assessed, in order to extract values that can be combined in mathematical models to provide mean opinion scores. Their accuracy depends on whether the models are realistic enough to determine perceived quality much as human assessors would on average. Typically, the models are constructed on the basis of theories of perception, empirical studies of influencing factors, and machine learning from subjective assessments. They do not take account of all system-related influencing factors, let alone all human-related and context-related influencing factors. Such models are now available in ITU-T recommendations for voice, video and multimedia applications under various conditions, such as adaptive streaming and progressive downloading to televisions and smartphones of multimedia by over-the-top (OTT) and managed content providers (ITU-T 2020).

Objective assessments can be performed in the laboratory or in the field, but they can also be performed in the network. They have the following advantages and disadvantages:

• Objective assessments in the laboratory have the same advantages and disadvantages as subjective assessments in the laboratory.
• Objective assessments in the field can be more reliable than subjective assessments because they do not require human assessors. They can be performed wherever and whenever the relevant assessment instrumentation is installed. Installation might require that users put hardware or software in their homes or on their terminals or that testers have schedules for travelling with hardware or software. Though achieving this might still be difficult, it should be easier than ensuring that users make tests regularly.
• Objective assessments in the network resemble objective assessments in the laboratory, in that they still are not performed in circumstances of normal use. Installation of the assessment instrumentation can be unnecessary and making measurements might simply involve reading values that are already available, if the hardware and software are already embedded in network nodes or support systems.

Objective assessments are also possible without making measurements of the systems being assessed. Instead values, based on prior knowledge, are assumed and combined in a model to provide mean opinion scores. This technique is useful when networks are being planned.

QoS assessments often consider just the influencing factors that can be affected by network operators and content providers. Then, in terms of ITU-T Recommendation G.1000 (ITU-T 2001):

• Subjective assessments using ratings by human assessors assess the QoS perceived by the user (or customer).
• Objective assessments using values extracted by making measurements and combined in a model assess the QoS achieved by the operator (or provider).
• Objective assessments using values assumed from prior knowledge and combined in a model assess the QoS offered by the operator (or provider).

Measurements can of course be made without then being combined in a model. In fact, this is typical of QoS measurements currently. For instance, the delay and the packet loss ratio in an IP network are typically quoted separately, though they could be combined with assumptions about some influencing factors to provide an MOS for phone call conversations according to ITU-T Recommendation G.107 (ITU-T 2015). Quoting them separately lets operators concentrate the improvements in network performance on one of them and can be more useful if they affect different services to different extents.

Objective assessments of QoS should be calibrated against subjective assessments by large numbers of users. These subjective assessments might be regarded as measuring QoE. However, they are often performed in the laboratory with users who are compensated in some way for their time; they are then not even representative of the environments of users, let alone of the feelings of one particular user.

Much work on QoE standards development relies on assessments in the laboratory. It might then result in products, composed of hardware and software for performing objective assessments that match subjective assessments on average. At that point, the products and the associated standards can become of interest to operators, who might install them for performing assessments in the field or in the network.

References

ITU-T. 2001. Communications Quality of Service: A Framework and Definitions. ITU-T Recommendation G.1000. https://www.itu.int/ITU-T/recommendations/rec.aspx?id=5597&lang=en.

ITU-T. 2015. The E-model: a Computational Model for Use in Transmission Planning. ITU-T Recommendation G.107. https://www.itu.int/ITU-T/recommendations/rec.aspx?id=12505&lang=en.

ITU-T. 2017. Vocabulary for Performance, Quality of Service and Quality of Experience. ITU-T Recommendation P.10/G.100. https://www.itu.int/ITU-T/recommendations/rec.aspx?id=13408&lang=en.

Qualinet. 2013. Qualinet White Paper on Definitions of Quality of Experience (2012). European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003). Version 1.2. March 12, 2013. http://www.qualinet.eu/images/stories/QoE_whitepaper_v1.2.pdf.