M364 Block 1, Unit 4, Activity 1

[Please complete the assignment on page 337 of the Set Book]

1. Reconsider the HutchWorld design and evaluation case study and note what was evaluated, why and when, and what was learned at each stage?
2. How was the design advanced after each round of evaluation?
3. What were the main constraints that influenced the evaluation?
4. How did the stages and choice of techniques build on and complement each other (ie triangulate)?
5. Which parts of the evaluation were directed at usability goals and which at user experience goals? Which additional goals not mentioned in the study could the evaluations have focussed on?

Reconsider the HutchWorld design and evaluation case study and note what was evaluated, why and when, and what was learned at each stage?

There were several rounds of evaluation. The design was evaluated in terms of usability and scope. Evaluation of the first two prototypes, including the first series of usability tests, led the team to learn that there were problems with the initial scope of the project, so they changed the scope, limiting the 3D virtual world to just Reception, but adding support for asynchronous messaging, games, and locating approved medical sites. When the application was redesigned as a portal it was re-tested and further refined, allowing the team to learn about more specific usability issues. One goal of the new testing was to ensure that the system supported multi-user interactions. 

In general evaluation seems to have started with requirements and scope, then moved on to usability and user experience, in other words from whether the project had the right aims and goals to whether these goals were being reached.

How was design advanced after each round of testing?

There were two main rounds of testing. The first round of testing led to a fairly radical advancement of the scope of the design, from an intensively 3D virtual reality experience to a portal with support for email. games, medical queries and rather limited 3D functionality. The second round was  more focussed on detailed usability testing and led to incremental improvements in the design.

What were the main constraints that influenced the evaluation?

The team found it difficult to arrange testers rapidly for reasons inherent in their choice of user group - the potential users were sick and had limited energy and availability.

How did the stages and choice of techniques build on and complement each other (ie triangulate)?

The evaluation techniques ranged in nature from quite open and general (interviews, focus groups) to very specific (scripted usability tests). The more open evaluation techniques provided aims and scenarios which provided the context and goals of the more specific evaluation techniques

Which parts of the evaluation were directed at usability goals and which at user experience goals? Which additional goals, not mentioned in the study, could the evaluations have focussed on?

The initial interviews and resulting scope analysis were largely expressed in terms of user experience - one over-arching requirement, for example, being to reduce the social isolation of patients and care-givers. The formal tests were more focussed on usability, but some of the results are again best described in terms of user experience - for example, the problem with the purely synchronous early prototype never reaching critical mass is most simply explained by the fact that it is no fun being in a chat room on your own. Similarly there are UX explanations for the patients' preference for online games (fun), the search for recommended medical sites (helpful) and email (emotionally fulfilling). In addition to the carefully scripted usability tests in each round, there was a short questionnaire which asked both usability and user-experience type questions.

I think that the project could have focussed on the user-experience goals of motivating and satisfying, since these would highly relevent to socially isolated users suffering from energy-sapping conditions.

M364 Block 1, Unit 3, Activity 9

How user-centred was the approach taken by Tokairo? Start by listing the main stakeholders - beneficiaries, decision makers, gatekeepers and workers - and discussing their roles in the development process.

Then analyse Tokairo's approach against each of the five principles of user-centered development given on page 286 of the Set Book

The main stakeholders in the project were, at a corporate level, Excel plc as parent of Tankfreight and Shell, the project's client. Shell were responsible for allowing Tankfreight and Tokairo access to their depot. Tankfreight were probably the main beneficiary since they commissioned the project, and automating the delivery reporting process should reduce costs and increase accuracy.

The more immediate stakeholders were Tankfreight project management (Hugh Rowntree and Rachel Darley), Tankfreight's account manager for Shell, the driver foremen, trade union representative and the drivers themselves.

Hugh and Rachel were gatekeepers to the drivers and Hugh was presumably a decision maker in commissioning the project, and, together with Rachel, signing-off design options and prototypes.

Tankfreight's account manager may have been a gatekeeper for access to the Shell depot, and would have had some involvement in high-level project decisions.

The driver foremen and union representatives are not only gatekeepers to the drivers but possibly themselves users of the system, as workers, and possibly beneficiaries, either as users or because it makes the drivers happier.  

The drivers themselves are the primary users of the system, possible beneficiaries (if the forms are more accurate and they get fewer forms being returned to them) and are involved as workers.

Presumably others are also involved, either as workers or beneficiaries - those who used to enter the drivers' forms manually, those who manufacture and sell the kiosks, and those who service them.

Now let's evaluate Tokairo's approach to the five principles of user-centered development.

Users' tasks and goals are the driving force behind the development

Tokairo's initial input comes from Hugh Rowntree and Rachel Darley, who know the drivers, their tasks and their environment well, and Tokairo had already had access to the users before the project even started. They appear to have a good understanding of users' tasks and goals. But satisfying the drivers' tasks and goals is probably a necessary rather than a sufficient condition for the project's success, the final criterion presumably being that the system reduces costs and increases accuracy. 

Users' behaviour and context of use are studied and the system is designed to support them

As part of the system audit, "Treve actually went initially to the oil terminals and depots. Treve met the drivers and the driver foremen". The team appear to have a detailed understanding of what the drivers do, and of where they do it, in the cab and at depot reception, and it is clear that design decisions directly reflect these factors.

Users' characteristics are captured and designed for

Once again the team appear to have a detailed picture of their user group, of what they have in common (being literate, non-academic males, short of time, primarily interested in earning a living and going home) and where they diverge (level of interest in computers). And again, this is clearly reflected in the team's account of how they made design decisions. 

Users are consulted throughout development from earliest phases to the latest, and their input is seriously taken into account

This principle was defintely observed during the requirements phase of this project. It is clear that Tokairo would normally prefer to consult users during the design and implementation phases but felt that there were specific reasons they shouldn't do so in this case, although the form did get some early testing, which is a kind of user-consultation.

All design decision are taken within the context of the users, their work, and their environment

This principle seems to have been observed very thoroughly. In fact it seems that the team ascribe the success of the project to this factor.

Given that the project followed at least four of the five principles, and that the remaining principle was partially missed for specific reasons rather than lack of interest or lack of user focus, I would say that the approach to this project was quite highly user-centered.

M364 Block 1, Unit 3, Activity 8

Describe the approaches taken to user involvement in the Tokairo case study and discuss these using the issues you identified in Review Question 5 [List the (eight) issues you need to consider when deciding on the appropriate level of user involvement]. What alternative approaches might they have taken? You should refer to the case studies in Section 9.2.1 of the Set Book as examples.

What advantages and disadvantages might these approaches have had?

The set book lists the following issues that needed to be considered when deciding the appropriate level of user involvement:

Can you identify the users, or are they the open market?

In this case the users were already identified, as being the drivers.

How many users are there? Tens or Thousands?

We don't know exactly how many users there are, though common sense suggests it might be tens or maybe low hundreds. In any case, given that the users are, by the nature of their job, never in the same place at the same time, it's too many people for them all to be consulted easily or cheaply.

How long is the project expected to take?

Again, we don't know how long the project was estimated to take or how long it actually took, but in terms of user involvement the answer is probably that it was going to take too long to consider co-opting a real user for the duration, but not so long that any significent user practices or requirements would change before the project was completed.

Do you want a major contribution from users, or just advice and guidance?

There seems to have been a clear assumption, initially from the client but accepted by Tokairo, that the real users' main contribution should be to the requirements process, to alpha-testing the form and to beta-testing the entire system. Rachel, the systems analyst, acted as a proxy user for reviewing the design, but her contribution (apart from suggesting that the buttons be colour-coded) seems to have been mainly evaluation. 

How many users do you want involved with the project?

The team didn't want all users to be involved but consulted user and stakeholder representatives widely during the requirements phase. For the main design options, design and evaluation activities they mainly used Rachel, the user proxy. The notable exception is that they tested the form design on one driver area before beta-testing the entire solution, presumably having identified this as being at higher risk of failure (eg due to the driver's environment when filling in the forms) than the kiosk design.

Is consistency of user input important?

There was no continuous user involvement apart from that of Rachel, the user proxy, so consistency of user input does not seem to have arisen as a question involving the drivers. The consistency of Rachel's involvement is likely to have been helpful to the design process.

How important is familiarity with the system?

Given the general non-involvement of the drivers, familiarity with the system under development does not seem to have been a significent issue. Rachel, as the sole proxy user, was familiar with the system as it was designed, and this would have helped her contribution.

How important is it for involved users to be in contact with the user group they represent?

As there does not seem to have been any change in the drivers' environment or practices during the project, this was probably not an issue for Rachel with respect to the drivers. 

Comparing this approach with the Microsoft or OU case studies in the Set book, Tokairo could have co-opted a driver to work with the team, presumably part time. This would have been useful if the team had worries about the correctness of their scope and requirements (as the Open University appears to have had) but would have involved disrupting the availability and working practices of the driver in question, and would have risked inappropriate feedback due to lack of appropriate user motivation. In fact the team seem to have regarded this as tightly scoped exercise with well-understood requirements (especially given the comprehensive requirements phase), so the advantages of this step would have been low. 

They could also have conducted workshops and prototyping sessions, as the OU did. This would have had the advantage of reducing the risk of "requiring more changes during the prototype ... and even [the] pre-live stage", but it would have been disruptive to operations, the drivers and their management, and premature exposure would have risked "destructive feedback".

They could also have performed lab-based usability testing, as Microsoft does. This would have reduced the risk of the overall system proving unacceptable during beta-testing but the team seem to have felt that the risk of this failure was not sufficiently high to warrant the required level of disruption and expense. 

M364 Block 1, Unit 3, activity 7

Look back through the previous sections. Describe Tokairo's approach to design. How these map  onto the ID activities and characteristics of the ID process?

Having established the user requirements, Tokairo bought their own experience into the design process. Some of the major decisions - such as the choice of touch-screen input and the choice of a "big lottery ticket" form - were made rapidly, after consideration of alternatives, but not necessarily after much iteration. In the case of the basic form design, rather than the ID activity of "Developing alternative designs" they evaluated alternative existing designs, to similar effect but presumably at rather lower cost. The use of the team's professional experience seems to have provided a similar short-cut for the kiosk design, and might be seen as a greater difference from the ID method. 

The form design was tested in the wild (in "the most militant driver area they could find") and maps to the ID activity "Building interactive versions of the design".

There was explicit evaluation of both the Kiosk and Form designs and this corresponds to the ID "Evaluating designs" activity.

The ID characteristic "focus on users" can be seen in the whole requirements stage, and in thepre-beta  field testing of the form, though less so in the implementation of the kiosk design where the client preferred to provide a user proxy.

The "Specific usability and user-experience goals" ID characteristic is arguable "absent but unnecessary" due to the short lines of communication and clear implicit focus in this area.

And the ID characteristic of "Iteration" can be seen in mainly within activities, in the refinement of the kiosk and form designs based on the feedback from users, user proxies and field testing.

M364 Block 1, Unit 3, Activity 6

Look back through the previous section and list the characteristics of the approach that Tokairo took to the requirement activity. How do these map onto the ID activities and the characteristics of the ID process (as discussed in Section 6.2.1 starting on page 168 of the Set Book)? 

What is their attitude to stakeholders?

Tokairo have a methodology with a first stage being the Site and System Audit. They try to identify "logical group[s]" of users by business function, who have characteristic concerns and requirements. Having already talked to the drivers and driver foremen, they went on to talk to the systems manager and systems analyst, and union representatives. This maps to the ID "needs and requirements" activity, and the ID characteristic of being "focussed on users".

The ID method has the characteristic of being "iterative". The Tokairo approach in this case appears to involve more iteration within the requirements activity than between activities when compared to the ID method, but they describe this as being due to the success of the requirements activity, and both are in fact present.

The results of the initial requirements activity were communicated verbally rather than as written deliverables, and the usability or user experience goals were probably set implicitly, in terms specific to the project (eg "suitable for well-motivated, literate drivers with big fingers who are in a bit of a hurry") rather than in the more general terms of the ID method, which may represent a departure from the ID characteristic of "Specific usability and User Experience goals".

The Tokairo attitude to stakeholders is open and respectful - because "if you just talk to the managers ... you will have a whole lot of surprises".

M364 Block 1, Unit 3, Activity 5

This activity builds upon Activity 6.4 on page 182 of the Set Book. Compare the two electronic calendar designs using the following usability and user experience goals:

• Efficiency: In particular, which design enables the user to find a given date most quickly?
• Learnability: which design will be easiest to learn?
• Aesthetically pleasing.
• Enjoyable.

Which design do you prefer, and why?

Efficiency: The desktop design appears to be very inefficient for finding dates since you have to "leaf through" the diary, so the time required to locate a date page is proportional to its distance from the currently opened date. The phone version requires keystrokes instead of mouse clicks but since you're entering a date the number of keystrokes is not large, and does not increase with calendar distance. The desktop diary is probably more efficient for viewing dates within the same week, the phone diary is probably more efficient for navigating to more distant dates.

Learnability: The desktop diary is probably easier to learn, especially given its limited date navigation and graphical rendering, but the simplicity of the phone diary makes it no harder to learn than necessary.

Aesthetically pleasing: The desktop diary is able to use a clean, helpful graphical rendering which offers hints about its further functionality (tabs for the notes and address book sections). The phone diary is written for a visually restricted environment, which means that any comparison is almost as much a comparison of the environments as of the designs. So I would say that the desktop design is more aesthetically pleasing, while noting that this more helpfully addresses the question "Which design would a user prefer?" than "Could the phone design be made more pleasing?"

Enjoyable: The cross-referencing possibilities of the desktop design might generate some enjoyment. Assuming we can click on "John" in an appointment, get taken to his address page, and then see all his appointments (in the same way that we can see all people listed for an appointment), this cross-referencing, together with the ability to read notes from previous meetings, might give a certain bookish enjoyment. The phone design offers less prospect of discovery, so I think scores even closer to a neutral zero.

Assuming that both designs are equally available (for example, that I have gone out and bought an iPhone) I would prefer the desktop design for its greater efficiency when dealing with nearby dates, its slightly greater enjoyability and its better aesthetics,  though I might find its lower efficiency for distant dates frustrating.