Improving your research validity

Research validity is incredibly important, without it you risk biasing and even invalidating your research efforts.

Good validity = good research

What is research validity?

Validity refers to the quality of research. In short, the rigor or trustworthiness. Joppe (2000) provides the following explanation of validity:

“Validity determines whether the research truly measures that which it was intended to measure or how truthful the research results are.”

How can it go wrong?

The main cause of poor research validity is caused by the interviewer and can be due to inexperience, poor interview skills or subjective bias that they’re unaware of. Asking leading questions ‘Would you say design A is the best then?’ is especially common as it can take years to perfect this art but leading questions give misleading and false answers, therefore biasing and even invalidating your data.

Can you imagine if you recommend going with design A when actually if the research had been conducted differently the answer might have been to go with design B.

It’s important to remember that users won’t just sit there and tell you their honest opinions, they want to be helpful, they want to please you, so they are highly influenced by the interviewer. It’s the interviewer’s job to be as objective and open minded as possible, not transferring any bias through poorly worded leading questions, tone of voice, facial expressions or body language to the user.

How to improve your validity

Unbias your discussion guide

5W-s

I’ve heard stories from clients of receiving discussion guides full of leading questions. Fail at this stage and your bound to fail at the interview stage. The first step is to ensure all the questions in the discussion guide are fully thought through, flow well and all leading questions re-worded – Try using open questions that begin with the 5 Ws: What, Why, Who, When, Where and How.

Practice like an actor

practice

The more familiar you are with what you’re testing and the questions in the discussion guide, the greater ability you’ll have to stray away from the discussion guide when you need to. This preparation will also free your mind to give you greater ability to construct questions on the spot that aren’t leading. The more your cognitive resources are taken up the less likely you’ll be able to construct a good question, so by reading through the discussion guide multiple times in the days leading up to the research you’ll hopefully be so familiar and comfortable that you’ll be able to conduct some great research!

Watch your performance critically

smile

When you re-watch the research interviews, watch out for moments where you may have influenced the participant. Think about how you could ask the same question but in a different way to get a better response. Depending on the amount of influence you may have had, you may need to remove that user’s response from your data analysis. However, being critical of your own performance and learning from your mistakes is the best thing you can do to improve for next time.

Don’t worry about the odd slip

oops

It’s bound to happen. More at the start and less the more experienced you get. But even with experience, the odd slip will happen. Don’t worry about it too much. With experience you’ll recognise your mistake the moment the words leave your mouth. Don’t let the frustration of your error show to the user, let them answer, then later on re-word the question in a different way to double check their response. If it differs from their previous response then you know you biased their previous response, just take their new answer as their true opinion.

A note from me:

Lisa Duddington CircleThis is a post I recently published on Linkedin Pulse. If you like it, make sure you follow my future posts or connect with me Linkedin

Don’t use one way mirrors for ux research

Do you use one way mirror labs? Do you value research that gets you the best results? Then you might want to re-consider using one way mirrors. Here’s why…

Talking to users is fascinating! It’s something I still love doing despite having conducted thousands of them over the last 10 years. When it comes to location, you can test almost anywhere but there’s one place that I now advise against, and that’s one way mirror labs.

What is a one way mirror lab?

A one way mirror lab (also known as two way) consists of two adjoining rooms with a mirror between them. One room is used to interview people and the mirror functions as a normal mirror from this side. On the other side of the mirror is the observation room where people watch the research taking place, from this side the mirror behaves as a window, enabling the observers to secretly observe what’s happening in the research room.

The negative consequences for research

I’ve used this setup many times and I’ve sat on both sides of the mirror. These are the problems:

Nervous users

As a researcher you should always tell the participant that there are observers behind the mirror. However, if I say to you now, don’t think of a pink elephant, the first thing you think about is a pink elephant. In psychology we call this the Ironic Process Theory or the White Bear Principle and it refers to the human tendency to continue to think about something after being told not to think about it.

So we’re ethically bound to tell people there are people behind the mirror but by doing that their attention is being drawn to it. Many users are fine with this and they’ll forget about the mirror. There are other users who will interview ok but afterwards they will mention how they felt like they were being watched and finally, some people simply do not interview well. They may appear nervous, glance at the mirror throughout, whisper some answers to you because they don’t want the people behind the mirror to hear any negative feedback, etc. And the mirror is a difficult thing for people to get over once they have a problem with it, because it’s such a huge object in the room and therefore a constant reminder.

Ask yourself, wouldn’t you feel uncomfortable knowing there were people watching you behind a mirror in the room?

Positively biased responses

If you knew there were a group of people watching you behind a mirror wouldn’t you be more inclined to give positive responses and to withhold negative opinions? But it’s really important that we understand negative feedback in order to make things better.

Sound leakage

Your observers need to be relatively quiet. I’ve seen labs provide headphones so that observers can turn audio volume up without sound leaking into the testing suite.

When two rooms are next to each other, it’s impossible to soundproof them completely. If the observers next door get quite loud, the sound can leak into the adjoining room. This can be fatal if they laugh and the user hears this. In some labs, the doors don’t close quietly either – this is then another reminder to the participant that there are people watching them.

Noisy cameras

One way mirror labs almost always have cameras that can be controlled in the observation room. These aren’t always silent though. You might be in the middle of a really interesting insight when suddenly you hear the buzz of the camera. Off-putting to say the least and yet another reminder to the user that they are being watched.

Dark, uninspiring observation room where no one speaks

Observation rooms in labs are awful places really. There are no windows and therefore no natural light, the lights have to be turned off (otherwise you can see straight through the mirror) so it’s a dark, dull, uninspiring room to be sat in all day. In one way mirror labs sometimes the observers can be much quieter than in labs without a mirror, because you can see how close the participant is to you.

The problem is these are great setups for observing research, especially focus groups, not UX research. If you have a team of designers observing research, the one thing they’re guaranteed to want to do is sketch, but how do they do that well when they’re sat in a dark room? It’s not an environment that encourages team collaboration, makes a team feel energised, inspired and creative. Conversation and teamworking should be encouraged – now’s the perfect time for the team to collaborate and get to work on designs.

Ironically, no one really observes what’s happening through the mirror!

We spend most of our time watching the TV screens, which give us consistent detail, clarity and control. The glass, for all its glamour, doesn’t always fulfil its worth.

In UX research, the most important interaction to focus on is that between the user and what’s being tested, and in this regard you can’t see anything through the mirror, the detail is through the cameras pointing at what the user is doing. Therefore, the majority of the time, observers are focussed on the tv screen – where the action is. Compare it to UX design…if you want the users attention to focus on something you might give it a more central position, make it bigger, put everything else around it. So when the UI is the most important thing for people to observe, why do labs show this on a small tv screen and give the highest visual prominence in the room to the mirror? It’s crazy!

The solution

The alternative, better solution is to use two rooms that have all the same technology to record and observe the user and their interaction but in the observation room, there are TV screens and no mirror. GDS (Government Digital Services) also use this setup which you can see here. Without a mirror, you’ll get better insights from your more relaxed users and the observation room can now be a creative haven. You can turn up the lights, have natural daylight (windows), have dynamic team discussions and work together on sketches and ideas.

It suddenly becomes an exciting and inspiring workshop to turn user feedback into better designs! And this, is the whole purpose of user research.

A note from me:

This is a post I recently published on Linkedin Pulse. If you like it, you can follow my future posts or connect with me Linkedin

Which do you remember? Computers of the past

Attending the Manchester Science Festival was incredible. Not only were there computers, mobiles and televisions from decades gone by, but they were all in full working condition, which meant I could re-live some of the highs (space invaders) and lows (error messages and recovery) of my first experiences with computers.

Did you have any of these computers? Did you play any of these games? Which was your favourite? Tweet me

BBC Micro: Chuckie Egg (1983)

I don’t remember this computer or Chuckie at all, but looking at the launch date of chuckie (1983) I was only a toddler so I’m pleased to say I’m too young to remember this one! Following Ricardo’s enthusiasm, I had a go at Chuckie and it was really difficult at first! Having to remember which letter or symbol did what took a bit of time to get the knack of, which is of course why UX is so important. But this game was great fun once I’d remembered the keys. I’d definitely play this. On another note, the tactile feedback from the keyboard felt hugely satisfying. You just don’t get deep key presses from modern day keyboards, and although it makes them slimmer and faster, you don’t get the cushiony, bouncy, weighty feel, which is really satisfying.

BBC Micro

Amiga – Lemmings (1985)

This is what I remember as our family computer. This and the ZX Spectrum. And I remember playing Lemmings all the time – I still think it’s one of the best games ever! But the game I used to play all the time was called Dungeon Master. Did anyone else play this? (please tweet me if you did!) The game started out in a chamber with portraits on the wall and you chose who you wanted in your team. Then you entered a maze and had to fight monsters, find food, complete tasks, find potions to make your way through the levels. I still remember my favourite character – he had a black cloak, red eyes and his name was Gothmog.

Commodore AMIGA

IBM

Ah these are the computers that we used at school. They never did what you told them to do and there were no end of hands going up in class for help with lost work, floppy disks that wouldn’t save, etc. And they took up the whole desk so you had to balance your school book on the edge of the table or on your lap. I don’t have particularly fond memories of this one!

IBM

Toshiba laptop

Toshiba Laptop

Oh no! It’s growing up with UI like this that drove me to get into usability. I remember constantly thinking, I know really clever people design and build technology, so why do they make them so difficult to use? Of course I know why now – if you’re too much an expert in something it’s difficult to look at things from a new users perspective. This error dialogue is a classic. Just look at the choice of colours used too – really poor readability on the command text at the bottom.

Toshiba Laptop Error

Floppy disks

There were lots of kids at the science festival and I wonder if any of them thought these were printed versions of the Save icon. I find it interesting that we’re still using this as a Save metaphor despite the fact that the true meaning is lost on many young people. However, they have learnt that it’s the Save icon, which begs the question should it really be updated or should it stay as a floppy disk, bearing in mind that although youngsters don’t understand what a floppy disk is, they do associate that icon with Save functionality. It’s a tough one.

Removable media

Other computers – do you know any of these?

Commodore 64

Dragon 32

Commodore PET

Macintosh

Google: Over 50% xmas enquiries will be mobile

Yesterday at the London e-commerce expo, Google’s Martijn Bertisen, spoke about e-commerce trends for 2015. Mobile is set to be BIG business and this christmas, over 50% of christmas retail enquiries are likely to come from smartphones. 2015 will be: The year that mobile takes over. The year of wearables – watch out for Google Glass. We’ll see this coming into the retail experience. Martijn said:

If there’s one message I wanted everybody to take away, it’s that what is increasingly enabling e-commerce is mobile devices. We’ve shifted from the desktop era into the mobile era and it’s still being underestimated… it’s where consumer behaviour is.

ux_advert