Lately I've been playing with fast, cheap, early testing methods for prototypes, wireframes and not-yet public pages.

The challenge here is of course that you can't run big numbers. You can't do A/B testing when you have no traffic.

Methods

First-click testing.

Put up a wireframe or a mockup and ask people to perform a task and record just the first click. You might think that this would not tell you a lot, but extensive testing on major sites (lots of government sites) for over a decade has shown that first-click success is the best predictor of overall success. I forget the exact numbers, but on large government sites there was a huge gulf between overall success of those who got the first click "right" and those who didn't.

In my own tests, we have sometimes found significant differences between people recruited through the testing service and those we recruit off our own Facebook page. Generally our people, which skew older and wealthier, do not do as well as the people who take tests on Usability Hub.

- Preferred provider: Usability Hub. Mostly because you can buy credits at $1 per tester, no subscription. You can also take tests and earn credits.

I strongly recommend taking at least a dozen tests before designing one. At least half of the tests you will take are utterly stupid. Stupid tests are mostly of two types
- Preference Test - you are shown two designs and asked which you prefer and you look and look and can't see the difference
- Five Second Test - you are shown a design for five seconds and then asked "How long was the money-back guarantee good for in the disclaimer"

Tree Testing

This is for those times when you are still putting together the IA and/or nav for a site. It removes all design elements and focuses just on organization. You input a tree and then give people a task. A few of you here did my first try at a tree test. It's a quick sanity check on whether your nav is minimally comprehensible.

In our case, I knew our nav was horrible, but nobody else considered it an urgent problem. The great thing about testing is you have discussions that go like this:

You: "I think it's a problem"

Them: "I don't think people have any problem understanding those terms."

You: "Okay, while I had 30 people try to find X. Zero found it first try. Six found it with 1-3 backtracks. 23 never found it."

Them: "Sure, but that's just in a tree without all the helpful design elements that will guide [read: confuse] them. I think in the live site it will be fine."

You: "Okay. I have data. You have an opinion. Data beats opinion. Better data beats worse data. Show me equal or better data and I'll concede."

- Preferred provider: Optimal Workshop Treejack - the reporting tools are fantastic.

Live User Testing

If you don't have live users you can use for in-person testing for whatever reason, you can use UserTesting.com but it gets pricey quickly. Once you've done your first 10 videos of live testing, it's $99/pop. So if you want to watch 5 users on your site, that's $500. However, if you want a super quick view for first-pass testing, they have a less well-known service called Peek that's free for 5 min max

http://peek.usertesting.com/

Sort of like Jing is free and the upsell costs money. Same idea. If you're still testing something simple, you can get some rough tests done here.

Prototypes

You can't do live testing unless you have a live site, right? Not exactly. A couple of services let you upload static jpegs, identify hotspots and have those link to the next jpeg, even transitioning with animations and so forth.
- http://invsionapp.com is the famous one that most people will know. Our design firm has been using this on our current project.
- https://marvelapp.com/features/ is a free alternative. I haven't actually tried it yet, but it looks amazing - everything Invision has. I'm not sure how they'll make enough money, but for now, jump in.

If you have wireframes or a comp, you can use this for your "website" for video testing

Peek + Marvel = completely free live testing on a functional prototype. This makes me feel like I'm in The Jetsons or something.

Click Mapping and Scroll Mapping

Once a page is live and has some actual traffic, I find that click mapping and scroll mapping show you results much quicker than waiting for lots of data.
- Crazy Egg is the obvious big player for both and their clickmapping has great reporting.
- SumoMe does scrollmapping that is actually way better than Crazy Egg and completely free.

Your Experiences?

Anyone else doing any testing? What's working and what's not?

Great post, thank you. I need to expand my current method, the "Travoli likes it" test.
$1 per tester isn't bad.

bloody hell.  Great post. I am way behind on this stuff.

Thats brilliant, thankyou ergophobe.

Today's test was a click mapping test using Crazy Egg on a landing page for an email. We expect low response (it's a purchased database, not "our" people) and we are hoping to add them to our database either through getting them to purchase or through signing up for special offers.

So far we only have 267 clicks, so a very small sample, but Crazy Egg lets me look at every click on the page and segment them in a couple dozen ways. So one of the things I was curious about was how click location correlated with time to click.

In the first 15 seconds, almost all clicks are on the button to buy/get a quote. From 15 seconds to 45 seconds, it's split between people clicking on the buttons and people clicking through the image carousel. After 45 seconds on page it's almost entirely clicks to the buy/quote button.

I'm not sure what that tells me in terms of actionable info, but it's curious.

The more interesting part is that the number who scrolled to the bottom of the page was quite high, which makes me think we might have done better with longer content.

A substantial number of clicks were to the footer navigation, which tells me that they got to the bottom, were still interested, but the page itself didn't deliver what they needed.

With traditional analytics, I would know which pages they went to, but I wouldn't know
- where they clicked on the page that kept them on page (carousels, calendar dropdowns)
- which link leading from this page to the next one they used (i.e. Google Analytics overlay shows what percentage of users went to a given destination, but it doesn't tell you which link they used to get there).

ergophobe gets my vote for Post of the year.

2nd place would be the "Travoli likes it" test. 

Quote from: Mackin USA on November 22, 2015, 03:49:35 PM
ergophobe gets my vote for Post of the year.

Thanks!

I'd like to hear what testing others are doing. It really ain't that hard. Just do it!

we do in house live-user testing - its incredibly interesting to watch and throws up all sorts of info - for me its substantially more useful than most other types of testing because we start by asking people to do something (research a holiday, buy a present, etc) and that generally means we see their search behaviour and how often they loop back on themselves, as well as a straight task on a website.

BUT if thats not practical and/or there's no budget any type of user insight is better than none imho - people behave so differently than you expect them to, and different groups of customers do entirely different things - I'm watching this one with interest as we do use lots of card sorting tools etc, split testing software etc, but there are always new and better ones out there


ergo - you can just label up each link from an email specifically (eg; segment header, body, message, footer links using one of the GA tags so you know if the homepage link came from your logo, text link, footer etc). Its not perfect but its better than just adding a single email tag to every link and then not knowing the click location?

Quote from: Gurtie on November 23, 2015, 12:37:05 PM
we do in house live-user testing

The situation I've been encountering lately is where we're in really early stages and want to make some decisions fast. We have done some live user testing as part of that process, but what's been interesting is how much insight you can get without that.

Quoteergo - you can just label up each link from an email specifically

That's a good idea - we have not been doing that with email. I'll have to float that with the team. We try to tag most other things.

In the case I mentioned, though, we knew for certain which link they were clicking in the email because there was only one link. This is a series of emails - about 6 over three months - that go out to customers at a sister property who is partnering with us and giving us a short blurb with a link in their newsletters.

The question is where are they clicking on once they arrive on the page and how far down are they reading.

I'm with Trav.

"If I had asked people what they wanted, they would have said faster horses." Henry Ford (allegedly)

Then we do some user testing and then split testing when live, never really highlighted anything of consequence.

Bonus is that it drives Gimpy crazy.

>> The question is where are they clicking on once they arrive on the page and how far down are they reading.

custom segment and then look at the traffic flow of that segment? And then something like this http://scrolldepth.parsnip.io/ ?

It doesn't replace other tools though - I like mouseflow, but its not free - its $15 for 1000 sessions though if you're quick to review and analyze, so not bad.

Gurtie - that's awesome. I had no idea you could get GA to record scroll depth.

>>mouseflow

Looks like a step up from CRazy Egg, but I'll have to read up on it. Nice comparison here
https://webanalytics.knoji.com/kissmetrics-vs-crazyegg-vs-mouseflow-heatmapping-tools-compared/