Pro tips for running a pilot or beta trial

The best way to test whether a product or service is ready for prime time is to release it to the wild. Some people call this a pilot trial or pilot test. Others may call it an alpha or beta trial.

Whatever we call it, the idea is the same. You present the product or service to the customer, on-board them and teach them how to use it. Then you leave them with the offering to experience it on their own for a period of time. Throughout this period, and also at the end of the trial, you keep in touch with the customer to collect their feedback. This feedback can then guide product design and development iterations.

This article outlines some pro-tips on when to use this technique, and how to use it effectively. If you plan and implement it well, this will become an indispensable tool that will help you build great solutions to customer problems.

Before you begin: What beta trials are good for 

Beta trials are great for some research goals. There are other techniques that are a better match for other goals. The key is to pick the right research technique that yields the best results for your current research goal.

Beta trials are a great fit for the following research goals.

• Testing whether the product or service works as advertised in the intended environment for an extended period of time
• Testing whether users will continue to use a service that requires continued engagement to justify a recurring revenue stream
• Testing whether economic buyers will pay to use the product or service (which validates both their pain points and their willingness and ability to pay)

This technique is not a great match for discovery research that leads to persona development, or feature set prioritization.

• If you are building personas, use interviews, shadowing, diary studies and immersion instead.
• If you are prioritizing features, use card sorting or conjoint analysis instead.

Market selection and persona development comes before beta trials

If your persona is not set and your segmentation and BHM are soft, work on that first. A pilot or beta trial, more than any other qualitative research techniques, takes a lot of time and resources to run well.

You cannot design a good pilot study without having rock-solid segmentation, market selection, and personas. This is because the persona is how you can guide the subject recruitment process.

You need laser focus in experimental design. You also need to tightly control who you do your beta trials with.

For B2B: Identify and engage your entire Decision Making Unit (DMU) - not just the end user. 

For B2B enterprises, you must build personas for all of the key stakeholders in your decision making unit. This then helps you engage them in your pilot.

For instance, you may find that there are more than one end user who have very different needs and wants. You want to find that out before you invest in building out a scalable product platform. You may also find that you need to win the support of somebody with veto power. You might then include a debrief meeting with that person to show them how your company's solution is delivering value to their company in real time.

Explicitly design and plan the beta as if you are designing a scientific experiment. 

Key parameters to define for any pilot study include the following:

• Research goals (please be granular and specific)
• Hypotheses you wish to test and success metrics for validation or invalidation
• When you want the pilot to start
• Number of people / companies you need to include in the pilot
• Number of devices (if needed) you have available to run the pilot with
• Amount of time it takes for your product or service to show its benefits
• What it takes to onboard your pilot customers; what it takes to do the care and feeding of the customers (because you will be doing a concierge MVP - i.e. using a high touch approach to talk to the customer frequently during the pilot because your product is not complete)

Plan the scale and scope carefully 

The number of people depends on the type of venture. B2C - at least 5, do them in batches of 5. B2B - at least 3 companies.

Either way, you will need to understand the workload to support the beta trial while it is running - and plan the human and technical resources to support your beta customers with the best possible service and experience.

Sometimes, limited headcount and resources may mean you will need to run a 30 person trial in batches of 5 - you will still get the full set of data, only over a longer period of elapsed time.

Pilot the pilot

Before you run the pilot, you should test the product or service in the target environment for the duration of the beta with one or two volunteer subjects to make sure you find and fix all the show-stopping problems before you launch a trial with a larger cohort of test subjects.

Just know that the purpose of this pilot of pilot phase is to find and fix bugs. While you will still collect data about product usage, you need to discount the results as they will be muddled by any problems that you find and fix during this phase.

Use the same trial design for the entire cohort of test subjects 

While you may do "rolling admissions" to manage your own workload, it is always good to design the study with a predetermined number of test subjects. You can then design your experiment to deliver the same benefits and experience to these test subjects. This holds true even if their pilot durations are staggered for load balancing reasons.

You do not want to change protocol with each person unless you are in the beginning of your exploration and you are using the pilot to define product requirements.

Paid pilots can be used to validate purchase intent 

Unpaid pilots are good for testing product functionality. You may decide to pay an honorarium if you have the budget - think of it as a focus group honorarium.

Paid pilots get you the real data about willingness and ability to pay and generally gets you much better engagement.

Engage the test subjects in conversation for data collection

While there are many ways for scalable data collection via on line surveys, chatbots or other techniques, nothing beats a conversation with your target customer.

This is especially true for the first 5 or 10 pilot subjects. In these early days, you don't know what you don't know. You need open ended dialog to capture the feedback.

For data collection, the minimum way is to call your customer after 1 day, 3 days, 1 week, etc. Prescheduling these touch points is a great way to manage the cadence of data collection.

Supplement data collection with scalable, on line tools

Once you get a good handle for how the offering is received and used in the field, you can consider supplementing your data collection process with on line surveys, photo essays, voice diaries and the like. Just know that compliance could become a problem.

At the end of the day, nothing beats an interactive conversation. This is especially true in business to business enterprise settings, where a verbal debrief may be the only way to capture feedback.

This article first appeared in the ConceptSpring blog by Elaine Chen.