​Worldwide Concept Art Survey that Increased Stakeholder Buy-in
A research project to learn what art concepts were popular worldwide and identify regional differences.
Determine popular art concepts to inform a release schedule tailored to each region.
Project Scope
​
-
Client: Confidential
-
Timeframe: 6 weeks
-
My Role: UX Researcher
-
Team: UX Researcher, UX Director, PM
-
Methods: Quantitative survey
-
Tools: Internal survey tool, Google Sheets, Google Slides
High-Level Timeframe
The Problem
The team had created 50+ concept art designs to be turned into 3D-models and sell within their mobile product. They needed to know which designs should be prioritized worldwide, and by region.
​
The team launched a survey the year before, however, it had a high attrition rate due to its poor usability, and needed to be retooled to collect valid data.
Assumptions and Hypotheses
-
"Cooler" concept art would be more popular.
-
Regional differences will be strong enough to warrant tailored release schedules.
-
Based on previous research, the PM felt that art featuring hip-hop imagery would be more popular.
-
Implementing a more usable survey design will reduce participant attrition.
My Role
I was the UX Research lead in charge of:
-
Stakeholder communication
-
Creating research plan
-
Programming survey
-
Analyzing data
-
Statistical analysis (ANOVA)
-
Creating deliverable and presenting to team
Methodology
The PM was located in a different country and was unable to meet with me virtually, so we communicated via email.
​
The initial mobile survey template had several usability issues that likely increased attrition rate (i.e. single page 100+ items with close-ended and open-ended questions, vertical and horizontal scrolling).
​
Immediately I mocked up a more usable survey design (randomly select 20 items from pool of 50, reduce Likert scale from 11pt to 5pt, remove open-ended responses due to power analysis requiring a sample size of 300 for each of the 50 concept arts).
​
I communicated the value an updated survey would provide to the team, they were in alignment and gave their blessing to use my survey design.
Survey Distribution: The survey was distributed by offering participants credits within the product. Due to the team wanting overall preferences worldwide, we did not use a screener survey.
​
Statistical Analysis: The previous survey did not use statistical analysis on their findings, due to the importance and quantitative re-work of the survey, I determined a power analysis of 300 responses per 50 concept art response was necessary to run an ANOVA (around 1500 responses total).
​
Limitation: Ideally I would have used MaxDiff to determine popularity, however the internal tool did not support it.
Breaking down the process
We ended up with roughly 1600 survey responses, enough to meet the power analysis for the ANOVA.
​
To analyze the large amount of quantitative data, I used Google Sheets with custom formulas to speed up the data analysis. This gave me more time to interpret the practical application and polish the deliverable on-schedule.
Major Learnings
"Cooler" designs were preferred worldwide.
Due to the target demographic and age-range of participants, designs that the team felt were "cooler" were seen as more popular world-wide.
​
This provided the team with a short-list of popular designs to launch early.
Regional differences existed, but were not practically significant.
While regional differences arose from the data, the top 5 popular art designs only shifted in order.
​
This meant that the team could reliably prioritize building the top 5 designs.
Hip-hop art concepts were the least popular.
Previous research had been misinterpreted. They team assumed that since users liked hip-hop music, they'd like generic hip-hop art.
​
This indicates that overall preference in one domain (music) does not generalize to all (visual).
TITLE OF THE CALLOUT BLOCK
Impact
The team now had a short-list of top 5 art concepts to prioritize world wide, and regional differences for the next 20. The PM determined a release schedule to align with the concept art popularity.
The team was so pleased with the data validity that they replaced the previous survey template with my design, speeding up future research.
The study results sparked a curiosity to better understand their user base, granting the green-light for me to lead a follow-up in-home field study to gain a deeper insight into user expectations and needs for the mobile product.
TITLE OF THE CALLOUT BLOCK
Lessons Learned
Advocating for improved data quality is worth fighting for.
I took the initiative to propose the survey redesign and it paid off in spades. Initially I was concerned since this was my first project for this client and I didn't want to damage the relationship before it was built, however they agreed with my value proposition and ended up preferring my design. This taught me that although it can be intimidating to suggest a change to the status quo, it can pay off with improved data quality and happier stakeholders.