Paul Terry on building a culture of optimization

The Optimization Summit is off to a fast start and I sat in on a terrific session by Paul Terry today. Paul is a Senior Manager of Optimization at PRIMEDIA, Inc., the parent company of Consumer Source, Inc., a national publisher and distributor of free print and online consumer guides for the apartment and home industries.

His focus is on testing a network of sites that includes ApartmentGuide.com and Rentals.com, among others. He was graciously transparent with their approach to optimization internally.

Paul explains that the organization has been in a rapid shift to digital over the past three years with big investments in SEO, SEM and online media. Testing and optimization have become fundamental to the way the company operates online.

The primary goals of the sites are lead conversion and maximizing advertising revenue. His team is in a constant state of optimization, working in an agile mode, having anywhere from 2-8 tests going on each site at a time.

There’s a belief within Consumer Source that almost anything should and can be tested (virtual high five, Paul). He walked the audience through a real example of how they approached their apartment listing page (See the actual live page)

This is a critical page which has been tested many times over, and continues to be a focal point of optimization.

The prior call to action on this page was “Contact Property.” In approaching the test, Paul’s team asked themselves “why does someone want to contact a property?” They looked at the actual conversations that were happening offline and found that a large percentage were asking about availability. This generated the idea to test a call to action that had “check availability” or “check current rent specials”

What a fantastic example of doing user research to uncover the real motivations and goals that can be aligned to the online channel.

“We’re in the lead generation business, which means we’re in the conversation business,” said Paul. “Our goal is to start conversations so our sales folks can follow up and continue them.
Other winning test variations on this page included

• Changing the location of the lead form to be front and center in the top left position
• Consolidating the first and last fields to one field called “your name” to reduce a field.
• Changing the “Submit” button to “Send”

They have also been testing how to display photos. He notes that photo interfaces on property listings are inherently clunky (see: Craigslist) and unusable, often resulting in slow download times that can drag on conversion.

They tested many options and ended up with an innovative winner that had what they call a community carousel display (can’t find a screen shot of this yet).

This variation had huge impact, resulting in 21% more visitors that send a lead, 26% more leads per visit and a 15% increase to request a credit score. Also, by placing the carousal across the fold (Paul called it a “fold buster”), they saw that it helped draw people down the page further and engage in deeper content.

He said one of the hardest parts of testing is the approach to analytics. He spends a lot of time figuring out the natural cycles of the duration of the tests and how it lines up to the overall cycles of the business (seasonally, and otherwise)

He also spoke to the value of segmentation, which hasn’t been a huge topic yet at the conference, but I expect it to come up more in later sessions. Segmentation is business critical according to Paul (another high five).

He’s found success slicing and dicing the segments based on folks who succeed, folks who fail and different browsers. He noted that Firefox was their worst converting browser, although didn’t have an answer why, or what he was doing to change that.

The testing tool Paul uses (SiteSpect) handles all of the confidence rating and statistical significance, but it’s a huge consideration for how they approach understanding testing outcomes. In general they like to see a confidence of 90% or higher.

I liked how he frames these rapid tests against a big picture. “We always try to understand the story behind the data” he says. “Establishing a framework of common metrics” allows them to look at the big picture to see if small tests have larger impacts (positive or negative) with the overall goals and KPIs.

He highlights an under discussed issue that, in testing, you’re often robbing performance from one area of your site in order to gain lift in another. Case in point, by emphasizing something on a homepage, you are putting less emphasis on something else.

Keeping the tests grounded in the big picture helps the business continue to move in the right direction. As testing becomes more organizational-wide, the tradeoffs can be more difficult to manage with groups fighting for their own lift at the expense of others in the company, or even to the detriment of the overall bottom line.

I love how Paul and Consumer Source are involving many folks inside the company to develop a culture of optimization that goes beyond running ad hoc tests with a few folks. It was a great session and inspirational for organizations building out their own in-house optimization efforts.