Wayfair Tech Blog

It started as a proof of concept prototype in the fall of 2010. The idea came from a meeting where we were discussing the porting of our storefront codebase from classic ASP to PHP. One of the discussion points was how to avoid simply porting the same logic from one scripting language to another, but rather finding ways to move some of that logic to other more suitable platforms, including service oriented solutions. The little program was written as a self-hosted WCF service written in VB.NET running on my Windows XP box as a console application. It implemented a RESTful API service that returned the number of products in a customer’s shopping cart. Really simple and modest in scope, the thing worked like a charm. I made a presentation about it during one of our Wayfair Engineering Lunch and Learn sessions about a week later. So far, so good.

Wayfair Tech Blog

The 24-Hour News Cycle

A few weeks ago, we celebrated Inc! Magazine’s great cover story about us, including an internal poll of our favorite item from the photo shoot (results: tie between the purple dragon and giant giraffe). Unbeknownst to us, however, the story was later picked up by Yahoo!’s news feed on April 10th and posted to the scroller on their homepage. This is where our story begins…

Wayfair Tech Blog

Software Adaptations - Wayfair's Partners

You would think data replication is a piece of cake these days given all the advances in database technology, and that’s true for the most part when you’re dealing with databases of the same type, but when you have to replicate parts of your product catalog with other companies, things get a bit tricky. At Wayfair Engineering we’ve figured out how to make it happen by creating a great software solution that keeps our retail partner operations working like a well-oiled machine.

Wayfair Tech Blog

Eliminate Implicit Conversions

At Wayfair, we are never done. And the DBA team here is a true example of it. We are constantly looking to improve performance and we rigorously tune our databases on a daily basis. We are always looking at ways to have our queries run faster – by maintaining indexes, optimizing queries and procedures, creating any missing indexes based on query usage, generating statistics on currently running queries, and filtering out queries with top CPU usage, among other improvements. Of late, we’ve been trying to eliminate any implicit data type conversions that happen at runtime. Implicit data type conversions come with cost, especially when the conversion is performed at the column side of the query – not the literal side. We have had scenarios where for high volume processing jobs (processing millions of records) we had index scan execution on queries due to implicit conversions. A simple demonstration of an implicit conversion is: WHERE a.OrderID = b.OrderNo; a.OrderID being varchar(30) and b.OrderNo is nvarchar(30). Here the execution plan would do an implicit cast to nvarchar(30) and would perform an index scan operation on the millions of records – with you waiting endlessly for the query or job to finish.

Wayfair Tech Blog

Recommendations with Markov Clustering

Our story begins in Holland in 1997, where a researcher named Stijn van Dongen, who is pretty good at Go, has a 5-minute flash of insight into modeling flows with stochastic matrices. He writes a thesis about it and makes a toolkit called MCL with a free software license.

recommendations_with_simple_correlation_metrics_article_image

Wayfair Tech Blog

Recommendations with Simple Correlation Metrics on Implied Preference Data

When you sit down to write a recommendations system, there are quite a few well-practiced techniques you can use, and it's difficult to know in advance how well they are going to work out when applied to your data. Thanks to the Netflix prize, which was initiated in 2006 and awarded in 2009, a lot has been written on recommender systems for the Netflix data set. If you happen to have a product catalogue similar to Netflix's (those movies from the 60s are still being viewed and rated), and your users happen to have scored it with a 5-point explicit ratings system, there are some awesome advanced techniques and frameworks that you can take for a spin. Does that sound like you? Show of hands? I didn't think so. Our data is certainly nothing like that.

Wayfair Tech Blog

Puppet provider for python library installation

We run a python/Tornado-based recommendations service behind the scenes at Wayfair. As part of our code deployments, we need to install various third-party libraries to our Tornado servers. The python tools that do this kind of thing are a bit half-baked, so we paper over their inadequacies with puppet.

Wayfair Tech Blog

MySQL Virtual Directories for Pure-FTPd

Here at Wayfair, we have thousands of suppliers we work with in order to provide our products to our customers. To automate the bulk of these interactions, we use Electronic Data Interchange (EDI), so that we can trade documents back and forth. FTP is still one of the predominant methods for transferring these documents, so we have had to build a robust FTP solution to handle this traffic.

Wayfair Tech Blog

SVN Commit Hooks For a Better Codebase

As we have mentioned before, the main source control system we use at Wayfair is SVN, with TortoiseSVN as our client. One of the things we love about SVN is the ability to add commit hooks, or checks that run when someone tries to commit a file to source control. By having a few key checks we can prevent bugs, ensure consistent coding practices, and generally have a cleaner codebase.