In early April, Wayfair introduced the Digital Design Studio in our AllModern Dedham store. The Digital Design Studio (DDS) is a photorealistic interactive experience that lets customers explore Wayfair’s endless catalog by designing 3D furniture arrangements with product cards. Creating this tool was a collaborative effort between Wayfair Next (our R&D group tasked with exploring various emerging tech use cases), Physical Retail, Visual Merchandising, Creative, 3D, and many other Wayfair teams.
Why did we create the Digital Design Studio (DDS) ?
At Wayfair, we’re always looking to meet customers wherever they are in their shopping journey. In 2022, we opened up three physical stores in the Boston Metro area to complement our online offering. These stores help us better serve our customers by allowing them to experience and interact with our products in person.
When shopping for furniture online on Wayfair, customers have access to a nearly endless selection of home furnishings. However, physical furniture stores have a limited selection of furnishings available to browse, constrained by the store’s showroom floorspace. The DDS presents shoppers at Wayfair’s physical stores with a novel way to browse and interact with Wayfair furnishings that are not on the showroom floor: they can view furniture in a different fabric and finishes or even put together small furniture arrangements.
When a customer walks up to the DDS, they’re met with a kiosk with a glass surface, a selection of product cards, and a screen that prompts them to place a card on the surface to start designing. The customer selects one of these product cards and places it onto the glass surface. The screen then changes as a photorealistic version of the product on the card is placed in a virtual modern room. As they place more cards on the table, and move and rotate the cards, they see the virtual scene update and the changes reflected on the screen. At this point, the customer may choose to use the experience to compare relative seat heights of multiple accent chairs or to design a living room arrangement. The customer can change the time of day by using special lighting cards, use the camera card to move the viewpoint around in the scene, and even switch out the background scene to better match their own home.
Previous Iterations
Digital Design Studio took a few different forms before becoming what visitors see in the store today.
In 2016, Wayfair launched Patio Playground, a virtual reality experience where users could put on a VR headset and design their patio by moving 3D printed furniture models on a glass table. Each 3D printed model had a marker underneath it, which allowed the system to identify it and understand its position and rotation (this information is typically called its pose) using a webcam below the table. As the user moved this miniature furniture, they’d see these corresponding movements reflected in the physically accurate 3D furniture visible through their VR headset. This was a new way of interacting – using tactile, 3D printed objects to control and design a virtual scene. Parts of this initial experience would find their way into the DDS, including the glass tabletop and the use of a webcam and markers to track spatial locations of objects.
A few years later, Wayfair began thinking about opening up physical stores. We wanted to create a playful experience for customers so we revisited the idea of moving 3D printed furniture on a glass table. What if we were able to project textures onto white dollhouse furniture miniatures? We built upon our learnings from Patio Playground, using the exact same glass tables and marker-tracking concept as before, to create LightSpace, an interactive, projection-mapped dollhouse. Our biggest challenge was fine tuning the system to accurately map textures onto moving objects to create the illusion of textured furniture. We added one more playful aspect to the experience – each 3D printed object would have a near-field communication (NFC) sticker attached and customers could use the buttons of a magic wand to tap on a furniture model and flip through the textures for that model.
Building Digital Design Studio
Our past projects provided the background and learnings that we directly built upon to create the DDS. The current iteration of the DDS came about through close collaboration with the Physical Retail team. In particular, we wanted to explore different ways of giving customers at the stores access to our digital and physical catalog that went beyond just browsing the website. By focusing on a way to visualize products in the store, we were able to capitalize on three trends that were simultaneously occurring–namely the first consumer GPUs to support real-time hardware ray tracing ( NVIDIA launched its RTX line of GPUs at the end of 2018), newly available software that rendered photorealistic materials in real-time ( Chaos Group created Vantage to run on the new RTX hardware), and Wayfair’s growing library of high-quality models used elsewhere for imagery generation.
Based on these three synergistic trends we pitched a variety of proposals to the Physical Retail team’s leadership, including: a shoppable wall or window that rendered furniture at life-scale, kiosks for creating “endless aisle” vignettes or arrangements, and a way to visualize architectural or interior designs. The shoppable wall was selected as the first project because it could take advantage of one of the most visually striking features of the initial stores: a large format video wall that was large enough to be visible even outside the store. Imagine being able to walk up next to a life-sized couch on a screen and be able to compare your height compared to the seat height of the couch!
The initial prototype was a system where store associates could use a web application to control the main screen to drive the customer shopping experience. Associates could search for furniture and load it into the scene at fixed locations. They could also move the viewpoint through the scene to examine the furniture from different angles.
We wanted to experiment with different tactile ways to interact with the system. For a quick proof of concept, we used the same 3D printed dollhouse items and marker tracking system from LightSpace to control the items in the scene. It was a fun and intuitive way of designing a space.
However, it would become difficult to scale this system–we would have to create a 3D printed version of each item in our catalog!
After brainstorming for a bit, we experimented with a new idea -- using product cards instead of 3D printed pieces of furniture. It kept the tactile element that we liked about LightSpace, while being easier to scale. The interaction would be familiar to most users -- simply placing playing cards onto a table! Customers would be able to select from a wide array of furniture cards to quickly design true to scale, virtual furniture arrangements.
As the prototype began to take shape, we began to evolve how it looked as well. We moved away from the large format display to a kiosk based approach that was more self-contained. We got help from our co-founder Steve Conine who helped us cut a hole through a Wayfair dining table and create an insert for the table that housed the webcam.
In addition to product cards, we also created more ways to interact with Digital Design Studio. Moving the camera card close to a product on the table would allow you to zoom in to view the finer details of the materials, taking advantage of the photorealistic output on screen. Because lighting conditions can drastically change how products are perceived, customers could place lighting cards to change the time of day. They could also use scene cards to switch through various backgrounds, to find one that most closely matches their own home.
DDS System Design
The final DDS system consists of three major components: 1) the sensing table, which captures the poses of the product cards, 2) a workstation running 3D digital content creation (DCC) software that receives the poses and places and orients the corresponding 3D furniture model in the virtual scene, 3) and a real-time ray-traced renderer that receives the geometry changes and renders a photorealistic image to a display.
Sensing Table and Product Cards
The sensing table consists of a table with a window cut into it with a translucent glass pane and a webcam positioned underneath, pointed upward at the window. Each product card has an image of a single product, lighting scenario, or camera on the front and a unique ArUco marker on the back. ArUco markers are a binary square fiducial marker that supports identification of the marker as well as the calculation of a 6-degrees of freedom (DOF) pose. Cards are placed on the translucent window of the table with the ArUco marker facing down and therefore visible to the webcam.
The frames from the webcam are processed with OpenCV to detect the markers and calculate the pose of each product card. This pose information is serialized as a JSON string and is sent via a network socket to a server once every second for rendering and visualization.
DCC Server
A Python-based socket server running in the DCC tool, in this case Autodesk's 3DS Max, has access to the virtual scene as well as a large catalog of 3D models. The server extracts the marker identifiers and poses from the JSON string. Most markers correspond to specific products, but some are reserved for other features such as camera or lighting condition controls. For product markers, the server retrieves the appropriate 3D model for each product card, applying the appropriate materials for the variant of the product, and places the specific variant of the model in the virtual scene at the location and orientation specified in the pose. For the camera marker, the server moves the camera to the location corresponding to the pose. For lighting markers, the server updates the global lighting conditions to either a sunny day, an overcast day, or nighttime.
Real-Time Ray Traced Renderer
A real-time ray tracer renders the staged virtual scene using physically based cameras, lights, materials, and global illumination. It streams geometry changes from the DCC tool and renders the changes in real-time. The ray tracer here, Chaos Vantage, requires a RTX class GPU for real-time rendering. This output is displayed on a display (e.g. a monitor, projector, or a video wall) so that users can get an accurate sense of the furniture. In this case for the final kiosk, we used a Frame TV made by Samsung as our display.
Final Build for the Store
With the internals and software built, we could shift focus on finding a good way to present the unit. For the version of the Digital Design Studio you see in-store today, we serendipitously stumbled upon an unused Wayfair-branded kiosk and ended up finding that it worked perfectly for us! We retrofitted it for the DDS by cutting a hole through the top, adding a monitor arm to hold the Frame TV, and rebranding the front panel for AllModern.
We Are Always Innovating
So, what’s next? It’s been a fun journey seeing the Digital Design Studio evolve over time! The DDS is a pilot experience that we’re testing, so we’ll be monitoring how customers use it and constantly improving it. We can already see how it can be a way for customers to quickly mock up spaces and we envision a future where a customer can bring a picture of their home to the store and be able to design that home with Wayfair’s endless catalog.
