Speaker: Julie Novak (Netflix)

Title: Statistical Methodologies in Streaming Experimentation at Netflix

Abstract: This talk will begin with an overview of how controlled randomized tests assist in informed data-driven decision making at Netflix. I will then go over streaming Quality of Experience and the components necessary to improve it. I plan to dive into two specific challenges associated with experiments in this space and describe the statistical solutions that we use to address them. First, the metrics we are comparing often have very ‘non-normal’ distributions and extreme outliers, so t-tests and standard non-parametric tests have undesirable properties. We develop a suite of bootstrapping tools which provide practical and statistical significance, as well as easy to interpret visualizations. Second, we need a fast, automated failure detection system to ensure that tests do not break any devices. To solve this we built a Bayesian regression model which updates its decision as data streams come in.

Roundtable Discussion will start immediately after this talk. The topics include: Data science and machine learning at Netflix; What skills does Netflix expect for new hires?

So, please attend both the seminar and the roundtable discussion!