Our fourth roundtable discussion took place after the statistics seminar talk of Julie Novak (Netflix) on “Statistical Methodologies in Streaming Experimentation at Netflix”.

This discussion was conducted in the Q&A format, with the majority of questions about the structure of data science work at Netflix. We structure the note accordingly.

Q: What is the most pressing problem in data science and machine learning at Netflix?

A: There are many challenging problems in data science and machine learning at Netflix. One example that the speaker works on is building models on large datasets with a need for low latency answers. When new code is deployed, the data scientist is responsible for building a statistical solution that determines as quickly and accurately as possible whether or not an error has occurred. Another problem is dealing with extremely rare examples, such as unusual hardware combined with uniquely formatted movies where the rate of occurrence is below one in a thousand.

 

Q: What do new hires coming from a statistics program need to know immediately?

A: How to adapt to new programming languages quickly. The speaker has to quickly pick up Python, Tableau, and SQL, even though she had only had experience with R before joining Netflix.

 

Q: If she could go back in time, what theory or algorithms would the speaker have learned?

A: Causal inference, as understanding whether something is a true causal effect is very valuable from a business perspective.

 

Q: How would the speaker change the statistics curriculum? What would she add or cut out?

A: The speaker emphasized that those with a rigorous background in statistics had a lot of respect in the data science team, so she would be loathe to suggest what to cut. She would add courses that improve oral and written communication skills, particularly for explaining concepts to non-experts, as well as more computer science fundamentals courses.

 

Q: How much of the speaker’s time is spent coding?

A: She spends a large portion of her time coding, as testing ideas discussed in meetings requires it. She has become somewhat familiar with the code for the video codecs because a detailed knowledge of the system proves useful to her. In Netflix, integration of research and development into production is also important.

 

Q: How hierarchical is the organizational structure at Netflix?

A: The famous Netflix culture puts an emphasis on independent decision making, and attempting to preserve the small company mindset as Netflix grows. This goes hand in hand with an emphasis on responsibility. The speaker identified it as her job to make sure that the engineers understood the tools she and the data scientists were making available to them. This also means she is free to try new ideas if she thinks they will have an impact on the business.

 

Q: Are there concerns about A/B testing on real people?

A: Netflix takes the anonymity of their customers very seriously and do not need individual customer identities in order to run experiments. They value A/B testing because the learnings from their experiments are used to improve the Netflix product to make their customers enjoy it more.

 

Q: What background do most of the data scientists at Netflix have?

A: Data scientists have a diverse set of backgrounds. Those with PhDs typically come from statistics, computer science, and physics; those who have masters usually have a couple of years at another job. Netflix doesn’t have boot camps for new hires coming in the door because they hire experienced people.

 

Q: How many data scientists do work at Netflix?

A: There are a lot more software engineers than data scientists.

 

Q: Does Netflix hire data scientists in other regions (i.e., emerging foreign markets)?

A: At the moment the central data science teams are located in the Bay Area and in Los Angeles.

 

Q: To what degree is there inter-corporation collaboration?

A: Meet-ups are pretty popular between various tech companies, including Netflix. There was also a large contingent of Netflix employees at JSM (Joint Statistical Meeting), some of whom were presenting. 

 

[Scribe: David Weber (GGAM)]