The Carlu, where the event was held, provided an intimate setting that fostered networking and engagement. The coffee bar and snack tables were well-stocked to fuel discussion, and the conference organizers did a fantastic job. Shout-out to Faraz Thambi for a great event!
A poll at the beginning of the event revealed only 58% of respondents had five or fewer models in production, and 25% had between 6-10. This, along with discussions we had with other attendees, confirmed what we have been seeing across the ML community at large, and with our own customers and colleagues-
There's not a one-size-fits-all solution or approach, nor do people seem to be looking for one. Instead, users want tools and techniques that help them gain deeper insight to model behavior quickly, know what to do to ensure fairness, mitigate bias, and accelerate performance.
Deploying more models to production faster and keeping models trustworthy seemed to be the biggest challenges we heard from people at the Summit. Many of the attendees who visited our booth shared that keeping models trustworthy is often an afterthought, becoming a significant barrier across the ML lifecycle.
At Vianai, we are deeply passionate about this as these problems become acute at scale - scale in terms of more models, more features per model, more inferences. Our focus is on helping users quickly identify:
We see this as a continuous cycle of operations that includes monitoring, retraining models, and validating them before redeployment.
Some data scientists (DS) we spoke with owned the full ML model lifecycle and were keen on gaining a deep understanding of each step in the process to learn how to continuously improve and build better models. Other DS saw their role as discrete from the rest of the MLOps process but struggled to manage clean hand-offs to machine learning engineers (MLE) and spent too much time answering questions and bridging the gap between the roles. One DS found that our simplified UX was exactly what she had been looking for so any other stakeholder could import a model and automate its deployment.
Our team was able to attend some pre-conference workshops and several sessions as well. I listened to Jesse Cresswell's talk "Navigating the Tradeoff Between Privacy and Fairness in ML" where he shared how privacy-enhancing technology can increase unfairness and bias in ML models, using federated learning and differential privacy as examples, and then suggested how to address the challenges.
I also listened to Hien Luu's talk "Scaling and Evolving the Machine Learning Platform at DoorDash" which gave insight into many lessons and technical decisions and tradeoffs that were made in building their platform to handle massive scale. There was a great selection of other topics that included:
We also met many accomplished and passionate students and recent graduates from the University of Toronto, University of Waterloo, and York University who were excited about ML - we hope to hire some of them! Overall, this event was thought-provoking, informative, educational, and fun! We can't wait to go back next year.