An ML manager, a data scientist and a product engineer get on a Zoom call…
Building a software or product feature can be challenging in the best of times, even with defined workflows involving an army of developers. Now, with the pandemic compelling companies to make rapid advancements in software development and automation, project management can seem daunting.
Imagine having to develop a machine learning product or feature in the midst of a lockdown. If distributed teams are to remain a prominent workplace feature, and offices are to be reimagined, companies need to re-orient themselves for complex projects factoring in the new realities.
Last year, I had the opportunity to lead a machine learning project to build a smart de-duplication tool that would remove duplicate and incorrect data that clog databases everywhere. The tool, which needed to be deployed at scale, was among the earliest projects to be built by a Freshworks team based outside the company’s office in Chennai, India. I learned a lot from that exercise.
In this blog, I delve particularly into developing a machine learning feature for a product company. Say, a customer churn prediction feature for a telecom company based on call activity, or lead scoring for a maker of customer relationship management products. On the consumer-facing side, think of a newsfeed ranking algorithm for a social media platform or a recommendation engine for a video streaming service. (Workflows for ML projects in services and consulting firms can be significantly different and will not be covered in this blog.)
Machine learning projects demand a somewhat different approach from regular software and product feature development. Building an ML feature involves experimentation with data as well as engineering heavy development. Each experiment that is successful bends future ML feature projects in new directions. But because of this, we do see a lot of friction between data scientists and engineers.
Engineers don’t like having to adjust to continuously changing requirements from data scientists. Data scientists get frustrated with any hurdles that keep them from taking their improved models to production. Product teams need a feature that works well for customers and can scale at the same time.
The challenge lies in managing these projects to perfection while maintaining a delicate balance.
Harnessing friction
Software development projects typically involve product managers, backend developers, frontend developers and software leads, their roles broadly well-defined. Most organizations have software to track and manage their work.
Machine learning projects involve a few additional specialist stakeholders—data scientists and machine learning engineers. In some organizations, the same person straddles the two roles.
When so many highly specialized engineers come together for a project, friction is inevitable. Desired, even. Complex engagements demand friction. The challenge lies in harnessing that energy for smooth and efficient execution. You need to begin with clearly defining the roles and responsibilities for product managers, data scientists, machine learning engineers, and the sundry others who will be involved.
As the manager of a machine learning team, my role primarily entails creating a collaborative environment for ML engineers and data scientists to work with product engineers. This is to provide as much clarity and independence to each stakeholder and empower them to be on top of their game.
Many a time, a complex model built by data scientists cannot be engineered within the timelines. I need to factor in such challenges that often crop up in ML projects while defining the roles and responsibilities.
Layering project workflows
Break down any software or feature development project into bite-sized tasks that can be more easily managed and tracked. These will form the building blocks of your project workflow. Based on my experience with machine learning projects, I have crafted a workflow that would be applicable to other similar projects as well.
At Freshworks, this framework is particularly relevant to ML/AI projects such as those involving Freddy, our artificial intelligence platform. Projects that deliver predictive insights, automate repetitive tasks, and surface new opportunities so customers can harness the power of AI.
I have split the workflow into five broad categories.
1. Defining the problem
A clear statement of the task at hand is crucial for aligning all the stakeholders with the project’s mission. The project manager, with help from the machine learning manager, should not just define the problem and success metrics but also translate and make them relatable to each of the stakeholders.
2. Data exploration and quality
This is the most important and yet the most ignored activity by machine learning teams. The more you dig into your data sources, the easier and better your end products will be. Data scientists and ML engineers should play tag team here.
This stage will set the rhythm for the rest of the project.
3. Modeling
This stage is data science-heavy and includes all the experimentation and modeling done by the data scientist. From a product development standpoint, it is crucial to figure how and when to stop with experimentation so the engineers can take over.
The data scientist needs to ensure that the models are not too complex to be deployed within the project’s timelines.
4. Productionization
This is a crucial stage where all the stakeholders meet again. Timelines, model accuracy, and engineering contracts should be revisited and finalized now. A crucial step that happens here is code handover from the data scientist to the machine learning engineers.
Once the first version of the framework/code is written, ML engineers should explain it to the data scientist. From here, any incremental changes to the model or feature engineering should be doable by the data scientist.
This is a typical point of friction between these two stakeholders.
5. Quality check
Broadly, there are two aspects of quality check in this stage:
1. Ensuring model accuracies are the same between the production code and experimentations that were done earlier; and
2. Ensuring that the customer-facing interface works smoothly with the backend ML engineering framework.
And you’re done!
I hope this framework for modeling ML project workflows is useful for teams in navigating the contours of their new workspaces.
While this blog focuses on a particular aspect of product or feature development, it offers a hassle-free template that can be adopted for other complex collaborations as well. Particularly if the only way to connect with the key stakeholders in your projects is virtual.
[Co-author: Feroze Jamal]
