Scientific paper proposal: An empirical guide to MLOps adoption

[PierreSegerstrom]

Assignment Proposal

Title

An empirical guide to MLOps adoption: Framework, maturity model and taxonomy

Names and KTH ID

• De Chi Hao (dchao@kth.se)
• Pierre Segerström (pise@kth.se)

Deadline

• Week 4

Category

• Scientific paper

Description

We want to present the paper: “An empirical guide to MLOps adoption: Framework, maturity model and taxonomy”.

The authors present an analysis of 14 companies, where the ultimate goal is to provide a “structured approach to adopt, assess and advance adoption of MLOps practices”. The end result is a “framework” that has been developed through analysis of these companies’ MLOps practices, which has also been through feedback from MLOps practitioners. In addition, this framework provides a taxonomy for classifying ML use cases based on their context and requirements. As a result, any organization can utilize these findings in order to advance their adoption of MLOps practices regardless of their current status.

During our presentation, we will:

• Explain  (concisely, a few sentences)
  the problem of applying traditional linear Software Development Lifecycle to iterative ML workflows,
  
  
• Describe  (quick overview, 1 slide)
  the proposed 5-dimensional MLOps framework, 5-stage maturity model, and taxonomy,
  
  
• Present  (technical analysis)
  
  • Visualize the final maturity stage "Kaizen" (as a diagram of the 5 dimensions: data, model, deployment, operations & infrastructure, organization).
  • Explain how this becomes an end-to-end, continously improving MLOps system design.
• Discuss
  • the main empirical results from this 14-company multi-case study showing gaps between actual and desired maturity, and
  • the limitations of the study, as well as the assumptions that the results rely on.
    
    
• Summarize
  how this structured roadmap can help organizations systematically advance their MLOps adoption and DevOps collaboration.

Relevance

This is relevant to DevOps, since the framework itself acts as a roadmap to reach higher levels of “MLOps maturity”, where each level corresponds to better application of DevOps practices as a consequence. The paper’s highest level of maturity, “Kaizen MLOps”, not only showcases continual and iterative development of the ML model itself through DevOps practices. It also addresses other aspects like data processes, communication, and organizational roles. MLOps goes beyond the pipelines that are needed for maintaining software, since more dimensions are required, such as versioning of datasets, reproducible experiments, drift monitoring, and collaboration between data scientists and operations.

Understanding the paper’s framework for “levels of maturity” showcases the unique challenges of MLOps specifically, but also guides the reader to reflect on how a similar “multi-level-maturity” framework can be mapped to any DevOps context.

[ericcornelissen]

The paper looks interesting and the proposal is pretty good. Before I approve this, I wonder how you plan to cover the following two aspects of the grading criteria: "Technical analysis" and "Critical".

[PierreSegerstrom]

Hey @ericcornelissen!
Sure thing, putting some notes here below on our planned approach to cover these.

Let us know if something else needs further clarification, thanks!

---

"Technical analysis":
Students analyze and explain a key technical component (e.g., algorithm, system design) in detail
To present:

• Visualize the final maturity stage "Kaizen" (as a diagram of the 5 dimensions: data, model, deployment, operations & infrastructure, organization).
• Explain how this becomes an end-to-end, continously improving MLOps system design.

"Critical":
Students critique the paper by identifying limitations, assumptions, or threats to validity.
To address/discuss:

• Limitations: paper relies on qualitative evidence, rather than objective operational metrics (deployment frequency, recovery time, model drift occurrences, etc.) – "it's useful for assessment and discussion, but is not a precise measurement system" (similar to the SEMAT "Essence" framework for software engineering teams)
• Assumptions: the model can simplify reality, assuming that all parts of an MLOps pipeline require a certain standard, to achieve a certain level of maturity. Reality doesn't always look this way. Certain parts of an MLOps pipeline can require different levels of maturity, depending on the context. It is therefore up to the person who apply this framework to use it as an initial reference, and make intentional trade-off decisions from there depending on the requirements. Based on the framework, identify bottlenecks and where pipelines is missing maturity, and address them accordingly.

[ericcornelissen]

Sounds good @PierreSegerstrom, please update the proposal accordingly and I will approve and merge it 🙂

[PierreSegerstrom]

Thanks @ericcornelissen, updated now!