Surfacing the Methodology

We recently designed a new feature in Plotly Studio called "Methodology".

It surfaces the data source, columns, formulas, assumptions, and approach front and center in the interface. We designed this so that it'd be easier to verify and trust the results from the agentic loop and to address the feeling of mental atrophy that is becoming all to common in our AI experiences.

In many AI products and productivity tools, AI is generating the majority of the output and our role has transformed from creating things to reviewing things.

Data is messy - full of ambiguities, tribal knowledge, null values, and assumptions. Many metrics can be defined in multiple ways and the details are important in how we understand and communicate the results. For example, the start and end dates for financial metrics, the window positioning for moving averages, the parameters and constants in scientific formulas.

Methodology surfaces the approach. It designed to be easy to scan but detailed, thorough view of the most important aspects of the data analysis step.

AI can often generate too much and too fast for us to comprehend, and many of the AI experiences on the market can often feel overwhelming once you move past the demo. Verification ends up being a way to slow it all down, learn as it works, and prevent mental atrophy.

In Plotly Studio, all of the analytics are backed by code. We've long believed in the power of code-based analytics. But many of the tools that we use for working with code in software engineering are ill-equipped for working with data.

Review is one of those. Each of these steps has 20-100 lines of code. While this is a manageable set of code to review, it's not always easy to pull out the data-related details of the code. The Methodology section helps put it in context. It also provides an implicit review checklist, by surfacing the important things front and center like any assumptions that were made or if any data points were hard-coded.

We built Methodology is created by a second agent of the main agentic loop and context. This has a two benefits:

1. It doesn't slow down the main agentic loop.
2. It isn't "influenced" by the previous context.

AI sessions can sometimes get "stuck in their ways". They can fib or make up data in order to satisfy user's request. By creating the Methodology as a separate AI call, we can avoid this contextual history and steering. The Methodology agent simply reviews the code and the result without any prior knowledge of the original user's request or the history of the session. This gives the results quite a bit more neutrality and consistency.

Methodology will be available in the next version of Plotly Studio. Give it a try and let me know what you think.