Case study: Leveraging generative AI to enhance data-driven decision making at a global retail company

With the release of ChatGPT there is increasing demand by employees to understand how to use ChatGPT to assist in decision making and problem solving, especially when teams are remote or newly formed. However, a key challenge to use ChatGPT at work is that employees should never enter confidential company information into these systems. So how do you help people use it responsibly? 

Large Language Models such as ChatGPT are general purpose, ubiquitously available technologies. ChatGPT has built representations of vast amounts of human knowledge and creativity and now makes this knowledge accessible to anyone. It communicates with us in ways that are natural and intuitive: through natural language, and soon, likely vision and audio. It can reason in a human-like manner to some degree across any field of enquiry. This makes it incredibly flexible and useful to individuals.

We’ve created a workshop version of our data-driven decision making training specifically designed for learning and fine-tuning the use of ChatGPT for helping make better decisions. Our focus is on the team: how can employees best use their digital partners? Every team has doubled in size as people bring their ChatGPTs to work. While each individual might be smarter, what does it take to make the team smarter and increase collective intelligence?

The Workshop Approach

Over the course of two days, participants interacted with ChatGPT to navigate through various stages of the decision-making process. The workshop was designed around a family of prompts, and individual and personal interactions that participants used to guide their interactions with ChatGPT.

How To Start

Some of the participants’ favorite uses of ChatGPT were to help them start out with setting up team roles, bias busting, providing a comprehensive process template, and to help them think through how to weigh up different options. 

For example: setting up team roles.

Prompt:

“You are a coach and strategic advisor. We’re trying to put together a team to make a decision about a marketing plan. We have five team members. Jan is the decision maker but she doesn’t understand the problem well because she is new. Han is the marketing guru who has been at the company for a long time and worked in stores as well as head office. Jaycee is the data scientist on the team but she’s so busy we can’t do all the analysis we’d like to. Kirstee knows the problem well and she is pretty set on a particular solution based on her experience. How should we structure the team at the start and who should do what in the decision making process? Be creative and think ahead to how we will need to make the decision by including a typical data-driven decision making process in marketing.”

ChatGPT:

‍Creating a team to make a decision about a marketing plan requires understanding the expertise and roles of each team member. Here's a way to structure the team and the decision-making process:

‍**Jan (Decision Maker)**: Jan's role as the decision maker is to finalize the marketing plan based on the inputs from other team members. Being new, she may lack some background knowledge. However, she can provide a fresh perspective and be open to new ideas. Jan should start by learning about the company's marketing history, the current marketing problem, and the objectives of the proposed marketing plan. Han can provide much of this context. Jan's primary role is to listen, learn, and eventually make the final decision based on the evidence and opinions provided by the team.

‍**Han (Marketing Guru)**: Han, with his long-standing experience in the company and industry knowledge, is the link between the marketing plan and the company's historical context. Han can provide insights into past marketing strategies and the responses from the market. Han should work closely with Jan to bring her up to speed, help define the marketing problem, and propose potential solutions based on his marketing expertise. 

‍**Jaycee (Data Scientist)**: Jaycee's role is crucial as data is fundamental to any marketing plan. Her job is to identify, gather, and analyze relevant data to inform the team's decisions. Given her workload, it's important to prioritize her tasks. The team should clearly define the key questions they need data to answer. If necessary, consider bringing in additional data science resources or finding ways to automate data collection and preliminary analysis.

‍**Kirstee (Subject Matter Expert)**: Kirstee's understanding of the problem and her experience are valuable to the team. However, it's important to ensure that Kirstee's preferred solution doesn't overshadow other potentially viable options. Kirstee should collaborate with Jaycee to identify what data would be useful in evaluating her proposed solution as well as alternative options. 

‍In terms of the decision-making process, here's a potential sequence:

‍1. **Define the Problem**: The team should clearly outline the marketing problem they aim to solve. Han's and Kirstee's understanding of the company and past marketing efforts will be crucial here. 

‍2. **Generate Possible Solutions**: Each team member should propose potential solutions. Han can propose strategies based on his marketing expertise, Kirstee can share her preferred solution, and Jaycee can suggest tactics driven by data insights.

‍3. **Collect and Analyze Data**: The team should identify what data they need to evaluate the potential solutions. Jaycee, in coordination with Han and Kirstee, will take the lead on this.

‍4. **Evaluate Solutions**: With the data insights, the team should evaluate each proposed solution. Here, Jan's role as an unbiased newcomer can be invaluable, as she can question assumptions and challenge conclusions.

‍5. **Make the Decision**: After thorough discussion and evaluation, Jan, as the decision maker, makes the final decision. She must consider all the evidence and opinions provided by the team. 

‍6. **Implement and Review**: The team implements the chosen solution and monitors its effectiveness, making adjustments as necessary. 

‍By defining these roles and a clear decision-making process, your team can effectively work together to create a well-informed marketing plan.

Now this is pretty generic stuff but it feels somewhat personalized, has a level of completeness, and is a useful starting point. It saved the team some time by short-cutting the process of team set-up. This was their first experience of ChatGPT: it’s often good enough.

How To Plan

Next, the team wanted to supplement their experience and knowledge with that of ChatGPT’s to create their analytical plan. Sometimes ChatGPT’s generic nature can be helpful. Humans are great at discriminating—choosing things they don’t want or eliminating things—but less good at generating complete lists. The team found that ChatGPT was invaluable for supplying lists of all the things they could (or should) do and then they made decisions about what was possible and practical. All of this without giving away any confidential information!

Prompt:

“You are a marketing analytics guru and you are tasked with helping our team develop an analytical plan to make decisions about a marketing plan for a new product. We have detailed and granular data available on sales, by region, by product, in near-real-time. What would you recommend is a best-practice analytical approach for launching a new product? b=Break this down into discrete analytical tasks.We have scarce resources for data analysis so help us prioritize what is most urgent for the decision and most relevant. Present this in the form of a table with headings who, what, why, how, when.”

ChatGPT:

‍Sure, here's a recommended analytical approach for launching a new product, broken down into discrete tasks. I've used the "who", "what", "why", "how", and "when" structure you requested:

‍In terms of prioritizing the tasks, we would suggest the following order, based on urgency and relevance to the decision-making process:

1. **Collect and clean historical sales data**: This is the foundation for all subsequent analysis and must be done first.

2. **Perform market segmentation analysis**: This is crucial for informing the design of the new product and the development of the marketing plan.

3. **Competitive analysis**: This is important for positioning the new product in the market and should be done in parallel with product design and marketing planning.

4. **Create predictive sales models**: This should be done after the product design is finalized but before the launch, as it will inform sales goals and the marketing budget.

5. **Develop a real-time sales tracking system**: This needs to be set up before the product launch to allow immediate feedback.

6. **Perform A/B testing on marketing channels**: This is important for optimizing the marketing campaign and should be done before the launch.

7. **Review and adjust the marketing plan**: This is an ongoing task that should begin immediately after the product launch.

This is where ChatGPT’s generic response disguises A HUGE AMOUNT OF WORK. So what happens in the team? As the group starts to parse this list and critique it as a potential plan they realize they have a lot to do and not enough time to do it. So they do what humans do: get creative and use their judgment. They appreciated that ChatGPT presented the plan in a table—it makes it faster and easier to compare the tasks. After this step the group was able to prioritize a subset of tasks much faster than if they had generated the list themselves. ‍

How To Decide

Once the team had used analysis and their combined experience to determine their various options, it was time to make a decision about which approach would work best. The data can only yield so much insight, analysis can predict only so much, and experience is only as good as the world is predictable. So there’s always judgment to be made in decision making. 

This team was most concerned with ensuring a good decision making process—they didn't want to fall prey to “resulting” where the decision is judged only on the outcome not on whether they had considered all dimensions of the decision. So they first used ChatGPT to help them lay out a series of meetings that would get them to a decision.

Prompt:

You are an expert decision coach. Assume we have completed our primary analysis and have developed some options to consider. Lay out a high quality process that will help us reach a decision. Offer practical suggestions, outline the meetings required, and suggest ways that the team can deal with disagreement. Be specific and practical, as well as creative. 

ChatGPT:

‍Certainly, here are the next important steps, taking into consideration creative ways to make good decisions:

‍1. **Bias Checkpoint**: Have a dedicated meeting to explicitly discuss and check for cognitive biases that could be affecting the team's judgment. This might include confirmation bias, anchoring, overconfidence, or others. Everyone should be encouraged to challenge each other's thinking in a constructive way.

‍2. **Listing Alternative Causes**: Conduct a session to brainstorm alternative causes or explanations for the data and insights you've gathered. This helps ensure you're not fixating on a single explanation and ignoring other possibilities. Consider using a facilitator to ensure a free flow of ideas and to prevent dominant voices from controlling the discussion.

‍3. **Evaluate Options Using the Data Analysis**: Perform thorough data analysis for each of the identified options. Jaycee, as the data scientist, should lead this process, but all team members should be involved so they can understand the implications of the data.

‍4. **Pre-Mortem Session**: Organize a pre-mortem session where the team imagines that the chosen option has failed spectacularly. Ask each team member to provide reasons for the 'failure'. This can help identify potential risks and challenges that weren't previously considered.

‍5. **Pre-Parade Session**: Similar to the pre-mortem, but in this case, imagine the chosen option has succeeded beyond all expectations. Ask team members to outline the factors that led to this 'success'. This can highlight strengths or opportunities you might not have considered.

‍6. **Make a Preliminary Decision**: Based on the analysis and sessions, make a tentative decision. Ensure everyone feels heard and that their input has been considered, even if their preferred option isn't chosen.

‍7. **Dealing with Disagreement**: If there's disagreement about the decision, encourage open dialogue to understand the reasons. If the disagreement persists, consider bringing in an impartial mediator, or use a structured decision-making tool such as multi-voting or decision matrix.

‍8. **Final Decision Meeting**: Host a final decision meeting where Jan, as the decision maker, communicates the final choice to the team. She should explain the rationale behind the decision, addressing the points made during the disagreement.

Here it got really interesting. Participants were curious to try out some of these ideas and appreciated that ChatGPT could suggest a series of meetings. It felt like a practical series of steps for them to debate and decide on. The process of deciding on the process gave them confidence in the quality of their process!

These were just a few highlights of the workshop. Participants left with a sense of the myriad ways that ChatGPT can be deployed in a team decision making process to help start, guide, and decide.