How to Review AI Chatbot Results After Black Friday: A 7 Step Checklist (2026)

Why the dashboard number is not your answer

Adobe reported that U.S. shoppers spent $44.2 billion online during Cyber Week 2025, up 7.7% year over year, with $14.25 billion on Cyber Monday alone. When traffic spikes like that, a chat agent talks to more shoppers who were going to buy anyway, and every tool that touched those shoppers wants credit.

Each tool counts in its own way. Here is what their public documentation says as of October 2026:

None of these definitions is wrong. They answer different questions. A shopper who opens a Klaviyo email on Black Friday, asks your agent about sizing that evening, and buys on Sunday can appear in the email report and in a 3 day or 14 day chat report at the same time. Add the reports together and you count one order two or three times.

Step 1: Set the review dates before the sale starts

Write the schedule down now, while you are calm. Each window must close before you can call the number final.

• Tuesday, December 1: the 24 hour window for Cyber Monday conversations has closed. Run a first pass on any tool that uses 24 hours.
• Thursday, December 3: 3 day windows for Cyber Monday have closed. Klaviyo's default 5 day email window for Cyber Monday closes on Saturday, December 5.
• Monday, December 14: a 14 day window for Cyber Monday conversations has closed. This is your final number if you run Fin or any tool with a two week window.

Freeze the period you are reviewing, for example Thanksgiving on November 26 through Cyber Monday on November 30, and use the same dates in every export. Changing them halfway through is how two people on one team get different answers.

Step 2: Export orders and match them by order ID

Pull two lists:

1. Your store's orders for the period, with order ID, timestamp, total, discount code, and refund status. On Shopify this is the Orders export.
2. Each tool's list of attributed orders, with order IDs. If a tool only shows a total and will not show which orders it counted, treat that total as a marketing number, not a measurement.

Put them side by side in one sheet. Give each store order one row and add a column for every tool that claims it. Then answer three questions:

• How many orders does exactly one tool claim? These are the easiest to credit.
• How many orders do two or more tools claim? Do not add their revenue twice. Decide one rule, for example "last interaction before purchase wins," and apply it to every overlap.
• How many claimed orders were refunded or cancelled? Holiday returns arrive late, so recheck refunds again in January. Ask each vendor whether a refund reduces the attributed amount, because the answer differs by tool.

If you run WooCommerce, note that its built in Order Attribution records the visitor's traffic source, so an order the chat helped close will usually show as Organic or Direct there. Our guide to the AI sales agent for WooCommerce explains why those two numbers should be matched by order ID, not added.

Step 3: Ask how many of these orders would have happened anyway

Order matching tells you which orders followed a conversation. It does not tell you which orders the conversation caused. Shoppers who start a chat on Black Friday are often already close to buying, so comparing "chatters convert at 9%, non chatters at 3%" overstates the effect.

You can still get a fair reading:

• Read the conversation, not just the order. For a sample of 30 to 50 attributed orders, open the transcript. Did the agent answer a question that was blocking the purchase (fit, delivery date, stock, compatibility), or did the shopper only ask where to enter a discount code? Mark each one "unblocked the sale" or "was along for the ride."
• Plan a holdout for the next peak. Showing the agent to a random share of visitors and hiding it from the rest is the only clean test. Our guide on how to measure chatbot attributed revenue covers how to set one up and why it needs a lot of traffic.

The share of "unblocked the sale" transcripts is your best quick estimate of real lift. Write it next to the attributed total so nobody reads the total alone.

Step 4: Sort the conversations that did not end in a sale

This is where the money for December sits. Export every conversation from the review period that did not lead to an attributed order and tag each one by the question the shopper asked. Most stores find the same short list:

For each tag, count three things: how often it was asked, how often the agent said it did not know or handed off, and how often it gave an answer that was later wrong. Sort by the first count times the second. The top three rows are your content backlog for this week.

Fix the source, not the bot: an AI agent answers from your store's content, so the fix is usually a missing sentence on a page. Our guides on backorder and preorder questions and return policy questions before purchase show how to write those pages so an agent can quote them.

Step 5: Check every promise the agent made

During a sale, the bigger risk is a confident wrong answer. Search your transcripts for these words and read every hit:

• "guarantee," "will arrive," "promise": delivery promises your warehouse never made.
• Discount codes and percentages: codes that expired at midnight on Cyber Monday but still appear in the agent's answers, or a percentage that was never on the site.
• Prices: answers that quote a sale price after the sale ended.

Discounts do not stop on Cyber Monday. Adobe found that after Cyber Week 2025, discounts through the first week of December still ran at 23% for toys, 19% for electronics, and 18% for apparel. If your store runs a softer December offer, your pages must say clearly which offer is live today, or the agent will mix them. Rerun the tests in our pre launch testing checklist after each change.

Step 6: Work out what each attributed order cost you

Revenue is only half of the review. Divide what you paid each tool for the period by the number of orders it can show you by order ID, after removing refunds and duplicates. That is your cost per attributed order, and it is the number to compare across tools.

The answer depends on what triggers the bill. Prices below are from each official page as of October 2026:

Be fair to each model. A free agent costs nothing, and if its assisted orders hold up in your order matching, it may be enough. A fixed credit plan is cheaper than SparkGPT once attributed sales pass a break even point: (40 minus 2.99) divided by 0.05 is about $740 a month against Chatbase Hobby, and (150 minus 2.99) divided by 0.05 is about $2,940 against Chatbase Basic.

What changes with SparkGPT is who carries the risk of a slow month. When a visitor talks to the agent, the session is tagged, and a purchase within 24 hours counts as an attributed sale. You pay 5% on those orders only, and every one is listed in the ROI dashboard with the details you need to match it to your store's orders in Step 2. If December brings many questions and few sales, the bill is $2.99. Billing runs on prepaid credit, so the agent pauses when the balance runs out instead of creating a surprise charge.

Step 7: Turn the review into December changes

You have about two weeks between the final Cyber Monday number and the last shipping dates, so keep the output short:

1. Three page fixes from the top of your Step 4 backlog, live before December 10.
2. One correction list from Step 5: expired codes removed, the current offer stated in one place.
3. Holiday shipping dates published. USPS recommends sending Ground Advantage by December 17 and Priority Mail by December 18 for delivery before December 25 in the lower 48 states. Your last order dates are earlier; our guide to holiday shipping cutoff questions shows how to calculate them.
4. One number for the team: attributed orders after deduplication, the share that "unblocked the sale," and cost per attributed order. Three figures, one line each.

Save the sheet for next year; our Black Friday and Cyber Monday preparation guide shows where these findings plug in.

Frequently Asked Questions

When should I review my chatbot's Black Friday results?

Run a first pass on December 1, 2026, when 24 hour windows for Cyber Monday have closed. Wait until December 14 for the final number if any tool you use counts orders up to 14 days after a conversation.

Why do my chatbot, helpdesk, and email reports add up to more than my total sales?

Each tool uses its own attribution window and credits any order inside it, so one order can appear in several reports. Match every claimed order to your store's order list by order ID and credit each order once.

What is a good way to find questions my AI chatbot could not answer?

Export the conversations that did not lead to an order, tag each by question type, and count how often the agent said it did not know or handed off. Fix the most frequent gaps by adding the missing facts to your product or policy pages.

How do I know if the chatbot caused the sale or the shopper would have bought anyway?

Read a sample of 30 to 50 attributed transcripts and mark whether the agent answered a question that was blocking the purchase. For a clean answer at the next peak, run a holdout test that hides the agent from a random share of visitors.

What should I do about refunds on orders the chatbot was credited for?

Recheck refund status in January, after holiday returns arrive, and remove refunded orders from your review. Ask each vendor whether refunds reduce the attributed amount it reports or bills.

How much does an AI sales agent cost in a slow December?

As of October 2026, Shopify's Inbox agent is free, Chatbase starts at $40 per month for 700 message credits, and SparkGPT charges $2.99 per month plus 5% of sales made within 24 hours of a conversation. A month with no attributed sales costs $2.99 on SparkGPT.

Review your season with numbers you can check

Set your review dates now, export orders by ID, and fix the top three unanswered questions before the December shipping cutoffs. SparkGPT builds an AI sales agent from your store URL in about 10 minutes, and the Free plan lets you build and test one agent with no credit card. When you go live, you pay $2.99 per month plus 5% of the sales it drives within 24 hours of a conversation, and every attributed order appears in your ROI dashboard. Questions can go to hello@sparkgpt.ai.

Build your free AI sales agent at www.sparkgpt.ai