Glowstick is proud to share our first post in our new series about prompt engineering. We hope this helps other people better navigate this new field!

Summarization of Sales Opportunities

Today we will cover the prompt we’ve shown on Glowstick website. This prompt is the result of a group effort where everyone contributed, you can learn more by reading our blog. Again, congrats to Shruti Gupta, our designer who crafted this prompt!

Task description

Glowstick is a platform that raises sales opportunities. Our task is to generate a headline from the snippet of conversation we detected as an opportunity. We have access to the conversation, the customer name and the product targeted by this opportunity.

Examples

Examples

Headline: BlueSky is interested in the Multi-Language product.
Conversation
Olivia Johnson: I suppose it would be good to understand your overall goals?
BlueSky: I know it was just an additional request that came up, but obviously since most of our business, or half our business is in Arabic, that's why it's very important for us.
Olivia Johnson: Yeah, Multi-Language can definitely meet that need, makes total sense, I’ll send you the numbers and copy the team too.

Headline: BlueSky will talk to their Engineering Director about Webhooks.
Conversation
Olivia Johnson: No questions, just wanted to appreciate all the work that you've put in and excited to see you run your first surveys!
BlueSky: Yeah, thanks for that, although about that Webhook stuff, we definitly want to have a conversation with our engineering director first to see if there is interest in taking more on that, back to the webhook stuff you're talking about.
Olivia Johnson: Should we schedule a call with her and go through the add-on together?

Concerns

In this project, our primary concerns were 1. Accuracy, 2. Conciseness, 3. Headlines should target a specific product.

Evaluation

To evaluate ourselves against our previous iteration, we ran a voting session at Glowstick and this agent won 70% of the time across our test set against 5 other agents.

Winning Prompt

We use GPT4 Assistant API with the following prompt, and run instructions:

You will read a portion of a longer conversation between a Sales and Customer success team and their customers. Summarize the main ideas from the snapshot of the conversation into a short summary not more than 10 words. Start the summary with the 'Customer name is'. If mentioned, highlight any interest in a product or service.

Run Instructions

The customer is {customer name}, the product is {product name}.

Customer_name is the name of this customer (BlueSky in the examples above), product_name is the name of the product (Webhooks, Multi-Language in the examples).

Analysis

A key item of our prompt is not more than 10 words. Depending on the version of GPT you’re using, the LLM can ignore this part and output a full paragraph. We also observed that in some cases, the LLM would continue the conversation, ignoring the instruction altogether.

Finally, we observed that this prompt would rarely ignore product_name even if it was wrong. Thus, our accuracy in product matching is more important than ever which triggered a new iteration of this system (that I will describe in a future post if there’s interest).

Next Steps

Now that we’re pretty happy with the state of sales opportunities summarization, we want to target new types of insights such as churn risks, CSQLs or even competitor mentions.

Tell us about your best summarization prompt and if you’ve tried our prompt let us know in the comments!

We would be eager to create a dataset for this particular task if people are up for it. Contact me on LinkedIn/Email.

You can learn more about what we do at glowstick.ai.