Prompting Like An Expert: Cognitive Behaviors That Matter

Summary

I recently created this meta-prompt after applying what I learned from this arXiv paper.

Behavior Driven Prompt

Exploring Expert Behavior Driven Prompting

The paper caught my attention as it seemed it could be a great resource for prompt design.

Just before reviewing it, I'd been struggling building a prompt with a strict set of output requirements. One of them was to keep the output under 2000 words.

It kept creating well formatted outputs about 2500 words long.

This wasn't surprising as LLMs often "think" in terms of token length as supposed to word length.

After summarizing the paper, I knew it could help. And when I was done it did. I was able to get high-quality output in the 1400 word range.

Using the HTML version of the paper I asked Claude 3.7 Sonnet to summarize the paper for me.

Summarize this research report for me.

[THE HTML VERSION OF THE arXiv PAPER (https://arxiv.org/html/2503.01307v1)]

Research Summary3.7 Sonnet Summary of arXiv paper

I then followed up by exploring how the research could be applied to prompt design.

How would the learnings from this paper be used when designing system prompts for llm chat assistants using the direct foundation model API (ex: openai, anthropic)

Applications of research

I then asked what a prompt would look like that incorporated the application of this research.

Here's what I got.

Behavior Driven Prompt v1

Now I could use this prompt to make it even better.

The v2 prompt was created using ChatGPT 4.5.

Creating Behavior DrivenPrompt v2

GPT-4.5 created an improved result which I was able to use for another round of improvement.

Creating Behavior DrivenPrompt v3

And I ended up the the final prompt to use.

Behavior Driven Prompt v3

I used both Claude 3.7 Sonnet and ChatGPT 4.5 to evaluate all three versions of the prompt.

Evaluating each prompt

It was satisfying to see the v2 prompt score higher than the v1 prompt and the v3 prompt score higher than the v2 prompt.

Eval Results

This entire process was a lot of fun. It's another example of how when we identify something potentially useful, we can leverage LLMs to quickly validate its potential usefulness and create new tools we can use.