Prompt Testing CLI Template

This document outlines a reusable command-line harness for quickly experimenting with prompts against language models. The goal is to standardize how you submit prompts, capture responses, and log results for later review.

Objectives

1. Quick iteration – Swap in different prompts and models without modifying code.
2. Consistent logging – Save all interactions to timestamped log files for analysis.
3. Extensible – Provide a foundation you can build on with additional features like batch testing or automated comparisons.

Setup

1. Create a new Python environment using venv or conda.
2. Install dependencies:

   pip install openai rich yaml
   

3. Scaffold the following files:
4. prompt_harness.py – main script to run prompts from the CLI.
5. prompts.yaml – list of prompts and parameters to test.
6. README.md – usage guide.

Basic Flow

flowchart TD
    Start(["Run prompt_harness.py"]) --> LoadPrompts{{"Read prompts.yaml"}}
    LoadPrompts --> Send["Send prompt to model"]
    Send --> Record["Capture model response"]
    Record --> Log["Write prompt & response to log file"]
    Log --> End(["Exit"])

Example prompts.yaml

model: "gpt-3.5-turbo"
prompts:
  - text: "Summarize the meeting notes"
  - text: "Generate a list of creative taglines for a coffee shop"
options:
  max_tokens: 150
  temperature: 0.7

Deliverables

1. Working prompt_harness.py that reads prompts from prompts.yaml and prints the model's responses.
2. Example log file showing at least two prompts and their outputs.
3. Updated README.md documenting configuration and how to add new prompts or change models.

Submission

Push your implementation to the repository and open a pull request. Include sample output from your log file. Note any limitations or ideas for future improvements in the PR description.