AI Integration Without the Buzzwords

The AI Decision Framework: When To Use It (and When Not To)

Before writing a single line of AI code, answer one question: can this problem be solved with a rule engine, a lookup table, or a database query?

If yes, use that. It will be faster, cheaper, more predictable, and easier to debug.

AI adds value in three specific scenarios:

1. Unstructured input. Natural language, images, audio. Data that cannot be cleanly parsed with regular expressions or schemas.
2. Classification with fuzzy boundaries. Categorizing support tickets, detecting sentiment, routing requests where the categories overlap and edge cases are common.
3. Generation with constraints. Drafting content, summarizing documents, translating between formats. Tasks where the output has a known structure but the content varies.

If your problem does not fall into one of these three categories, AI is probably not the right tool. I have saved clients tens of thousands in API costs by replacing planned AI features with well-designed database queries.

Choosing the Right Model

Model selection is an engineering decision, not a marketing one. The "best" model depends on four variables:

1. Task complexity Simple classification (spam/not spam, positive/negative) does not need GPT-4. A fine-tuned smaller model or even a traditional ML classifier will outperform it at 1/100th the cost.

2. Latency requirements If your feature needs sub-200ms responses (autocomplete, real-time suggestions), large models are out. Use smaller, faster models or pre-compute results.

3. Cost at scale This is where most teams get surprised. A proof-of-concept that costs $20/month at 100 requests/day costs $6,000/month at 10,000 requests/day. Model the cost curve before you build.

4. Data sensitivity If your data cannot leave your infrastructure (healthcare, finance, legal), you need self-hosted models or providers with SOC 2 compliance and data processing agreements.

The Model Ladder

For most production use cases, I follow this progression:

• First try: Rule engine or heuristic
• Second try: Small, fast model (Claude Haiku, GPT-4o-mini)
• Third try: Medium model (Claude Sonnet, GPT-4o)
• Last resort: Large model (Claude Opus, GPT-4.5)

Each step up costs 5-10x more per request. Only escalate when the lower tier genuinely cannot produce acceptable output quality.

RAG: The 80% Solution for Domain Knowledge

The most common request I hear: "I need the AI to know about our products/documentation/internal processes." The answer is almost always RAG (Retrieval-Augmented Generation), not fine-tuning.

How RAG works:

1. Index: Split your documents into chunks (300-500 tokens each). Generate vector embeddings for each chunk. Store them in a vector database (Pinecone, pgvector, Qdrant).
2. Retrieve: When a user asks a question, embed the question and find the most similar document chunks (typically top 3-5).
3. Generate: Send the retrieved chunks plus the user question to the LLM as context. The model answers using the provided context, not its training data.

Why RAG over fine-tuning:

• Updates in minutes (re-index new documents) vs. hours (retrain the model)
• No risk of catastrophic forgetting (fine-tuning can degrade general capabilities)
• Easily auditable (you can see exactly which documents influenced the answer)
• 10-100x cheaper to maintain

RAG Implementation Checklist

1. Chunk size matters. Too small (100 tokens) loses context. Too large (1000 tokens) dilutes relevance. 300-500 tokens with 50-token overlap is a safe default.
2. Embedding model choice affects retrieval quality more than the LLM choice affects generation quality. Use the best embedding model you can afford.
3. Hybrid search (vector similarity + keyword matching) outperforms pure vector search in production. Use both.
4. Always include a "no relevant context found" fallback. Do not let the LLM hallucinate when retrieval returns poor results.

Prompt Engineering for Production (Not Demos)

Demo-quality prompts and production-quality prompts are different things. A demo prompt that works 90% of the time is impressive. A production prompt that fails 10% of the time is a liability.

Production prompt principles:

1. Structured output always. Never ask for freeform text when you need structured data. Use JSON mode, function calling, or explicit output schemas. Parse and validate every response before using it.

2. Clear constraints. Specify exactly what the model should and should not do. "Respond in JSON with these exact fields" is better than "give me the data in a structured format."

3. Examples in the prompt. Few-shot prompting (2-3 examples of input/output pairs) reduces error rates by 40-60% in my testing. Always include edge cases in your examples.

4. Retry with escalation. First attempt: fast model with strict parsing. If parsing fails, retry with the same model and a rephrased prompt. If it fails again, escalate to a larger model. Three strikes and you return a graceful error.

5. Log everything. Log every prompt, every response, every parsing result, and every retry. This data is how you improve the system over time. It is also how you debug production issues.

The Cost of Bad Prompts

A poorly designed prompt that triggers unnecessary retries costs 2-3x more per request than a well-designed one. Multiply that by thousands of daily requests and you are burning money on preventable failures.

The clearest test of this was SocrateOS, a cognitive operating system where the memory graph, not the model, is what makes it work. Read how it was built.

AI Cost Management: The Unsexy Reality

AI features have a marginal cost that scales with usage. This is fundamentally different from traditional software where the cost of serving an additional user is nearly zero.

Cost modeling framework:

For every AI feature, calculate: - Cost per request (input tokens + output tokens at the model's rate) - Requests per user per session (how many AI calls does one user interaction generate?) - Daily active users (realistic, not aspirational) - Monthly cost = cost per request x requests per user x sessions per user x DAU x 30

Real Numbers

A support chatbot using GPT-4o with average prompt length of 800 input tokens and 400 output tokens: - Cost per request: ~$0.006 - Average 3 messages per conversation: $0.018 per conversation - 500 conversations/day: $270/month - 5,000 conversations/day: $2,700/month

That same chatbot using GPT-4o-mini: - Cost per request: ~$0.0003 - 5,000 conversations/day: $135/month

20x cost difference for a model that produces acceptable output for 85% of support queries. Route the remaining 15% (complex cases) to the larger model. Blended cost: ~$500/month instead of $2,700.

Cost Reduction Strategies

1. Caching. If the same question appears frequently, cache the response. A Redis cache with a 1-hour TTL eliminates 30-50% of API calls for support chatbots.
2. Prompt compression. Remove unnecessary context from prompts. Every token costs money. Compress system prompts, use shorthand in few-shot examples.
3. Model routing. Classify query complexity first (cheap), then route to the appropriate model (expensive only when needed).
4. Batch processing. For non-real-time tasks (email summarization, report generation), batch requests and use cheaper batch API pricing.

The Production AI Checklist

Before shipping any AI feature to production, verify:

Reliability - Timeouts set on every API call (LLM providers go down) - Retry logic with exponential backoff - Graceful fallback when AI is unavailable (static response, cached result, or "try again later") - Rate limiting to prevent cost spikes

Quality - Output validation (schema check, length bounds, content policy) - Automated quality scoring on a sample of responses - Human review pipeline for edge cases - A/B testing framework to compare model versions

Security - Input sanitization (prompt injection prevention) - PII detection and scrubbing before sending data to external APIs - Audit logging for compliance - Data processing agreement with every AI provider

Cost - Per-request cost tracking - Daily/weekly cost alerts with anomaly detection - Usage dashboards visible to engineering and business - Automatic throttling if spend exceeds budget

This checklist is not optional. Every item exists because I have seen the failure mode it prevents.