Building a Humor Reward Model for LLMs: Lessons from Reddit-Driven Reinforcement Learning + Video

By /

Listen to this Post

Featured Image

Introduction

Teaching large language models to generate genuinely funny responses represents one of the most nuanced challenges in AI alignment today. Unlike factual accuracy or instructional compliance—which can be optimized through straightforward supervised fine-tuning—humor requires contextual sensitivity, cultural awareness, and an understanding of timing and delivery that defies simple prompting strategies. As Dhruv Nigam’s experience at Dream11’s AI Companion project illustrates, telling an LLM to “be funny” yields only marginal improvements, while few-shot prompting hits a performance ceiling. The solution lies in reinforcement learning from human feedback (RLHF), specifically training a reward model that can quantitatively evaluate “funniness” using real-world social interaction data from platforms like Reddit. This article explores the technical architecture, implementation steps, and critical pitfalls of building a humor-oriented reward model, drawing on recent academic frameworks including KARMA (Karma-Aligned Reward Model Adaptation) and CLoST (Creative Leap of Structured Thought).

Learning Objectives & Secrets

  • Objective 1: Build a Reward Model That Predicts Conversational Effectiveness – Train a classifier on Reddit conversation data where comments are labeled as “rewarding” if they outperform their parent comment in engagement metrics (upvotes, replies), using this signal as a proxy for humor and contextual appropriateness. Secret tip: Remove community-specific metadata (subreddit identity, timestamps) from the reward model’s input—while this reduces predictive accuracy on held-out Reddit data, it forces the model to learn generalizable pragmatic signals rather than platform-specific heuristics.

  • Objective 2: Apply PPO-Based Reinforcement Learning for Downstream Alignment – Use Proximal Policy Optimization (PPO) to fine-tune the base LLM using the trained reward model as the optimization signal, encouraging responses that score higher on the humor/reward dimension without direct exposure to Reddit data. Secret tip: Fine-tune on general benign chat data rather than Reddit’s raw conversational distribution—this reduces toxicity and bias amplification while preserving pragmatic benefits.

  • Objective 3: Implement Iterative Self-Correction Through Structured Thought – Beyond reward modeling, implement a structured thinking framework where the model generates multiple candidate responses, evaluates them against the reward model, and iteratively refines its output through chain-of-thought reasoning. Secret tip: Design judgment-oriented instructions and use evolutionary methods to expand the instruction set—this unlocks creative associations that single-pass generation cannot achieve.

You Should Know

  1. Data Curation and Labeling Strategy for Humor Reward Models

The foundation of any effective humor reward model is high-quality training data with meaningful labels. The July 2025 Pushshift Reddit dataset provides a scalable source: approximately 80,000 posts and 400,000 comments across 14,000 communities. The labeling heuristic is elegantly simple—a comment is labeled as “rewarding” if it receives more upvotes than its parent comment, and “not rewarding” otherwise. This approach leverages Reddit’s voting system as an implicit human preference signal, capturing real-world judgments about what constitutes an engaging, funny, or contextually appropriate response.

Step-by-Step Guide:

  1. Extract and Filter Data: Download the Pushshift Reddit dataset. Filter out explicit content, non-text posts, and non-English comments. Retain only threads with multiple high-quality responses.
  2. Construct Conversation Pairs: For each comment thread, create training pairs consisting of (parent comment context, child comment) with a binary label indicating whether the child outperformed the parent in upvotes.
  3. Remove Metadata Leakage: Strip subreddit identifiers, timestamps, and any community-specific signals from the input. Research shows that including this metadata improves reward model F1 and AUC but degrades downstream alignment quality—the model learns to exploit surface-level features rather than genuine pragmatic understanding.
  4. Format as Chat-Like Structure: Standardize inputs into a conversational format (e.g., 
     parent_comment [bash] child_comment</code>) to encourage generalizable learning.</li>
<li>Train the Reward Model: Fine-tune a base model (e.g., LLaMA 1B) as a classifier on this labeled dataset. The model learns to predict the probability that a given response would be perceived as rewarding in context.</li>
</ol>

<h2 style="color: yellow;">Linux Command Example (Data Processing):</h2>

[bash]
 Download and filter Reddit data using Apache Arrow and DuckDB
pip install duckdb pandas pyarrow

Extract and label comments
duckdb <<EOF
CREATE TABLE reddit_data AS 
SELECT  FROM read_parquet('pushshift_2025-07.parquet');

CREATE TABLE training_pairs AS
SELECT 
parent.body AS context,
child.body AS response,
CASE WHEN child.score > parent.score THEN 1 ELSE 0 END AS label
FROM reddit_data parent
JOIN reddit_data child ON child.parent_id = parent.id
WHERE parent.language = 'en' 
AND child.language = 'en'
AND parent.is_deleted = FALSE
AND child.is_deleted = FALSE
AND parent.score IS NOT NULL
AND child.score IS NOT NULL;
EOF

    2. Reward Model Training and Evaluation with PPO

    Once the reward model is trained, the next phase involves using it as the optimization signal for reinforcement learning. Proximal Policy Optimization (PPO) is the standard algorithm here—it updates the policy model (the LLM being trained) in a way that balances exploration and stability, using the reward model’s scores to guide which responses are reinforced. Critically, the downstream model should not be directly exposed to Reddit data during fine-tuning; instead, the reward model serves as a bridge, translating social signals into a generalizable reward function.

    Step-by-Step Guide:

    1. Initialize Policy Model: Start with a pretrained LLM (e.g., LLaMA 3, DeepSeek R1, or Gemma 2.0). Research indicates that fine-tuned LLaMA 3 can generate funnier responses than GPT-4 in some evaluations.
    2. Set Up PPO Training Loop: For each training batch, generate responses from the current policy model given input prompts. Pass these responses through the frozen reward model to obtain reward scores.
    3. Compute Advantage Estimates: Use Generalized Advantage Estimation (GAE) to calculate how much better each response was compared to the policy’s expected performance.
    4. Update Policy with PPO Clipping: Apply the PPO objective function with clipping to prevent destructively large policy updates. Use a KL penalty to keep the policy close to the reference model.
    5. Iterate: Repeat for multiple epochs, monitoring both reward scores and auxiliary metrics (toxicity, factual consistency, reasoning ability).
    6. Evaluate on Hold-Out Benchmarks: Test the aligned model on humor recognition, sentiment classification, and other pragmatics-mediated tasks.

    Python Code Snippet (PPO Training Setup with TRL):

    from trl import PPOConfig, PPOTrainer, AutoModelForCausalLMWithValueHead
from transformers import AutoTokenizer
import torch

Load policy model with value head for PPO
model = AutoModelForCausalLMWithValueHead.from_pretrained("meta-llama/Llama-3-8b")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3-8b")
tokenizer.pad_token = tokenizer.eos_token

Load pre-trained reward model
reward_model = AutoModelForSequenceClassification.from_pretrained(
"your-humor-reward-model"
).to("cuda")

Configure PPO
config = PPOConfig(
model_name="llama-3-8b-humor",
learning_rate=1.41e-5,
batch_size=32,
mini_batch_size=8,
gradient_accumulation_steps=4,
ppo_epochs=4,
cliprange=0.2,
cliprange_value=0.2,
init_kl_coef=0.05,
)

Initialize trainer
ppo_trainer = PPOTrainer(config, model, tokenizer, dataset, data_collator)

Training loop
for epoch in range(config.ppo_epochs):
for batch in ppo_trainer.dataloader:
queries = batch["input_ids"]
response_tensors = ppo_trainer.generate(queries, generation_kwargs)

Get reward scores
rewards = reward_model(response_tensors).logits.squeeze(-1)

PPO step
stats = ppo_trainer.step(queries, response_tensors, rewards)

    3. Handling the Factuality-Pragmatics Trade-off

    One of the most critical findings from KARMA research is that optimizing for humor and conversational engagement consistently degrades factual accuracy. This trade-off appears to be embedded in the reward signal itself—what makes a response funny or engaging is not always what makes it factually correct. Practitioners must therefore implement safeguards: maintain a separate factual consistency classifier during training, use rejection sampling to filter out factually incorrect generations, or implement a hybrid reward that combines humor scores with factual correctness penalties.

    Mitigation Strategies:

    • Use a Dual-Reward Architecture: Combine the humor reward model with a factual consistency model (e.g., based on RAG or factual QA benchmarks). Weight the final reward as R_total = α R_humor + (1-α) R_factual.
    • Implement Factuality Guards: After generation, run responses through a factual verification pipeline. Reject or re-rank generations that contain hallucinated claims.
    • Monitor with Benchmarks: Track performance on reasoning benchmarks (e.g., MMLU, GSM8K) during training. If factual capability drops beyond an acceptable threshold, adjust the KL penalty or reduce the influence of the humor reward.
    1. Avoiding the Metadata Pitfall in Reward Model Design

    A surprising finding from recent research is that reward models with higher predictive accuracy on validation data can produce worse downstream alignment. When reward models are given access to metadata such as subreddit identity or timestamps, they learn to exploit these platform-specific heuristics rather than genuine pragmatic signals. The result is a reward model that performs well on Reddit data but fails to generalize to other conversational contexts.

    Step-by-Step Guide to Robust Reward Model Design:

    1. Define Input Scope: Restrict the reward model’s input to the conversational context only (the parent comment and any preceding thread). Exclude all metadata fields.
    2. Validate on Out-of-Distribution Data: Evaluate the reward model not just on held-out Reddit data, but also on data from other platforms (Twitter, news comments, etc.) to ensure generalizability.
    3. Ablation Studies: Run ablation experiments where metadata is progressively added to the input. Observe the impact on both reward model accuracy and downstream alignment quality.
    4. Monitor for Shortcut Learning: If the reward model achieves suspiciously high accuracy on certain subreddits or topics, investigate whether it has learned superficial patterns rather than genuine understanding of humor.

    5. Iterative Self-Correction Through Structured Thought

    Beyond reward modeling, recent work on CLoST (Creative Leap of Structured Thought) demonstrates that humor generation benefits from an iterative refinement process. Instead of generating a single response, the model produces multiple candidate responses, evaluates them against the reward model, and refines its reasoning through chain-of-thought. This approach mirrors how humans craft humor—through reflection, revision, and creative association.

    Implementation Steps:

    1. Generate Multiple Candidates: For a given prompt, generate 5–10 candidate responses using diverse sampling strategies (different temperatures, top-p values).
    2. Reward Model Scoring: Pass all candidates through the humor reward model. Rank them by reward score.
    3. Chain-of-Thought Refinement: For the top candidate, prompt the model to explain why it is funny and how it could be improved. Generate a revised version incorporating this self-critique.
    4. Iterate: Repeat the refinement process 2–3 times, each time using the reward model to guide the selection.

    5. Final Selection: Output the highest-scoring refined response.

    What Undercode Say:

    • Key Takeaway 1: Humor cannot be effectively taught through prompting alone—few-shot learning hits a ceiling, and reward modeling with RLHF is the only scalable path to genuinely funny LLM responses. Reddit’s voting data provides a uniquely rich and scalable source of implicit human preference signals, capturing real-world judgments about what constitutes engaging, contextually appropriate humor.

    • Key Takeaway 2: The most accurate reward model is not necessarily the best for downstream alignment. Removing metadata and restricting inputs to conversational context reduces predictive accuracy on held-out data but produces substantially better generalization and downstream performance. This counterintuitive finding underscores the importance of designing reward models for alignment quality, not just predictive metrics.

    Analysis: The Dream11 AI Companion project illustrates a broader trend in LLM development: the shift from instruction-tuned models to reinforcement learning from human feedback as the primary mechanism for imbuing models with subjective, context-dependent capabilities like humor, empathy, and conversational nuance. However, the factuality-pragmatics trade-off remains an unsolved challenge—optimizing for engagement often comes at the cost of factual accuracy, and practitioners must implement careful safeguards to prevent their models from becoming entertaining but unreliable. The KARMA framework’s finding that reward model design choices (metadata inclusion/exclusion) dramatically impact downstream behavior offers a crucial lesson for the field: alignment quality is not a monotonic function of reward model accuracy, and careful, deliberate design is essential.

    Prediction:

    • +1 The integration of Reddit-driven reward models will become standard practice for conversational AI systems in entertainment, gaming, and social applications over the next 12–18 months, with companies like Dream11 leading the way in productionizing these techniques.

    • -1 The factuality-pragmatics trade-off will create significant reputational risks for organizations deploying humor-optimized LLMs without adequate factual guards, potentially leading to high-profile incidents of entertaining but harmful misinformation.

    • +1 Iterative refinement frameworks like CLoST will evolve into the dominant paradigm for creative LLM applications, enabling models to generate humor that rivals human comedians in wit and contextual appropriateness.

    • -1 The reliance on Reddit data as a preference signal introduces demographic and cultural biases that may limit the generalizability of humor-optimized models across different user populations and geographic regions.

    • +1 Open-source implementations of humor reward models and PPO training pipelines will accelerate research and democratize access to these techniques, fostering innovation across the AI community.

    ▶️ Related Video (84% Match):

    https://www.youtube.com/watch?v=2HPKmgmi50E

    🎯Let’s Practice For Free:

    🎓 Live Courses & Certifications:

    Join Undercode Academy for Verified Certifications

    🚀 Request a Custom Project:

    Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
    [email protected]
    💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands

    IT/Security Reporter URL:

    Reported By: https://lnkd.in/p/eVSVTbry - Hackers Feeds
    Extra Hub: Undercode MoN
    Basic Verification: Pass ✅

    🔐JOIN OUR CYBER WORLD [ CVE News • HackMonitor • UndercodeNews ]

    💬 Whatsapp | 💬 Telegram

    📢 Follow UndercodeTesting & Stay Tuned:

    𝕏 formerly Twitter 🐦 | @ Threads | 🔗 Linkedin | 🦋BlueSky

Related Posts: