The Agent's First Day: Benchmarking Learning, Exploration, and Scheduling in the Workplace Scenarios

🍞 Hook: Imagine it’s your first day as a school helper. Teachers give you jobs at random times—put up posters, sort books, and run to the office before the bell. Some instructions are missing, so you must ask questions or look around. You also need to remember what you learned yesterday so you don’t repeat mistakes.

🥬 The World Before: For years, AI models—especially multimodal large language models (MLLMs)—got really good at solving fixed problems where everything was known in advance. Benchmarks usually gave a single, clear task with all the info provided, and the AI solved it in one go. This proved best-case skill, but not reliability when life gets noisy and tasks overlap.

🍞 Anchor: Think of a math worksheet with all problems on one page and plenty of time. That’s what many old benchmarks felt like—orderly and complete.

🍞 Hook: You know how class can get chaotic? A friend needs help, the bell rings, and a teacher changes the plan. You juggle it all by planning, asking smart questions, and remembering what worked before.

🥬 The Problem: Real workplaces are dynamic. Tasks arrive as a stream, deadlines collide, crucial info is hidden in files or with coworkers, and new tasks appear mid-meeting. Today’s agents often hallucinate (confidently guess), forget context after switching tasks, and fail to reuse experience.

🍞 Anchor: It’s like being told “Plan Friday’s event,” but the event date is in a calendar on the wall, the budget is in a spreadsheet, and the room is booked unless you check first.

🍞 Hook: Picture a chef who has to cook multiple dishes while deliveries arrive late and recipes are missing. They must reorder steps, ask the supplier, and remember last time’s fix.

🥬 Failed Attempts: Many prior agent tests were static, fully visible, and one-shot. Agents looked good when everything was provided, but broke when info was missing or when a second task interrupted the first. Tool-use tests helped, but most still didn’t measure exploration (asking, looking) or learning across days.

🍞 Anchor: Practicing only on practice tests with answer keys doesn’t teach you how to handle pop quizzes with missing pages.

🍞 Hook: You know how coaches make drills that feel like real games—timers, defenders, and noise—so players can perform under pressure?

🥬 The Gap: We needed a benchmark that (1) streams tasks over time with priorities and deadlines, (2) hides necessary clues so the agent must explore and ask, and (3) repeats patterned tasks across days so the agent can learn general strategies, not just memorize examples.

🍞 Anchor: It’s like a school day simulator: announcements interrupt you, some instructions are on posters in the hallway, and you compare yesterday’s mistakes to do better today.

— New Concepts —

🍞 Hook: Imagine a super student who can read text, look at pictures, and talk. 🥬 Multi‑modal Large Language Models (MLLMs): What: MLLMs are AIs that understand and generate multiple kinds of data (like text and images) and can use tools. How: They read inputs (text/images), plan steps, call tools (e.g., calendars, files), and produce answers. Why: Without MLLMs, agents can’t handle real offices where info is in emails, images, and apps. 🍞 Anchor: Like a student who reads a memo, checks a map, and sends an email.

🍞 Hook: You know how teachers give pop-up chores during class? 🥬 Dynamic Task Scheduling: What: Organizing incoming tasks on the fly by priority and deadline. How: Track all tasks → pick what matters now → pause and resume as things change. Why: Without it, agents miss deadlines and lose context when interrupted. 🍞 Anchor: Choosing to grab the attendance sheet now because the bell rings in 2 minutes.

🍞 Hook: When you don’t know an answer, you raise your hand or look it up. 🥬 Active Exploration: What: Purposefully seeking missing info (ask, search, open files) instead of guessing. How: Notice uncertainty → plan questions → use tools/NPCs → verify clues. Why: Without it, agents hallucinate and break tasks. 🍞 Anchor: Asking the librarian where the science books are before writing a report.

🍞 Hook: Remember how you improve at a game by learning patterns? 🥬 Continual Learning: What: Learning over time without forgetting, turning experience into general strategies. How: Get feedback → reflect/summarize → store lessons → apply next time. Why: Without it, agents repeat the same mistakes. 🍞 Anchor: After missing the bus yesterday, you leave earlier today.

🍞 Hook: Imagine a realistic “first day at work” simulator where jobs arrive during the day, key info is hidden in files or with coworkers, and meetings interrupt your plans.

🥬 Aha! Moment (in one sentence): Test agents the way real work happens—streamed, partially hidden, and changing—so we measure scheduling, exploration, and learning, not just final answers.

🍞 Anchor: It’s like a school day simulation that checks if you can plan, ask for missing info, and get better tomorrow.

— Three Analogies —

1. Theme park day: New rides open, lines change, and a friend holds the map. Great performance means reordering plans, asking staff, and remembering shortcuts.
2. Kitchen rush: Orders stream in, ingredients are missing, and ovens ping mid-recipe. You reprioritize, call the supplier, and reuse last week’s prep tricks.
3. Detective case: Clues are scattered, suspects appear mid-interview, and you must recall patterns from old cases.

— Before vs After —

• Before: Benchmarks were like neat worksheets—single-shot, all info given.
• After: Trainee-Bench is like a live school day—tasks stream in, some info is hidden, and you’re graded along the way.

— Why It Works (intuition) —

• Randomization prevents memorizing shortcuts.
• Hidden clues force active exploration instead of guessing.
• Timed, overlapping tasks reveal if agents can switch context and keep track of goals.
• Checkpoints show partial progress and give feedback for learning.

— Building Blocks (with Sandwiches) —

🍞 Hook: Think of “game levels” that produce countless unique missions. 🥬 Meta‑tasks (rule-based templates): What: Abstract task blueprints (e.g., “plan a meeting”) that can generate many concrete jobs. How: Fill templates with randomized people, files, and data to create fresh instances. Why: Without templates, agents could memorize fixed tasks. 🍞 Anchor: A “write a letter” prompt that changes recipient, topic, and deadline each time.

🍞 Hook: Like rolling different dice each round. 🥬 Randomization Function: What: A controlled generator that varies names, file paths, numbers, and schedules. How: Use a seed to create NPC profiles and environment data; plug them into rules. Why: Without randomness, evaluations repeat and don’t test generalization. 🍞 Anchor: Same recipe, new ingredients and cooking times.

🍞 Hook: Sometimes the instruction sheet is missing a page. 🥬 Partial Observability: What: The task goal is given, but key clues (manuals, passwords, instructions) are hidden. How: The agent must ask NPCs, open folders, and search to reveal clues. Why: Without this, agents just follow the prompt and never learn to explore. 🍞 Anchor: You must ask the office manager for the Wi‑Fi password before you can send the report.

🍞 Hook: Picture a timeline where alarms occasionally force you to switch tasks. 🥬 Dynamic Composite Scenarios: What: Multiple tasks stitched into a day with deadlines, interrupts, and dependencies. How: Assemble task instances along time; ensure names don’t clash; add meetings that preempt work; reveal new tasks during old ones. Why: Without composition, agents aren’t tested on juggling and adapting. 🍞 Anchor: Stop writing to attend homeroom, then resume exactly where you left off.

🍞 Hook: Instead of only checking the final grade, imagine getting credit for each correct step. 🥬 Checkpoints & Automated Verification: What: Milestones that can be automatically checked to score progress. How: Each task has verifiable steps; passing steps increases the score and triggers feedback. Why: Without checkpoints, we can’t see where agents stumble or learn from mistakes. 🍞 Anchor: You get points for finding the right date, more points for booking the room, and feedback if you forgot snacks.

🍞 Hook: Think of three players passing the ball. 🥬 Interaction Protocol (Environment↔Agent↔MLLM): What: A standard loop where the environment offers tools/tasks, the agent plans, and the MLLM produces actions. How: Maintain history → call model → parse tool calls → update state → repeat. Why: Without a clear protocol, different agents can’t be compared fairly. 🍞 Anchor: Teacher (environment) posts tasks, student (agent) plans, calculator (MLLM) computes, then the class list updates.

At a high level: Input (rule library + random seed + tool APIs) → Step A: Instantiate meta-tasks → Step B: Compose a daily scenario with timelines and priorities → Step C: Run the interaction loop (agent ↔ MLLM ↔ environment) → Step D: Verify with checkpoints and produce feedback → Output: Scores, traces, and experience for learning.

Step A — Task Instantiation (What happens):

• Pick a meta-task rule (e.g., “Event Planning”).
• Use the randomization function to generate NPCs (names, roles), files (paths, data tables), and constraints.
• Produce a natural-language task description, a hidden set of clues (e.g., where the manual is), and a list of checkpoints. Why it exists: Prevents agents from memorizing fixed answers; forces general reasoning. Example: “Plan a team-building day under $2,000. Availability is hidden in calendar.csv; travel times in map.json.” Clues (hidden): “Check CloudDisk:/events/manual.md for scoring rules.”

Step B — Build a Dynamic Composite Scenario (What happens):

• Combine 2–6 task instances into one timeline.
• Enforce conflict resolution (unique names/IDs) and define conditional triggers for NPCs who guard multiple clues.
• Add temporal prioritization (e.g., a 10:30 meeting that preempts current work).
• Insert inter-task dependencies (a new assignment revealed mid-meeting). Why it exists: Tests scheduling, context switching, and plan updating under uncertainty. Example: While drafting an ad plan, an urgent meeting pops up; during that meeting, a manager announces a new budget constraint that changes the plan.

Step C — Interaction Protocol (What happens):

• The environment provides: system prompt, tools (OpenFolderInCloudDisk, SendMessage, CalendarAPI), and current time.
• The agent maintains history, queries the MLLM for the next action, and executes tool calls.
• Partial observability means the agent must explore: open folders, ask NPCs, and verify results. Why it exists: Standardizes how agents act and think so comparisons are fair. Example with data: The agent calls OpenFolderInCloudDisk({path: "CloudDisk:/ads_strategy/"}) and finds ads_strategy_handbook.md; then messages Sarah (NPC) to confirm targeting rules.

Step D — Automated Verification & Feedback (What happens):

• Each task has checkpoints like: “Found valid date,” “Booked room,” “Calculated exposure correctly,” “Sent confirmation email.”
• The system auto-checks passed steps and normalizes to a scenario score.
• Each checkpoint emits natural-language feedback for missed items. Why it exists: Measures partial progress and provides learning signals. Example: Feedback says, “You didn’t request the authorization code from the manager,” guiding the agent next time.

Step E — Continual Learning Flow (Optional):

• Day 1: Run the scenario, collect feedback, and summarize lessons (e.g., “Always check CloudDisk:/manuals_for_intern.md before planning”).
• Day 2: Similar tasks are re-instantiated with different details; the agent uses Day 1 lessons to adapt. Why it exists: Tests whether agents extract general strategies, not just memorize. Example: Yesterday the agent missed ‘ask HR who maintains the website’; today it proactively asks HR at the start.

The Secret Sauce:

• Partial observability + randomization make guessing risky and exploration necessary.
• Composite timelines stress-test scheduling and memory.
• Checkpoints provide dense, actionable feedback that fuels continual learning.

Concrete Mini-Case — Advertising Campaign Planning:

• Input: Heatmaps images, channels.csv with cost/exposure, budget.
• Agent must: Identify high-density areas, pick channels to maximize exposure under budget.
• Hidden clue: “Treat as a knapsack optimization; use provided density matrix instead of inferring from images.”
• What breaks without steps: If the agent skips exploration, it may misread data; if it can’t schedule, it misses the budget meeting; without learning, it repeats cost calculation mistakes.

Concrete Mini-Case — Event Planning:

• Input: availability.csv, locations.json, map distances.
• Agent must: Choose a valid date, build an itinerary, and compute distances/time within constraints.
• Hidden clue: Valid date range is buried in the shared calendar; scoring rules are in a manual file.
• What breaks: Missing the valid date invalidates the whole plan; failing to verify distances leads to wrong scores.

The Test: Researchers built 50 dynamic scenarios, each with 2–6 tasks. They measured four things: (1) Success Rate (percentage of tasks finished), (2) Checkpoint Score (how many steps were correctly done), (3) Average Steps (depth of interaction), and (4) Average Tool Calls (tool-use effort). Models followed the same protocol and got summarized history if the context got too long.

The Competition: Seven modern models were tested, including Gemini-3-Flash, Claude-4-Sonnet, GPT-5.1, Grok-4, GPT-4o, Qwen3-VL-235B-A22B, and Llama-4-maverick.

The Scoreboard (with context):

• Gemini-3-Flash led with a 35% Success Rate and a 0.639 Checkpoint Score—like getting a strong B when most are at C levels—while also taking more steps (90) and tool calls (232), showing that thorough exploration helps.
• Claude-4-Sonnet reached a 23% Success Rate and a 0.593 Checkpoint Score—solid but still far from workplace reliability.
• GPT-4o and Grok-4 performed moderately; Qwen3-VL was lower; Llama-4-maverick struggled due to tool-calling errors and instruction-following issues.

Workload Matters: As the number of concurrent tasks rose from 2 to 6, models like Gemini-3-Flash, Grok-4, and GPT-4o declined in Success Rate (e.g., Gemini-3-Flash from 50% to 36%), revealing how context switching and temporal uncertainty challenge agents.

Task Difficulty Hits Hard: Splitting tasks into Easy vs Hard showed sharp drops on Hard tasks. For example, Grok-4 fell from 34% to 7% Success Rate. Gemini-3-Flash still held 32% on Hard tasks, showing better robustness in complex reasoning and exploration but still leaving much room for improvement.

— Sandwich: Metrics — 🍞 Hook: Think of grading both your final answer and each step you got right. 🥬 What/How/Why: Success Rate is the share of fully solved tasks; Checkpoint Score is the fraction of steps passed, rewarding partial progress; Steps and Tool Calls show how much effort/exploration the agent invested. Without these, we can’t tell whether an agent barely tried or carefully explored. 🍞 Anchor: You might not finish the whole project, but you still get credit for picking the right date and booking the room.

Continual Learning Results: Using a leading framework (MUSE) with GPT-4o, Day 2 sometimes got worse overall (Checkpoint Score dropped from 0.42 to 0.36). Why? Because the feedback from Day 1 targeted one missed checkpoint, but Day 2 failed at a different step after randomization—so yesterday’s tip didn’t match today’s need. Interestingly, hard tasks improved slightly (0.20 → 0.24) because Day 1 produced richer lessons, while easy tasks got worse (0.61 → 0.50) due to misleading or sparse experience.

Human Guidance Helps a Lot: Tiered hints on hard tasks boosted average scores from 0.24 (no hints) to 0.83 with three levels of guidance—like moving from a D to an A-. Multiple rounds of self-evolution alone added only +0.04. This exposes current limits in autonomous exploration and precise execution.

Surprises:

• More steps and tool calls correlated with better outcomes for the top model—effortful exploration pays off.
• Continual learning can hurt easy tasks if experiences don’t generalize.
• Even with strong reasoning, precise calculations (e.g., travel distances, budget exposure) are brittle without the right data and verification.

Limitations:

• Scenario complexity: Current task compositions don’t yet include deep causal chains; future versions will add richer dependencies.
• Manual rule authoring: Meta-task rules are handcrafted, which limits scalability; automated rule generation would expand coverage.
• Scope: Experiments are focused on office-like settings and a subset of agent frameworks.

Resources Needed:

• Access to strong MLLMs or APIs, a tool-execution sandbox, and enough compute for long interaction traces.
• Logging and storage for histories, feedback, and experience summaries.

When Not to Use:

• If you only need to test single, fully-specified tasks with no interruptions.
• If your agent cannot call tools or interact with files/NPCs; Trainee-Bench requires exploration.
• If you seek pure final-answer accuracy without process/step analysis.

Open Questions:

• How can agents decide when to explore vs when to act to minimize wasted steps?
• What experience formats (rules, skills, memories) transfer best across randomized tasks?
• Can we design self-supervised signals that replace human hints but still drive big gains?
• How should agents manage long-term memory to survive frequent context switches?
• Which curricula of tasks accelerate robust scheduling and exploration skills?

3-Sentence Summary: Trainee-Bench tests AI agents in realistic, dynamic workplace scenarios where tasks stream in, key clues are hidden, and priorities change. It measures three vital abilities—scheduling, active exploration, and continual learning—using randomized meta-tasks, composite timelines, and checkpoint-based feedback. Results show that even top models struggle, especially on hard tasks and autonomous learning, while human hints dramatically improve performance.

Main Achievement: Shifting evaluation from neat, static tests to a production-style environment with partial observability and temporal complexity—plus fine-grained, feedback-rich scoring.

Future Directions: Add deeper causal chains and broader domains, automate rule generation, refine learning protocols that generalize across randomization, and develop exploration strategies that balance curiosity with efficiency.

Why Remember This: If we want dependable AI coworkers, we must train and test them in the messy reality they’ll face—where information is missing, clocks are ticking, and yesterday’s lessons must guide today’s choices.