LLM Fine-Tuning and Dataset Curation

Pathrule2 Rules • 3 Memories • 2 Skills

Fine-tuning changes model behavior through a dataset and training recipe, so provenance, consent, privacy, formatting, deduplication, train-test leakage, base-model identity, checkpoint selection, and serving compatibility become release-critical inputs. This pattern constrains dataset admission and evaluation separation, records objective and checkpoint decisions, and supplies separate dataset and training-release procedures. It complements RAG and LLM eval patterns by focusing on changing model weights; retrieval changes context at inference time, while evaluation measures behavior without authorizing training data.

Suggested path map

Pathrule places each piece on the matching path, so your assistant only sees it where it belongs. This is the scoping you get on import; you can adjust it in your workspace.

/ workspace root

curate-fine-tuning-dataset

release-fine-tuned-model

datasets/

Admit training examples through provenance and privacy review

evals/

Keep evaluation data outside training and selection leakage

Checkpoint selection is a multi-objective release decision

training/

The training objective matches the production interaction

An adapter is coupled to one base-model identity

Rules

Admit training examples through provenance and privacy review/datasetshighstrictRecord source, rights, consent, purpose, retention, sensitivity, and allowed transformations before data enters a training corpus.

1	Training can memorize and redistribute patterns from data, and model weights are harder to delete or inspect than an ordinary record. A convenient internal log or scraped source is not automatically authorized training material.
2
3	- Assign every source a provenance record and documented legal or product basis for the intended model and distribution.
4	- Remove secrets, credentials, private keys, personal identifiers, unnecessary user content, and restricted material before formatting examples.
5	- Honor deletion, retention, geographic, customer, and licensing constraints through dataset versions and derived artifacts, not an informal one-time export.
6	- Quarantine uncertain sources and fail the dataset build when an example lacks the required provenance or policy fields.
7
8	See /training for the adjacent decision or procedure that completes this constraint.

Keep evaluation data outside training and selection leakage/evalshighstrictDeduplicate across splits and prevent prompts, references, and hidden labels from entering training, tuning, or manual prompt iteration.

1	A model can appear improved by memorizing evaluation examples or near duplicates. Leakage also occurs when developers repeatedly tune prompts and checkpoint choice against the final test set.
2
3	- Split by source, entity, time, conversation, or other leakage boundary before creating transformed examples.
4	- Run exact and semantic duplicate checks across train, validation, test, and benchmark sources and review high-similarity clusters.
5	- Use validation data for training and checkpoint decisions and reserve a sealed test set for final release comparison.
6	- Track who accessed hidden references and regenerate compromised test slices rather than accepting an inflated metric.
7
8	See /datasets for the adjacent decision or procedure that completes this constraint.

Memories

The training objective matches the production interaction/trainingFormat examples with the exact roles, templates, tools, outputs, and loss boundaries required by the served task.

1	A dataset of plausible text is not enough. If production uses chat templates, tool calls, structured output, or selective assistant loss, the training representation must preserve those semantics.
2
3	- Pin the tokenizer and chat template from the base model and render training examples through the same canonical formatter used for evaluation.
4	- Define which tokens contribute to loss and verify user instructions, tool outputs, padding, and control tokens are masked or weighted as intended.
5	- Preserve multi-turn context only where it represents production behavior and remove accidental system or hidden annotation leakage.
6	- Validate serialized examples by decoding samples and exercising the served model interface, not only by checking dataset columns.
7
8	See /datasets for the rule or workflow that puts this decision into practice.

An adapter is coupled to one base-model identity/trainingRecord base revision, tokenizer, architecture, target modules, precision, rank, and merge state with every checkpoint.

1	Parameter-efficient adapters rely on the exact structure and weights they modify. Loading one on a similarly named but different base revision can fail or silently change behavior.
2
3	- Pin immutable base-model and tokenizer revisions and store their checksums or repository identities with the run.
4	- Record adapter method, target modules, rank, scaling, dropout, precision, quantization, and library versions.
5	- Distinguish adapter-only, merged, and quantized deployment artifacts and test the exact one production will load.
6	- Expose combined model identity in serving and retain the prior compatible base and adapter pair for rollback.
7
8	See /evals for the rule or workflow that puts this decision into practice.

Checkpoint selection is a multi-objective release decision/evalsChoose from held-out task quality, regressions, safety, format adherence, latency, size, and serving stability rather than minimum training loss.

1	Training loss measures fit to the training objective. It can improve while generalization, calibration, safety, long-context behavior, or product constraints regress.
2
3	- Evaluate each candidate on fixed held-out tasks and critical slices with identical prompts, decoding, tools, and runtime.
4	- Compare the base model and prior release, not only neighboring checkpoints from the same run.
5	- Include refusal, privacy, memorization, structured-output, tool, and adversarial cases appropriate to the data and intended use.
6	- Review qualitative failures and operational cost before selecting a checkpoint, and document accepted regressions explicitly.
7
8	See /training for the rule or workflow that puts this decision into practice.

Skills

curate-fine-tuning-dataset/rootBuild a versioned training dataset through provenance, privacy, formatting, deduplication, split, and quality evidence.

1	---
2	name: curate-fine-tuning-dataset
3	description: Create or revise a dataset for language-model fine-tuning.
4	---
5
6	# Curate Fine Tuning Dataset
7
8	Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result.
9
10	1. Define the behavior objective, target distribution, exclusion policy, source rights, privacy constraints, and acceptance metrics before collecting examples.
11	2. Ingest through provenance records, remove prohibited data, normalize schema, and retain source-to-example lineage without exposing restricted content broadly.
12	3. Deduplicate within and across sources, resolve conflicting examples, and inspect length, language, topic, label, tool, and quality distributions.
13	4. Split by the correct leakage boundary, seal the test set, and render examples through the pinned tokenizer and production-compatible template.
14	5. Version the manifest, code, source identities, transformations, statistics, approvals, and hashes so the dataset can be reproduced or withdrawn.
15
16	## Exit criteria
17
18	The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.

release-fine-tuned-model/rootReproduce training, evaluate candidate checkpoints, package serving artifacts, canary behavior, and retain rollback.

1	---
2	name: release-fine-tuned-model
3	description: Release a fine-tuned model or adapter into an inference service.
4	---
5
6	# Release Fine Tuned Model
7
8	Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result.
9
10	1. Resolve immutable base, dataset, code, recipe, random seeds, hardware, libraries, tokenizer, template, and output-artifact configuration.
11	2. Train with checkpoint, metric, resource, instability, and interruption evidence and reproduce a small run before spending the full budget.
12	3. Evaluate candidates against base and prior release on held-out quality, slices, safety, privacy, format, tools, latency, memory, and serving errors.
13	4. Package the exact adapter or merged artifact with base identity and load it in the production runtime for compatibility and capacity tests.
14	5. Canary bounded traffic with attributable model identity, monitor regressions and feedback, and roll back both artifact and serving configuration on acceptance failure.
15
16	## Exit criteria
17
18	The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.

Why this pattern

AI agents often train on unlicensed or secret data, leak evaluation examples into training, compare checkpoints with changed prompts, or deploy an adapter against a different base model.

Built for AI teams adapting language models for domain behavior, formats, tools, style, or specialized tasks.

Keeps your assistant from:

Training on data without documented rights or retention
Inflating evaluation scores through train-test contamination
Serving an adapter on the wrong base-model revision
Losing the dataset and recipe needed to reproduce a checkpoint

View as markdown

License: Apache-2.0
Version: 1.0.0
Updated: 2026-08-25