Synthetic loan portfolio environments for training and evaluating AI agents

What Credit Gym offers

Credit Gym is a set of environments in which an AI agent is given the materials a debt buyer receives from a seller and is asked to make the decisions a credit analyst would make. The portfolios are generated by a simulator, so the true position of every account is known and every answer can be graded exactly.

The tapes are deliberately imperfect. Dates arrive in several formats, balances are stored as text, status codes follow legacy conventions and some accounts cannot lawfully be enforced. An agent that accepts the seller’s data at face value scores poorly.

Intended users

Evaluation teams

To measure how reliably a model handles messy financial data and domain rules, with scores that do not depend on a model grader.

Post-training and reinforcement learning teams

To train agents on tasks with exact, automatically computed rewards and fresh held-out sets.

Agent and tool developers

To test spreadsheet, code and analysis agents on a realistic multi-step analytical workflow.

Task types

How the environments are constructed

1. Ground truthAccounts, balances, defaults and legal events are simulated so that the true position of every account is known.
2. Behaviour modelPayment and collection behaviour is simulated over time, which produces the static pool history.
3. Data degradationThe tape is then degraded as real seller data often is, with mixed date formats, balances held as text, legacy status codes, duplicates and free-text notes.
4. GradingAnswers are scored deterministically against the hidden ground truth, with partial credit and with asymmetric penalties where a credit desk would apply them.

Rules reflected in the tasks include the Limitation Act 1980, the Prescription and Limitation (Scotland) Act 1973, IVAs, trust deeds and bankruptcy, county court judgments and static pool collections analysis. The rules are simplified for evaluation purposes and are set out in each prompt. They do not constitute legal advice.

Task explorer

The twenty version 0 tasks form the current sample set. All data is synthetic. Answer keys are withheld and used only for grading.

Delivery and licensing

Credit Gym is offered in three ways. In each case the simulator remains with Neurithmetic. Answer keys are not shipped. Answers are scored with Neurithmetic graders under arrangements agreed with each customer.

Format

cg-v0-016-price/
├── prompt.md             task instructions
├── tape.csv              seller’s loan tape
├── data_dictionary.md    seller’s data dictionary (incomplete)
└── history.csv           static pool collections (forecast and pricing tasks)
grade.py                  deterministic grader: scores a JSON answer from 0 to 1

Agents read the files and return a single JSON answer, so the tasks can be run in most evaluation harnesses. Delivery in other environment formats can be discussed.

How tasks are scored

Each task has a hidden answer derived from the simulator’s ground truth. The graders are deterministic Python functions and do not call a language model.

Score scale

Scores range from 0 to 1. Tasks that require a list of accounts (tape cleaning and exclusions) are scored on precision and recall against the hidden set. Numerical answers such as face value and collections forecasts are scored on relative error, with full credit within a tolerance band. Pricing tasks penalise overpayment twice as heavily as underpayment, in line with the approach of a buyer’s risk function.

Validation of the graders

Each release is checked in three ways. A perfect answer must score 1.00. A naive answer, which accepts the seller’s tape at face value, must score low (a mean of 0.16 on the version 0 set). A rule-based reference solver must score between the two.

Protection against memorisation

Answer keys are never published. New tasks are generated from fresh random seeds, so a held-out set can be produced for each evaluation.

Use of real data

No real data is used. Every account, balance and customer reference is synthetic. Postcode districts use real UK area prefixes (for example EH, G or M) so that the different rules in England and Wales and in Scotland can be tested. They are not linked to any individual.