Record AR-SIM-063 · Data Intake and Validation
Every conclusion on this site
is downstream of one file.
The Data Laboratory is the module that turns a trade history into something an engine is allowed to reason about. This page documents the intake pipeline in full: what is inspected, what is mapped, what is validated, what is normalised, what is excluded, and what is recorded so that any figure can be traced back to the rows that produced it.
Nothing on this page uploads anything
There is no file input, no upload endpoint and no processing of your data anywhere on this site. The mapping station below runs against a fixed demonstration export whose contents are described openly. Nothing is uploaded, processed or transmitted. The interactive part is honest in scope: changing a column mapping changes the validation states the page displays, because that relationship is arithmetic and can be shown truthfully.
A file of trades
is not yet evidence.
The gap between "I have my history" and "I have a dataset" is where most retail analysis quietly fails. Not because the numbers are wrong, but because nobody established what the numbers were before averaging them.
A broker export is an accounting artefact. It exists to reconcile a balance, not to support research, and its structure follows from that purpose. Rows are ordered by close time rather than open time. Partial closes appear as separate tickets with no link to the original position. Financing is applied on a schedule unrelated to the trade. The Profit column is net of some costs and gross of others, and which ones depends on the broker.
A journal has the opposite failure. It is structured for research and unreliable as a record, because it was written by the person whose behaviour it describes. Entries thin out during losing periods — the exact population the analysis needs most.
Neither is unusable. Both are unusable as supplied. Validation is the work of establishing what each row actually asserts, what it omits, and whether a measurement built on it would survive contact with someone asking how it was calculated.
The standard the pipeline works to
A dataset is validated when every row either carries a complete analytical record or carries a written reason for its absence.
That is a deliberately low bar and almost no exported history clears it on the first attempt. It is low because it makes no claim about quality — only about accounting. Every row is present in the ledger, in one of two states, and the count of both is reported next to any result derived from them.
The number that moves most
In practice the single correction that changes conclusions most is fee decomposition. A framework with a modest edge per trade and unaccounted round-turn costs can move from convincingly profitable to negative on the same rows, with no other change.
Ten steps,
in one direction.
The pipeline is ordered so that no step can be completed on assumptions the previous step has not established. Each stage carries an honest functionality level: what is built now, what is a preview, and what is intended production capability.
Steps 01–04 · establishing what you have
Nothing is interpreted yet. The file is read as bytes, resolved into rows and columns, and proposed against the analytical schema. A proposal is not a mapping — the confirmation step exists because header names lie routinely, and a column called Profit means at least four different things across brokers.
Steps 05–08 · establishing what is usable
Validation, normalisation and the exclusion register. This is where the dataset shrinks, and where the pipeline earns its place: every removed row is named, counted and attributed to a rule, so the shrinkage is a finding rather than a silent adjustment.
Steps 09–10 · making it repeatable
A saved mapping turns a recurring chore into an import. The same broker, the same export format and the same corrections apply next quarter without repeating the judgement calls — and, more importantly, without applying different judgement calls to the next batch.
The mapping station,
shown against a fixed file.
A demonstration export stands in for the real pipeline. Its contents are described below in the same detail the inspection step would report, and the mapping table beneath is live: change a target field and the validation states, exclusion counts and quality score recompute.
Preview — at the point of display
Everything in this section is a demonstration. The file named below is a fictional export constructed for this page; no file is uploaded, read, processed, stored or transmitted by this site, and no upload control exists. The numbers are internally consistent illustrations of what the inspection and validation steps would report.
Source file
DemonstrationRow quality score
Quality = (clean + 0.6 × warned) ÷ total rows
Warned rows count at reduced weight because they remain analysable while carrying a known distortion. Excluded rows count as zero and are reported separately — they are never removed from the denominator, because hiding them would make a damaged file look clean.
Column mapping · 15 of 17 detected columns shown
Interactive preview| Source column | Analytical field | Detected type | Status |
|---|
Five rows carry a select. The rest were detected unambiguously and are shown as confirmed; the two columns not listed here — a running balance and an empty tax column — carry no trade data.
Resulting validation states
Simulated demonstration data
Fourteen fields,
seven of which are not negotiable.
The analytical schema is deliberately small. Every field earns its place by unlocking a specific family of measurements, and a field that unlocks nothing is not collected — which is why there is no column for trade rationale sentiment or screenshot links.
| Field | Requirement | Type | What it unlocks |
|---|
Derivable is not the same as optional
Risk amount and session are marked derivable rather than optional. If the inputs they derive from are present, the pipeline computes them; if not, they must be supplied directly. A derivable field with no derivation path is a required field that is missing, and it is reported as one.
MAE and MFE are almost never present
No mainstream broker export contains excursion data, because the broker has no reason to record it. Without MAE and MFE the entire execution-quality family — capture efficiency, giveback, stop consumption, asymmetry — cannot be computed from history alone. They have to be captured live by the journal or reconstructed from price data, which is a separate problem with its own error terms.
Nine states,
three severities.
Every row leaves validation carrying exactly one state per rule it touched. Blocking states remove the row from analysis; warnings keep it and mark the distortion; flags keep it, keep it visible, and refuse to decide on the trader's behalf.
| State | Severity | What causes it | What to do |
|---|
Blocking
The row cannot be converted into an analytical record at all. It stays in the ledger with its reason attached and is counted in every disposition report, but it contributes to no measurement.
Warning
The row is usable and known to be distorted in a specific direction. Warnings propagate: a result computed from warned rows carries the warning count alongside it rather than presenting as clean.
Flag
The row is implausible but not invalid — a result far outside the expected range, which may be the single most informative trade in the file or a data-entry error. It is surfaced for a human decision and never removed automatically.
Exclusions are a result,
not housekeeping.
A pipeline that silently drops rows produces a cleaner dataset and a worse conclusion. Every exclusion here is counted, attributed to a named rule, and reported alongside anything computed from the rows that survived.
How the score is meant to be read
A quality score is a summary, and summaries are dangerous in exactly the way this module exists to prevent. It is reported next to the disposition counts precisely so it cannot travel alone: a score of ninety on a file with two hundred rows and one on a file with two thousand are not comparable statements.
It is also not a grade for the trader. A file scoring in the seventies usually reflects an export format, not undisciplined record-keeping — and the correct response is to fix the export or the mapping, not to feel judged by a number.
The exclusion register
Every excluded row is retained with four attributes: its source line number, the rule that excluded it, the value that triggered the rule, and whether a human overrode the exclusion. A dataset can therefore be recomputed with a rule relaxed, and the two results compared, without re-importing anything.
When exclusions are not random
The most dangerous exclusion pattern is a correlated one. If blank stop prices cluster in a particular month, or on a particular instrument, removing those rows removes a population rather than noise — and the surviving dataset is biased in a direction nobody chose. Disposition is therefore reported by month and by instrument, not only in total.
Four transformations
that make rows comparable.
Validation establishes that a row is usable. Normalisation establishes that it is comparable to every other row — across time, across instruments, across account sizes and across the years in which the account grew.
Transformation 01
Timezone alignment
Broker timestamps are recorded in server time, which is usually neither UTC nor the trader's local time, and which frequently shifts by an hour twice a year without announcement. Every timestamp is converted to UTC, then attributed to a session using exchange hours rather than a fixed clock offset.
The check that catches most errors is a distribution test: if trade activity appears to begin exactly one hour earlier for part of the year, the offset is wrong and every session conclusion drawn from it is wrong with it.
Transformation 02
Instrument naming
The same market arrives as EURUSD, EUR/USD, EURUSD.pro and EURUSD_i depending on the account type and the year. Unresolved, these appear as four instruments with small samples instead of one with an adequate one.
Names are resolved to a canonical symbol carrying contract size, tick value, quote currency and the pip definition used for friction. Suffix variants are mapped rather than stripped, because a suffix sometimes denotes genuinely different execution terms.
Transformation 03
Fee decomposition
A single Profit figure is separated into gross result, commission, financing and — where the entry price and quoted price both exist — the spread paid on entry. Each component is then expressed in R against the working stop so it can be subtracted from expectancy directly rather than discussed in currency.
Financing is the awkward one. It accrues by calendar day, not by trade, so a position held over a weekend carries a cost unrelated to its outcome. It is attributed to the trade, marked as holding-period cost, and reported separately in any friction breakdown.
Transformation 04
R derivation
R is the unit everything else is expressed in. It is derived per row, not applied as an account average, so a period of larger positions does not silently dominate the expectancy figure.
Risk = |entry − stop| × size × tick value ; R = net result ÷ Risk
Where a stop was never placed, or was placed after entry, no risk amount exists for that row and no R can be derived from it.
The single most common gap in broker exports
R cannot be derived without a risk amount, and a risk amount cannot be derived without a stop. Exports from accounts traded without a resting stop — managed manually, or closed on a mental level — contain no path to R at all. Those rows are not merely lower quality; they are structurally unanalysable in R space, and every expectancy, RAER and MAE measurement in the Foundry is unavailable for them. The remedy is upstream and unglamorous: place the stop with the order, and record the intended risk at the moment the position is opened.
Any figure should lead back
to the rows that made it.
Provenance is what separates a research environment from a calculator. The transformation log is designed so that a result produced six months ago can be reconstructed exactly, or shown to be irreproducible — which is itself a finding worth having.
Why the mapping is versioned
The mapping is a set of interpretive decisions — that this column is the stop, that this suffix is the same instrument, that this comment field carries the branch tag. Those decisions change results, so they are recorded with the same seriousness as the data.
When a mapping is revised, prior results are not silently updated. They keep the mapping version they were computed under, and the difference between versions can be inspected directly. A conclusion that changes when the mapping changes was never a conclusion about the market.
Reproduction test
The design target is simple to state and hard to meet: given the source file, the mapping version and the pipeline version, re-running the import must produce a byte-identical analytical dataset. Anything that prevents that — a non-deterministic sort, a locale-dependent number parse, a timestamp resolved against "now" — is a defect rather than a quirk.
What is deliberately not recorded
Account numbers, broker credentials, balances and any identifying detail are outside the analytical schema. Nothing in the pipeline needs them, so nothing in the pipeline is designed to hold them.
Seven intended sources,
at three levels of readiness.
Each source arrives with a different shape, a different failure mode and a different amount of work before it becomes a specimen. They are listed with an honest readiness level rather than as a feature list.
How this page was built,
and what it does not do.
The methodology matters here more than on most pages, because a data module that overstates its own capability is failing at the exact thing it claims to be for.
Methodology
The demonstration export is a constructed specification, not a real file: a row count, a size, an encoding, a delimiter, a column list and a set of defect counts, chosen to be internally consistent and to resemble the shape of a common retail export. Every number displayed in the mapping station is derived from that specification by arithmetic in the page script.
The mapping interaction is limited to what can be shown truthfully. Changing a target field changes which validation rules can be satisfied, and that relationship is deterministic — a missing stop column means no risk amount, which means no R, which means the affected rows cannot be scored. The page computes that chain and displays it. It does not simulate parsing, because nothing is being parsed.
The three figures are drawn from the same disposition arithmetic and recompute when the mapping changes, so no figure can disagree with the readout beside it.
No file handling exists
This phase contains no upload control, no parser, no storage and no transmission. The full pipeline is a planned capability of the application, not a feature of this website.
The demonstration file is idealised
Real exports are worse. They contain merged cells from a spreadsheet round-trip, footer summary rows, thousands separators inside numeric fields and occasionally a second table below the first.
Validation cannot detect honesty
Every check here is structural. A file can pass all of them and still describe a history with trades removed, sizes edited or a period omitted. No pipeline resolves that; only the discipline of the person exporting the file does.
Quality scores are not comparable across files
The score describes one file against its own row count. Comparing scores between two exports of different sizes, formats or brokers is meaningless and is not offered anywhere in the interface.
Next
A validated dataset is the entry condition, not the destination.
Once a history has been mapped, validated and normalised it becomes a specimen: something the simulation, regime, execution and governance modules are permitted to reason about. If you want to be notified when the intake pipeline moves from preview to production, the research register is where that is recorded.