Skip to content

Validation

Every stage is tested on data it was neither estimated nor calibrated on. Following Roorda, Miller and Habib (2008), verification compares a model with the data it was estimated on and validation with data used in no estimation or calibration step. Each stage page shows the headline numbers, the main figure and every comparison in detail.

Stage Tested against
Synthetic population ACS PUMS 2014–18, held-out PUMS test split, travel survey
Land use ACS, Census estimates, JOLTS, permits, county flows; out of sample 2021–24
School location Survey students, ACS enrolment
Activities Survey hold-out half; ATUS, ACS 2024, LODES
Traffic MDOT SHA 2024 counts, TMAS continuous counts, screenlines, MDOT SHA mileage
County view Synthetic population by county, all 24 Maryland counties

Traffic is compared as a ratio, because the assignment carries residents only: simulated passenger-vehicle volumes are 0.81 of the 2024 counts at 2,308 mainline stations (R² 0.90 across stations).

Strategy

Each stage is estimated on one set of data and tested on another:

Stage Estimated on Tested on
Synthetic population ACS PUMS 2012–16, training split Held-out test split; ACS 2014–18, Maryland and 96 PUMAs
Land use ACS PUMS movers 2014–18 and households; survey estimation half; calibrated on 2017–20 Held-out PUMS movers; ACS and Census 2021–24 (out of sample)
Schools Survey estimation half Survey hold-out half; ACS enrolment
Activities Survey estimation half (10,889 households) Survey hold-out half (10,899 households); chain test on the synthetic population; ATUS, ACS and LODES for 2024
Traffic Nothing estimated; network from fixed rules 2024 counts, continuous counters, screenlines, mileage

The 2017/18 Regional Travel Survey is split at random into two halves by household, stratified by home county and household size. The hold-out half enters no estimation or calibration step. TASHA was validated on a later survey of the same region; no later diary exists here, so the hold-out half and the 2024 benchmarks take its place (details).

Measures and criteria

Stage Measure Criterion (source) Runs
Population synthesis Marginals and joint tables against the PUMS test split SRMSE at the sampling floor (own) 1
Land use, consistency County population; employment; vacancies; construction Imposed; county population ±2 % (own) 5 seeds
Land use, tests Event rates; household size; car ownership; moves and mover flows; commuting Rates ±5 %; shares ±5 pp (own) 5 seeds
Location choice Hold-out log-likelihood; trip-length distribution (KS); shares by area KS below its 5 % critical value; mean within 5 % (FHWA, 2010) –
Activities Activities by purpose; start-time and duration profiles (KS on hourly values); trips per chain Start times within 2 pp per hour (Roorda et al., 2008) 10 replications
Assignment AAWDT by facility and volume group; hourly counts %RMSE by volume group (FDOT, 2008; Wegmann, 2016); GEH < 5 for > 85 % of flows (DfT) 5 seeds

Own: threshold chosen for this study where no published guideline exists. SRMSE: standardised root-mean-square error; KS: Kolmogorov–Smirnov distance; pp: percentage points; GEH: Geoffrey E. Havers statistic of hourly flows. Location choice is tested on the survey hold-out half, which involves no simulation.

The land-use results are the mean and range of five seeds. The activity model is run with ten replications on each population; for 2024 on the land-use seed closest to the mean of the five, with a 10 % household sample, and the assignment receives the replication closest to the mean.

What the differences mean is stated in Known limitations.