Evaluation

Evaluation Metrics

For each trajectory segment, participants are required to submit only the predicted longitudinal positions of the following automated vehicle (FAV) over the prediction horizon. The organizing team then derives all other evaluation variables, including speed, acceleration, jerk, spatial headway, spatial gap, and time-to-collision (TTC), from the submitted FAV trajectory and the provided lead-vehicle (LV) trajectory.

The evaluation is based on three aspects: Accuracy, Safety, and Comfort.

The final submission score is defined as the average per-segment total score across all evaluated trajectory segments.

Notation

Variable	Description
$i \in \mathcal{I}$	Trajectory-segment index
$t \in \mathcal{T}_i$	Discrete time index for trajectory segment $i$
$\Delta t$	Simulation time interval
$x_{it}^{l}$	Provided longitudinal position of the lead vehicle at time $t$
$\hat{x}_{it}^{f}$	Predicted longitudinal position of the FAV at time $t$
$x_{it}^{f}$	Reference longitudinal position of the FAV at time $t$
$v_{it}^{l}$	Speed of the LV derived from the provided trajectory
$\hat{v}_{it}^{f}$	Predicted speed of the FAV
$v_{it}^{f}$	Reference speed of the FAV
$\hat{a}_{it}^{f}$	Predicted acceleration of the FAV
$a_{it}^{f}$	Reference acceleration of the FAV
$L^l$	Length of the lead vehicle (LV)
$L^f$	Length of the following automated vehicle (FAV)
$\hat{h}_{it}$	Predicted spatial headway
$h_{it}$	Reference spatial headway
$\hat{g}_{it}$	Predicted spatial gap
$g_{it}$	Reference spatial gap
$\hat{j}_{it}^{f}$	Predicted longitudinal jerk of the FAV
$\mathrm{TTC}_{it}$	Time-to-collision at time $t$
$\mathrm{TTC}_{\mathrm{th}}$	TTC safety threshold
$R_v^{(i)}$	RMSE of FAV speed for trajectory segment $i$
$R_h^{(i)}$	RMSE of spatial headway for trajectory segment $i$
$R_a^{(i)}$	RMSE of FAV acceleration for trajectory segment $i$
$R_j^{(i)}$	RMS jerk of the predicted FAV trajectory
$R_{v,\mathrm{th}}$	Normalization threshold for speed RMSE
$R_{h,\mathrm{th}}$	Normalization threshold for headway RMSE
$R_{a,\mathrm{th}}$	Normalization threshold for acceleration RMSE
$R_{j,\mathrm{th}}$	Normalization threshold for jerk
$S_{\mathrm{speed}}^{(i)}$	Normalized speed score
$S_{\mathrm{headway}}^{(i)}$	Normalized headway score
$S_{\mathrm{acc}}^{(i)}$	Normalized acceleration score
$S_a^{(i)}$	Accuracy score
$S_s^{(i)}$	Safety score
$S_c^{(i)}$	Comfort score
$S_{\mathrm{total}}^{(i)}$	Total score for trajectory segment $i$
$S_f$	Final submission score

For evaluation, the following derived quantities are used:

\hat{v}_{it}^{f}=\frac{\hat{x}_{i,t+1}^{f}-\hat{x}_{it}^{f}}{\Delta t}, \qquad v_{it}^{f}=\frac{x_{i,t+1}^{f}-x_{it}^{f}}{\Delta t}, \qquad v_{it}^{l}=\frac{x_{i,t+1}^{l}-x_{it}^{l}}{\Delta t}

\hat{a}_{it}^{f}=\frac{\hat{v}_{i,t+1}^{f}-\hat{v}_{it}^{f}}{\Delta t}, \qquad a_{it}^{f}=\frac{v_{i,t+1}^{f}-v_{it}^{f}}{\Delta t}

The spatial headway and spatial gap are defined as

\hat{h}_{it}=x_{it}^{l}-\hat{x}_{it}^{f}, \qquad h_{it}=x_{it}^{l}-x_{it}^{f}

\hat{g}_{it}=x_{it}^{l}-\hat{x}_{it}^{f}-\frac{L^l+L^f}{2}, \qquad g_{it}=x_{it}^{l}-x_{it}^{f}-\frac{L^l+L^f}{2}

Accuracy

The accuracy score measures how closely the predicted FAV trajectory matches the reference trajectory in terms of speed, spatial headway, and acceleration.

R_v^{(i)}= \sqrt{ \frac{1}{|\mathcal{T}_i|} \sum_{t\in\mathcal{T}_i} \left(\hat{v}_{it}^{f}-v_{it}^{f}\right)^2 }

R_h^{(i)}= \sqrt{ \frac{1}{|\mathcal{T}_i|} \sum_{t\in\mathcal{T}_i} \left(\hat{h}_{it}-h_{it}\right)^2 }

R_a^{(i)}= \sqrt{ \frac{1}{|\mathcal{T}_i|} \sum_{t\in\mathcal{T}_i} \left(\hat{a}_{it}^{f}-a_{it}^{f}\right)^2 }

These RMSE values are normalized into dimensionless scores in $[0,1]$ using predefined thresholds:

S_{\mathrm{speed}}^{(i)}= \max\left(0,\;1-\frac{R_v^{(i)}}{R_{v,\mathrm{th}}}\right)

S_{\mathrm{headway}}^{(i)}= \max\left(0,\;1-\frac{R_h^{(i)}}{R_{h,\mathrm{th}}}\right)

S_{\mathrm{acc}}^{(i)}= \max\left(0,\;1-\frac{R_a^{(i)}}{R_{a,\mathrm{th}}}\right)

The accuracy score is then computed as

S_a^{(i)}= 0.4\,S_{\mathrm{speed}}^{(i)} +0.4\,S_{\mathrm{headway}}^{(i)} +0.2\,S_{\mathrm{acc}}^{(i)}

Safety

The safety score evaluates collision risk at the trajectory-segment level. A collision is identified when the predicted spatial gap becomes negative at any time step and is treated as a hard failure. If no collision occurs, safety is evaluated based on the proportion of time steps whose TTC falls below a predefined safety threshold.

For each trajectory segment $i$ and time step $t$ , TTC is defined as

\mathrm{TTC}_{it}= \begin{cases} \dfrac{\hat{g}_{it}}{\hat{v}_{it}^{f}-v_{it}^{l}}, & \text{if } \hat{v}_{it}^{f}>v_{it}^{l},\\[8pt] +\infty, & \text{otherwise} \end{cases}

A TTC violation indicator is defined as

\delta_{it}= \begin{cases} 1, & \text{if } \mathrm{TTC}_{it}<\mathrm{TTC}_{\mathrm{th}}\\ 0, & \text{otherwise} \end{cases}

The TTC violation ratio for trajectory segment $i$ is

R_{\mathrm{viol}}^{(i)}= \frac{1}{|\mathcal{T}_i|} \sum_{t\in\mathcal{T}_i}\delta_{it}

The safety score is then defined as

S_s^{(i)}= \begin{cases} 0, & \text{if } \hat{g}_{it}<0 \text{ for any } t\in\mathcal{T}_i\\[6pt] 1-R_{\mathrm{viol}}^{(i)}, & \text{otherwise} \end{cases}

Comfort

The comfort score evaluates the smoothness of the predicted FAV trajectory using longitudinal jerk.

\hat{j}_{it}^{f}= \frac{\hat{a}_{i,t+1}^{f}-\hat{a}_{it}^{f}}{\Delta t}

R_j^{(i)}= \sqrt{ \frac{1}{|\mathcal{T}_i|-1} \sum_{t=1}^{|\mathcal{T}_i|-1} \left(\hat{j}_{it}^{f}\right)^2 }

The comfort score is defined as

S_c^{(i)}= \max\left(0,\;1-\frac{R_j^{(i)}}{R_{j,\mathrm{th}}}\right)

Final Score

For each trajectory segment, the total score is computed as

S_{\mathrm{total}}^{(i)}= 0.5\,S_a^{(i)} +0.3\,S_s^{(i)} +0.2\,S_c^{(i)}

with

0 \le S_{\mathrm{total}}^{(i)} \le 1

The final submission score is the average total score across all evaluated trajectory segments:

S_f= \frac{1}{|\mathcal{I}|} \sum_{i\in\mathcal{I}} S_{\mathrm{total}}^{(i)}