Publishing where your data fails

The tempting move

Marker tracking does not work equally well across every run. Some configurations are low-contrast, some strokes leave the field, some frames are simply bad. You end up with a detection rate per run: the fraction of frames where the tracker actually found what it was looking for.

The tempting move is to take the mean anyway. Every run contributes, the summary table has no gaps, and nobody asks an awkward question.

It is also the move that puts a confident number on top of data that cannot support one.

Gating instead

The alternative costs almost nothing to implement and changes what the results mean.

Every statistic in my analysis is gated on detection rate. Below the threshold, the run does not contribute to the aggregate. Not down-weighted, not interpolated — excluded, and recorded as excluded.

That leaves a hole in the table, which is the point. The hole is information.

The failure map

So the second half is publishing where the holes are. Alongside the results, the pipeline emits a tracking-failure heatmap across the full configuration grid: six fibre configurations against thirteen curvature channels, shaded by detection rate.

It is not a flattering figure. It shows, unambiguously, which corners of my own experiment did not work.

It is also the most useful thing in the output, for three reasons:

• A reader can see whether their region of interest is one I actually measured.
• The next person to run the rig knows which configurations need a better detector before they waste a day.
• It made a real limit explicit — a coverage claim I could only defend over a bounded range, and said so.

If you cannot show where your method fails, you have not established that it works anywhere.

Why this is not just tidiness

Averaging over bad data is not a small sin that gets corrected later. It produces a result that is specifically hard to catch, because the output looks exactly like a good result: a smooth curve, a plausible magnitude, a tight-looking interval.

Publishing the failure map is cheaper than the alternative, which is somebody trusting a number that was never earned.