2022-03-08 14:44:20
Greetings, everyone. We stand strong and continue to broaden our knowledge in ML. Here is a review of a detailed paper on the topic of
confusion matrices.
Having interviewed some engineers and data scientists, the authors did quite a bit of research on the topic. They came up with a following tasks:
- Visualise metrics while enabling flexible data analysis, including
scaling and normalisation;
- Traverse and visualise
hierarchical labels;
- Transform and visualise
multi-output labels;
- Share confusion matrix
analysis and
configs.
Three techniques for transforming high-dimensional multi-output labels are proposed:
-
Conditioning primes a probability distribution on given values. This operation can be used to extract sub-views of a larger confusion matrix;
-
Marginalisation allows the discarding of variables of multivariate distributions that are irrelevant by summing over all such variables;
- A
nesting operation is also proposed. It is useful to investigate multiple labels simultaneously.
The resulting algebra models confusion matrices as
probability distributions and expresses more variations of confusion matrices, e.g.,
datasets with hierarchical and multi-labels.
Based on this algebra,
Neo was designed – a visual analytics system that allows practitioners to flexibly author, interact with, and share confusion matrices. The proposed visualisation system is
agnostic to the model architecture and data. The system is web-based, built with
Svelte, TypeScript, and D3.
Neo’s
JSON specification configures confusion matrix based on the selected
normalisation scheme, visualisation
encoding, and desired
measures (metrics), but also saves the
shown state of the hierarchy (
collapsed,
filter) and multi-output labels using either
marginalisation (
classes),
nesting (order of
classes array), or
conditioning (
where).
The authors demonstrate Neo’s utility with three model evaluation scenarios that help people better understand model performance and reveal hidden confusions:
- A confusion matrix for an
object detection model that computes and shows user-specified performance
metrics, such as recall or precision, per class alongside the matrix visualisation.
- A sub-hierarchy of a confusion matrix for a
1,000-class image classifier compares the confusions of different vegetables, such as
"cucumber and mushroom", against the sub-hierarchies of
"cruciferous vegetables and squash".
- A confusion matrix for a naive multi-output
online toxicity detector that conditions and filters by
identity hate confusions, nests
obscene confusions under
mild toxic confusions, and finds that the model misses to identify many
obscene comments.
Overall, confusion matrices continue to be an
excellent tool to teach modelling fundamentals to novices and an invaluable tool for practitioners building industry-scale systems.
4 views11:44
R
