Greetings, everyone. We stand strong and continue to broaden o | Read & listen

Greetings, everyone. We stand strong and continue to broaden our knowledge in ML. Here is a review of a detailed paper on the topic of confusion matrices.

Having interviewed some engineers and data scientists, the authors did quite a bit of research on the topic. They came up with a following tasks:
- Visualise metrics while enabling flexible data analysis, including scaling and normalisation;
- Traverse and visualise hierarchical labels;
- Transform and visualise multi-output labels;
- Share confusion matrix analysis and configs.

Three techniques for transforming high-dimensional multi-output labels are proposed:
- Conditioning primes a probability distribution on given values. This operation can be used to extract sub-views of a larger confusion matrix;
- Marginalisation allows the discarding of variables of multivariate distributions that are irrelevant by summing over all such variables;
- A nesting operation is also proposed. It is useful to investigate multiple labels simultaneously.

The resulting algebra models confusion matrices as probability distributions and expresses more variations of confusion matrices, e.g., datasets with hierarchical and multi-labels.

Based on this algebra, Neo was designed – a visual analytics system that allows practitioners to flexibly author, interact with, and share confusion matrices. The proposed visualisation system is agnostic to the model architecture and data. The system is web-based, built with Svelte, TypeScript, and D3.

Neo’s JSON specification configures confusion matrix based on the selected normalisation scheme, visualisation encoding, and desired measures (metrics), but also saves the
shown state of the hierarchy (collapsed, filter) and multi-output labels using either marginalisation (classes), nesting (order of classes array), or conditioning (where).

The authors demonstrate Neo’s utility with three model evaluation scenarios that help people better understand model performance and reveal hidden confusions:

- A confusion matrix for an object detection model that computes and shows user-specified performance metrics, such as recall or precision, per class alongside the matrix visualisation.
- A sub-hierarchy of a confusion matrix for a 1,000-class image classifier compares the confusions of different vegetables, such as "cucumber and mushroom", against the sub-hierarchies of "cruciferous vegetables and squash".
- A confusion matrix for a naive multi-output online toxicity detector that conditions and filters by identity hate confusions, nests obscene confusions under mild toxic confusions, and finds that the model misses to identify many obscene comments.

Overall, confusion matrices continue to be an excellent tool to teach modelling fundamentals to novices and an invaluable tool for practitioners building industry-scale systems.