Hmm… For now, maybe something like this?:
I think MicroMass is useful, but I would not make it the first or only benchmark for this engine.
The UCI MicroMass dataset is reasonably high-dimensional at 1,300 variables, and it has more structure than an ordinary classification dataset: the associated study used 571 reference spectra from 20 bacterial species, plus 360 spectra from controlled two-species mixtures at nine nominal concentration ratios. That makes it a potentially good real-data sanity check.
However, it does not provide a known true latent loading matrix, it is smaller than the thousands-by-thousands scale you are looking for, and the number of species is not automatically the true latent dimension. I would therefore use it after a small synthetic benchmark rather than as the primary test.
My default route would be:
| Stage |
Main purpose |
| 1. Rotation-only synthetic benchmark |
Test recovery of known sparse structure and avoidance of basis collapse |
| 2. Sampled synthetic benchmark |
Add sampling noise and factor-extraction error |
| 3. MicroMass |
Test behavior on structured real spectra and controlled mixtures |
| 4. Large real dataset |
Test runtime, memory use, stability, and scaling |
The useful distinction, at least as I understand the earlier description of the engine, is:
Does it recover sparse/simple structure?
and separately:
Does it avoid singular, duplicated, or badly conditioned bases while doing so?
Those are not the same success criterion. A method can produce very sparse-looking loadings by driving the transformation toward a nearly singular solution, while another method can remain numerically stable without recovering the planted sparse structure.
For literature searches, nearby established terms appear to be oblique simple-structure rotation, sparse-loading rotation, hyperplane rotation, and factor-rotation local solutions. I would keep your term “soft sparsity,” but adding those terms in the documentation would help future readers find relevant comparisons. Your original Functionplane paper is itself a useful anchor: it describes a smooth oblique simple-structure criterion that can be optimized by gradient methods.
A practical first release would not need a huge dataset. A small benchmark with known answers would probably demonstrate the distinctive behavior more clearly.
1. A small benchmark separating sparsity recovery from collapse avoidance
Start with a rotation-only problem
Before generating observations, begin with a known factor-pattern matrix.
For example:
- 12–60 observed variables
- 3–6 factors
- four or more primary variables per factor
- primary loadings around 0.7–0.9
- optional genuine cross-loadings around 0.1–0.2
- optional factor correlations of 0, 0.3, and 0.6
Let the known sparse pattern be Lambda_true.
Create the starting loading matrix by applying a known invertible transformation:
\Lambda_{\mathrm{start}} = \Lambda_{\mathrm{true}} H
The engine then receives Lambda_start and attempts to recover a simple structure.
This first test deliberately excludes:
- sampling error;
- covariance estimation;
- uncertainty in the number of factors;
- differences between factor-extraction methods.
It tests only the rotation or transformation engine.
That makes failure easier to interpret. If the planted pattern cannot be recovered here, there is no need to blame the dataset, sample size, factor extraction, or measurement noise.
A compact benchmark battery
I would give the cases stable identifiers so that results from different implementations can be compared.
| ID |
True structure and starting condition |
Main question |
| B0 |
Already-simple pattern, identity or nearly identity start |
Does the engine leave a good solution alone? |
| B1 |
Exactly sparse truth, well-conditioned transform |
Can it recover basic sparse structure? |
| B2 |
Approximately sparse truth with genuine small cross-loadings |
Does it preserve small but real structure? |
| B3 |
Dense truth |
Does it manufacture sparsity where none exists? |
| B4 |
Sparse truth with increasingly ill-conditioned starts |
Does it avoid basis collapse? |
| B5 |
Symmetric or ambiguous truth with several equivalent solutions |
Does it report non-uniqueness rather than invent certainty? |
| B6 |
Observations sampled from B1–B4 |
What changes after sampling and factor extraction? |
B0: no-op control
Give the program a loading matrix that already has a clean simple structure.
A successful method should not substantially damage:
- loading recovery;
- factor correlations;
- transformation conditioning;
- factor separation.
This is a useful check because an optimizer may slightly improve its own objective while making an already satisfactory representation less interpretable or less stable.
B1: exact sparse truth
Each variable loads primarily on one factor, with all other entries exactly zero.
This tests the easiest form of planted recovery.
B2: approximate sparsity
Add a few genuine cross-loadings, for example 0.10–0.20.
This is particularly relevant to a “soft” sparsity method. It tests whether the method can distinguish:
- an exact structural zero;
- a small but genuine loading;
- a noisy or unstable small coefficient.
A method that simply suppresses every small value may look sparse while recovering the wrong population structure.
B3: dense negative control
Create a loading matrix in which most entries are genuinely nonzero.
Here, a very sparse answer is not automatically a good answer. This control separates:
- “the criterion prefers sparse solutions” from
- “the method recovered the true sparse structure.”
It would be quite informative to show simplicity scores improving while truth-recovery error becomes worse. That is not necessarily a bug; it demonstrates what the objective is encouraging.
B4: conditioning ladder
Keep Lambda_true fixed and control the singular values of the starting transform.
One reproducible construction is:
H = U D V^{\mathsf T}
where U and V are orthogonal matrices and D contains chosen singular values.
For example:
| Level |
Approximate singular-value range |
Intended difficulty |
| Easy |
1.0 to 0.5 |
Well conditioned |
| Moderate |
1.0 to 0.1 |
Mild stress |
| Difficult |
1.0 to 0.01 |
Strong stress |
| Near-singular |
1.0 to 0.001 |
Collapse stress |
The exact values are not important; recording them is.
This distinguishes three materially different outcomes:
- the algorithm starts in a badly conditioned representation and repairs it;
- it starts well conditioned and becomes badly conditioned;
- it remains badly conditioned but completes without a numerical exception.
Those should not all be reported merely as “converged.”
B5: ambiguous solutions
Create a symmetric pattern for which several permutations or rotations have essentially equivalent simple structure.
Different runs may then return different valid solutions.
The useful questions become:
- Are the solutions equivalent after sign and permutation alignment?
- Do they occupy the same subspace?
- Do they have similar conditioning?
- Are several stable solution families repeatedly found?
- Are equal or nearly equal objective values associated with materially different patterns?
This is important because non-uniqueness is not the same as optimizer failure.
B6: sampled observations
Only after the rotation-only tests are understood, generate observations from the planted model and repeat the comparison.
That introduces:
- sample-size effects;
- noise;
- model approximation error;
- uncertainty in the unrotated loading estimate;
- dependence on the extraction method.
Keeping B1–B4 as population-level tests makes it possible to tell whether a later failure came from the rotation or from the earlier statistical estimation step.
2. What I would record: four separate result families
I would avoid a single aggregate score. At minimum I would keep four columns:
| Result family |
What it answers |
| Objective optimization |
Did the optimizer reduce its own criterion and meet its stopping rule? |
| Ground-truth recovery |
Did it recover the planted factor pattern? |
| Simple-structure quality |
Is the returned pattern sparse or concentrated? |
| Stability and collapse |
Is the solution reproducible and numerically non-degenerate? |
A. Objective optimization
Record:
- initial and final criterion value;
- criterion trajectory;
- gradient norm;
- step size;
- iteration count;
- stopping reason;
- maximum-iteration failures;
- runtime.
Criterion values are most useful for comparing starts or variants using the same criterion.
Values from different rotation criteria are generally not directly comparable because they optimize different functions and may use different scales.
B. Ground-truth recovery
Before comparing columns, align the estimated factors with the true factors.
A reasonable procedure is:
- compute a similarity matrix between true and estimated factor columns;
- solve the one-to-one assignment problem;
- reorder the estimated columns;
- flip signs where necessary;
- apply an explicitly stated scale convention;
- then compute the recovery metrics.
Useful metrics include:
- loading RMSE after alignment;
- Tucker congruence or column cosine similarity;
- support precision and recall;
- precision–recall curves over several thresholds;
- recovery of genuine cross-loadings;
- error in the factor-correlation matrix;
- subspace distance.
The fungible package contains factor-alignment and local-solution utilities, including alignment before comparing rotated patterns. The psych::factor.congruence documentation is also a useful reference for congruence coefficients.
Column matching is not enough when:
- one factor splits into two;
- two factors merge;
- the estimated and true dimensions differ;
- factors become nearly collinear.
For those cases, subspace and conditioning diagnostics remain important.
C. Criterion-independent simple-structure diagnostics
The current GPArotation documentation includes several simple-structure measures, including:
- AUC-style loading concentration;
- Factor Simplicity Index;
- Gini-type measures;
- Bentler-style simplicity;
- Hoffman measures;
- hyperplane counts.
These can be useful because they are not simply the method’s own optimization objective.
However, they are not truth-recovery metrics.
For example:
- a dense true pattern may receive a worse simplicity score even when correctly recovered;
- a highly correlated but clean oblique solution may behave differently under some indices;
- hyperplane counts depend on a selected cutoff.
I would therefore report several cutoffs, for example 0.05, 0.10, and 0.15, rather than presenting a single hyperplane count as definitive.
For oblique rotation it is also important to state whether a metric is being calculated from:
- the pattern matrix;
- the structure matrix;
- the factor-correlation matrix.
Those are not interchangeable.
D. Collapse and numerical stability
Because “basis collapse” can refer to several related numerical or geometric events, I would initially report a diagnostic bundle instead of choosing one number in advance.
Transformation geometry
- smallest singular value of the effective transform;
- transform condition number;
- determinant only if the scale convention is fixed;
- ratio of largest to smallest column norm;
- maximum absolute cosine between basis directions.
Factor geometry
- smallest eigenvalue of the factor-correlation matrix;
- condition number of the factor-correlation matrix;
- largest absolute off-diagonal factor correlation;
- duplicated or nearly duplicated factor columns;
- factors that disappear or absorb most of the primary loadings.
Numerical behavior
- inverse or factorization failure;
- NaN or infinity;
- overflow or underflow;
- extremely small steps with a large remaining gradient;
- a falling objective accompanied by rapidly worsening conditioning;
- sensitivity to floating-point precision.
The most revealing plots may be:
- criterion value versus iteration;
- condition number versus iteration;
- minimum singular value versus iteration;
- truth-recovery error versus iteration.
A final solution may look acceptable even though the path approached a numerical singularity, and the opposite can also happen.
If your original definition of “basis collapse” is specifically one of these events, that quantity can be designated as the primary endpoint while the others remain secondary diagnostics.
3. Comparison methods, controls, and local solutions
Reference methods
For an initial comparison I would keep the baseline set small:
- unrotated loading matrix;
- quartimin;
- oblimin;
- geomin;
- the proposed smooth criterion;
- a modern sparse-loading rotation.
GPArotation provides commonly used orthogonal and oblique rotation criteria and accepts an initial loading matrix. It is useful as:
- a reference implementation;
- a source of standard comparison criteria;
- a model for convergence diagnostics;
- a random-start comparison.
I would not treat it as an authority that determines which answer is correct.
A particularly close modern comparison is Rotation to Sparse Loadings Using L^p Losses. It addresses sparse oblique loadings, although its loss and optimizer differ from the smooth criterion described here. The authors provide a reproduction repository containing simulation studies, Big Five data, geomin comparisons, raw results, and grid checks for some solutions.
That makes it useful both as a baseline and as an example of how to package a simulation study.
Keep extraction fixed
If the program receives an already extracted loading matrix, this is straightforward: give every rotation method the same matrix.
If the program receives a covariance or correlation matrix:
- save the extracted unrotated loadings;
- use the same extraction method for every rotation;
- separately report extraction fit and rotation results.
If the program receives raw observations:
- use the same preprocessing and factor extraction for every method;
- save the resulting initial loading matrix;
- also run the rotation-only benchmark on that matrix.
Otherwise, a difference caused by factor extraction can easily be mistaken for a difference caused by rotation.
Random starts
Smoothness does not by itself imply a unique optimum.
A large study of local minima in factor rotation found multiple local solutions under some conditions for varimax, oblimin, entropy, and orthogonal and oblique geomin. The study also found cases where the lowest criterion value was not the population-closest loading pattern.
The current GPArotation interface therefore supports random-start wrappers and explicitly notes that the best returned random-start solution is still not guaranteed to be global.
I would not assume in advance that 10, 20, or 100 starts is sufficient. Instead, one small result could be a saturation curve:
| Starts attempted |
Best objective |
Best truth recovery |
Distinct patterns |
Collapse rate |
Runtime |
| 1 |
|
|
|
|
|
| 5 |
|
|
|
|
|
| 10 |
|
|
|
|
|
| 20 |
|
|
|
|
|
| 50 |
|
|
|
|
|
| 100 |
|
|
|
|
|
The useful number of starts is then an empirical property of the method and benchmark rather than an arbitrary requirement.
Do not identify solutions only by objective value
Two runs can have:
- clearly different objective values;
- nearly equal objective values but different loading patterns;
- different column orders or signs but equivalent patterns.
After factor alignment, cluster the returned patterns by loading RMSD or another stated distance. Save:
- objective value;
- aligned pattern;
- factor correlations;
- conditioning;
- frequency with which each solution family appeared.
The faMain and faLocalMin facilities in fungible are useful examples of this style of local-solution analysis.
Ablation, if the code permits it
If the implementation makes the components separable, a particularly informative comparison would be:
- the full method with the metric and inverse-metric adaptation;
- the same smooth sparsity criterion with that adaptation disabled or held fixed.
That could help distinguish:
- the effect of the sparsity criterion;
- the effect of the adaptive metric;
- the effect of the inverse metric on singularity avoidance.
If the implementation does not allow a clean ablation, there is no need to redesign it merely for this purpose. Comparisons against standard rotations under the same starts and stopping budget would still be useful.
4. Adding sampled data without losing the known answer
For the second stage, an existing generator may save some work.
The R package fungible contains simFA, which can generate factor-analysis population models and Monte Carlo samples with:
- user-defined pattern matrices;
- orthogonal or oblique factors;
- controlled factor correlations;
- fixed or sampled primary loadings;
- controlled cross-loadings;
- model approximation error;
- raw observations;
- population correlation matrices;
- unrotated loadings;
- fixed random seeds.
This does not make simFA the only acceptable generator. Its value is that it already exposes many of the controls needed for a reproducible study.
A compact public suite might have three scale levels:
| Configuration |
Cases |
Variables |
Factors |
Purpose |
| Toy |
500 |
60 |
6 |
Easy inspection and debugging |
| Medium |
2,000 |
500 |
20 |
Statistical comparison |
| Stress |
5,000 or more |
2,000 or more |
50 or 100 |
Scaling and conditioning |
Those are only example values. Synthetic generation is the easiest way to obtain exactly the requested combination of:
- thousands of cases;
- thousands of variables;
- 10–100 known latent dimensions.
For each configuration I would preserve:
- the population pattern;
- population factor correlations;
- generated observations;
- unrotated estimated loadings;
- seeds;
- preprocessing;
- extraction method;
- all rotated solutions.
This separates three errors:
- model generation and sampling error;
- factor-extraction error;
- rotation error.
A real dataset cannot normally provide that separation.
5. Where MicroMass fits
MicroMass looks useful as the third stage: a real-data test with unusually helpful external structure.
The associated Bioinformatics paper describes:
- 571 reference spectra;
- 20 species from nine genera;
- 213 strains;
- 10 two-species mixture families;
- two strain pairs per mixture family;
- nine nominal concentration ratios;
- two replicate spectra per ratio and strain pair;
- 360 mixture-set spectra in total.
The mixture families also vary in taxonomic difficulty:
- four same-genus pairs;
- two different-genus but same-Gram-type pairs;
- four different-Gram-type pairs.
That supports several checks more informative than ordinary classification accuracy.
M1. Pure endpoints
At the 1:0 and 0:1 endpoints, does the representation favor the corresponding component or direction?
M2. Mixture-path ordering
Across:
1:0 → 10:1 → 5:1 → 2:1 → 1:1 → 1:2 → 1:5 → 1:10 → 0:1
does the relevant score or projection change in a broadly ordered or monotonic way?
I would not require exact proportionality.
The original paper reports real biological and preparation effects, including a case where nominal optical-density matching did not correspond to equal cell counts. Thus, “nominal 1:1 concentration” is not necessarily an exact linear latent-score target.
Endpoints, ordering, rank correlation, and replicate consistency are safer controls than requiring the recovered score ratio to equal the nominal mixing ratio.
M3. Replicate stability
Do the two replicate spectra for the same strain pair and nominal ratio produce similar results?
M4. Strain-pair generalization
Each species mixture was represented by two pairs of strains.
One option is to derive or tune the representation using one strain pair and examine the corresponding path on the other pair. This is stronger than a random row split because it tests some biological generalization.
M5. Difficulty strata
Report results separately for:
- same genus;
- different genus, same Gram type;
- different Gram type.
The original study found that spectral separability and biological similarity affected mixture identification. Pooling everything into one average could hide that structure.
M6. Minority-component behavior
Check whether the less abundant component disappears first at 10:1 or 1:10 and whether this behavior is consistent across replicates.
This is relevant to sparsity because an aggressive sparsity mechanism may remove a real but weak component.
M7. Grouped splitting
Avoid treating all 931 rows as independent interchangeable samples.
Depending on the experiment, grouping may need to be by:
- strain;
- strain pair;
- species pair;
- mixture series;
- replicate family.
A random row split could place extremely related spectra on both sides of the evaluation.
What MicroMass would and would not establish
It could provide evidence about:
- behavior on real mass spectra;
- stability across replicates;
- response to controlled mixture paths;
- recovery of weak versus strong components;
- sensitivity to biological similarity;
- practical runtime at 1,300 variables.
It would not by itself establish:
- recovery of a true factor-loading matrix;
- a true latent dimension of 20;
- correct behavior at thousands of cases and thousands of variables;
- general superiority to other sparse factor methods.
That is why it fits well after the planted tests rather than replacing them.
6. A larger Hugging Face stress test
For a dataset already on the Hub, SetFit/20_newsgroups is one fairly easy stress-test option.
It currently has about 18,800 documents and 20 labels. A TF–IDF representation can readily produce:
- 5,000–20,000 variables;
- more than 10,000 cases;
- a chosen reduced dimension between 20 and 100.
For example:
- construct a sparse TF–IDF document-term matrix;
- obtain an initial 20-, 50-, or 100-dimensional representation;
- apply the rotation engine to the resulting loading or component matrix;
- compare stability, conditioning, runtime, and top-loading terms.
This would test:
- sparse high-dimensional input;
- scalability;
- memory behavior;
- random-start stability;
- interpretability of rotated directions.
It would not provide a true latent factor pattern.
The 20 newsgroup labels are external annotations, not proof that the data have exactly 20 latent dimensions. I would therefore call this a stress test, not a recovery benchmark.
Other possible scale tests include:
- gene-expression or single-cell matrices with thousands of measured variables;
- large document-term matrices;
- larger mass-spectrometry collections;
- hyperspectral data.
Each has its own preprocessing and dependence structure. I would select only one initially, because the planted benchmark plus MicroMass already cover the more important scientific questions.
A nearby controlled-mixture problem is hyperspectral unmixing. For example, this laboratory-created hyperspectral mixture dataset has known materials and compositions over many pixels. It has fewer spectral variables than requested and solves an adjacent rather than identical problem, but its benchmark design may be useful.
7. A reproducible result package
For publication, I would separate the code, benchmark data, and optional interactive demonstration.
GitHub or another source repository
Best suited for:
- Fortran source;
- Octave wrapper;
- build command;
- license;
- version history;
- benchmark generator;
- comparison scripts;
- tests;
- issues;
CITATION.cff.
Hugging Face Dataset repository
Best suited for:
- fixed benchmark inputs;
- planted loading matrices;
- generated observations;
- baseline outputs;
- result tables;
- trajectory logs;
- MicroMass-derived experiment metadata;
- reproducibility fixtures.
The official Dataset Card documentation describes the README.md card and its metadata. The Hub also supports multiple manually configured subsets and splits, so the repository could expose configurations such as:
toy_exact_sparse;toy_crossloadings;toy_dense_control;conditioning_ladder;medium;stress;micromass.
Optional Space
A small browser demonstration could be useful after the command-line example works.
Because the implementation uses Fortran and GNU Octave, a Docker Space may be simpler than trying to force it into a Python-only environment.
A minimal interface could let a user:
- select a benchmark;
- choose the criterion or mode;
- run the engine;
- view before/after loading heatmaps;
- inspect the objective and conditioning trajectories;
- download the result bundle.
Model repository
A Model repo can still be used if the released object behaves like a reusable model or component, but the official Model Card guidance is mainly organized around documenting a model artifact, its intended uses, limitations, data, and evaluation.
For a historical algorithm plus source code and benchmarks, I would still make the source repository and Dataset repo the primary homes, with a Model repo only if there is a clearly loadable model-like artifact.
Suggested run artifact
Something like this would make third-party comparisons easier:
benchmark_id/
├── config.json
├── true_pattern.csv
├── true_phi.csv
├── initial_pattern.csv
├── initial_transform.csv
├── observed_data.csv
├── rotated_pattern.csv
├── rotated_structure.csv
├── estimated_phi.csv
├── effective_transform.csv
├── alignment.json
├── trajectory.csv
├── metrics.json
├── stdout.txt
├── stderr.txt
└── environment.txt
trajectory.csv could include:
iteration
objective
gradient_norm
step_size
transform_condition_number
minimum_singular_value
maximum_basis_cosine
metrics.json could keep the result families separate:
truth_recovery
simple_structure
conditioning
optimization
runtime
failure_status
This is more useful than preserving only the final rotated matrix, because a future implementation can inspect not only where the optimizer ended, but how it got there.
A manageable first version
If I were reducing this to the smallest useful public example, I would start with:
- one planted 15-by-3 or 60-by-6 loading pattern;
- B0, B1, B2, B3, and a four-level B4 conditioning ladder;
- an identity start plus a modest random-start saturation experiment;
- unrotated, quartimin, oblimin, geomin, one sparse-loading baseline, and the proposed method;
- separate tables for truth recovery, simple structure, and conditioning;
- one MicroMass mixture-path demonstration;
- fixed seeds and machine-readable result files.
That would already answer several important questions:
- Does the engine recover a known sparse pattern?
- Does it preserve genuine small cross-loadings?
- Does it falsely sparsify a dense pattern?
- Does it remain nonsingular under a controlled conditioning stress?
- Does the metric/inverse-metric mechanism visibly change that behavior?
- Are the results stable across starts?
- Does the method show coherent behavior on real controlled mixtures?
After that, increasing the synthetic dimensions to thousands of variables and cases is mostly a scaling exercise rather than a search for a perfect real dataset.
So my answer to the original dataset question would be:
- Yes, MicroMass is worth using, especially because of its pure endpoints, mixture series, strain pairs, and replicates.
- No, I would not use it as the sole or primary proof of sparse-factor recovery.
- For the exact requested dimensions and a known latent truth, a generated benchmark is probably the cleanest solution.
- The most informative first benchmark would measure sparse-structure recovery and collapse avoidance as two separate outcomes.