Hi everyone, I am still learning about this model and there is something I havent managed to wrap my head around: How do you calculate the number of parameters of the model?
From what I understand, the mamba block has 3ED^2 + D parameters from the linear projections and the RMSNorm per layer. Then, the embedding and LM head parts both have D * Vocab size.
So from the config params we would get:
layers * 3 E * D^2 + 2D*V= 24 * 3 * 2 * 768^2 + 2 * 768 * 50277 = 162,178,560
The model card shows 130M, what am I missing?