Lstm in c++/training tips

Beginners
1

i’ll give other posters the choice on whether to v.2 my previous thread on rnns in c with my lstm code. instead of posting it all first,

i have a 2 layer lstm (i think!) using 96 node “hot one” char i/o layers, tried resizing the hidden layers and the BPTT timesteps but so far i’ve only successfully managed to take a single sentence down to a low error coefficient. i’ve been using a translation of the tao te ching at around 56k chars, which i’m assured ought to be within scope for two hidden layers of say 64 or 128 nodes. (256 nodes is too heavy for my little machine.)

having spent some time now training eg. incorrect architectures, i’m aware of various dynamics and trends in ‘training behaviour,’ but i’m finding lstm to be quite evasive. if left unsupervised the error coefficient doesn’t just stop progressing but can climb and climb astronomically. i often observe the same learning rate train through a few times with excellent progress then eat its way well past the ~3.0 threshold of “just plain guessing now” when it begins the next test.

the architecture trains successfully on elementary tests (“abababab”) and the overfit training sentence, am i misinformed about the capacity of such a model to store the data?

i don’t really want to restructure my code for a single hidden layer, though this would of course be sensate, just avoidable work since the two layer source allows me to build variable layer architecture in the future.

i’ve been using gradient normalisation at 1.0, gradient clipping at 1.0 is faster, occasionally ADAM at 1e-8 and if i drop the learning rate just right i can get close to 2.0 if i’m super lucky but one wrong corner and it’s toast. basically i’m still looking at 3.0 or 2.68 again if i step away for a moment. it loves to stick and loves ~3.0.

2

okay… going through the hochreiter and schmidhuber paper on lstm, it doesn’t behave like they describe so i believe my implementation is incorrect,

i accidentally “hid” this post by using 20th cent. british vernacular with a “b”, i think it’s now unhid.. since then i’ve u/led code to git hub. com/atomictraveller/scary-mess (unworking code repository), the lstm forward and backward pass in “rnn-lstm.h”

instead of just code, here are the particularities of method:

all biases are model initialised at zero except forget gates are set to 1.0
all weights are x.g. initialised in the range of -/+ sqrt(6/(inputs + outputs))

for each training data round, long and short term memory cells are set to 0.

at the end of each BPTT, the long and short term states (‘c’ and ‘h’) for the last time step are stored to use for the t-1 values for next BPTT’s step 0.

for form, here’s my lstm hidden layer forward pass,
for (unsigned int i = 0; i < hidd; i++) { // forget gate HIDDEN LAYER 2
float sum = hfbias[i];
for (unsigned int j = 0; j < hidd; j++) sum += nnhif[j][i] * h0[twin][j];
for (unsigned int j = 0; j < hidd; j++) sum += nnhhf[j][i] * h[tprev][j];
hf[twin][i] = sum = 1.f / (1.f + exp(-sum));
c[twin][i] = c[tprev][i] * sum;
}
for (unsigned int i = 0; i < hidd; i++) { // input gate
float sum = hibias[i]; // input gate
for (unsigned int j = 0; j < hidd; j++) sum += nnhii[j][i] * h0[twin][j];
for (unsigned int j = 0; j < hidd; j++) sum += nnhhi[j][i] * h[tprev][j];
hi[twin][i] = sum = 1.f / (1.f + exp(-sum));
float sum2 = hmbias[i]; // input node/potential memory
for (unsigned int j = 0; j < hidd; j++) sum2 += nnhim[j][i] * h0[twin][j];
for (unsigned int j = 0; j < hidd; j++) sum2 += nnhhm[j][i] * h[tprev][j];
hm[twin][i] = sum2 = tanh(sum2);
c[twin][i] += sum * sum2;
}
for (unsigned int i = 0; i < hidd; i++) { // output gate
float sum = hobias[i];
for (unsigned int j = 0; j < hidd; j++) sum += nnhio[j][i] * h0[twin][j];
for (unsigned int j = 0; j < hidd; j++) sum += nnhho[j][i] * h[tprev][j];
ho[twin][i] = sum = 1.f / (1.f + exp(-sum));
h[twin][i] = tanh(c[twin][i]) * sum;
}

noting my [i][j] dimensions are swapped, maybe you have lived through such things before.. (edits were made for correct [input][output] dimensions going into the first hidden layer)

my lstm uses consts idimfor input/output layer size and hidd for hidden layer size.
BPTT window length is defined by wind. winm and winp are wind-1 and wind+1 respectively. arrays for ‘c’ and ‘h’ are sized at [winp] so the t-1 time step between windows is stored in index wind out of the way :slight_smile:

dear god what a mess, whole thing is at bottom of rnn-lstm.h

BPTT:
for (int iter = winm; iter > -1; iter--) { // BPTT 'back propogation through time' int iprev = (bool)iter ? iter - 1 : wind; for (unsigned int i = 0; i < idim; i++) { dobias[i] += netout[iter][i]; for (unsigned int j = 0; j < hidd; j++) dnno[j][i] += netout[iter][i] * h[iter][j]; } for (int i = 0; i < hidd; i++) { float sum = dh_next[i]; for (int j = 0; j < idim; j++) sum += nno[i][j] * netout[iter][j]; dh[i] = sum; } for (int i = 0; i < hidd; i++) { float tanh_c = tanh(c[iter][i]); float dc_total = dc_next[i] + (dh[i] * ho[iter][i] * (1.f - tanh_c * tanh_c)); float df_raw = dc_total * c[iprev][i] * (hf[iter][i] * (1.f - hf[iter][i])); float di_raw = dc_total * hm[iter][i] * (hi[iter][i] * (1.f - hi[iter][i])); float dm_raw = dc_total * hi[iter][i] * (1.f - hm[iter][i] * hm[iter][i]); float do_raw = dh[i] * tanh_c * (ho[iter][i] * (1.f - ho[iter][i])); dhf_raw[i] = df_raw; dhi_raw[i] = di_raw; dhm_raw[i] = dm_raw; dho_raw[i] = do_raw; hfbias[i] += df_raw; hibias[i] += di_raw; hmbias[i] += dm_raw; hobias[i] += do_raw; dh0[i] = 0.f; for (unsigned int j = 0; j < hidd; j++) { dnnhhf[j][i] += df_raw * h[iprev][j]; dnnhhi[j][i] += di_raw * h[iprev][j]; dnnhhm[j][i] += dm_raw * h[iprev][j]; dnnhho[j][i] += do_raw * h[iprev][j]; dnnhif[j][i] += df_raw * h0[iter][j]; dnnhii[j][i] += di_raw * h0[iter][j]; dnnhim[j][i] += dm_raw * h0[iter][j]; dnnhio[j][i] += do_raw * h0[iter][j]; dh0[i] += nnhif[j][i] * df_raw + nnhii[j][i] * di_raw + nnhim[j][i] * dm_raw + nnhio[j][i] * do_raw; } dc_next[i] = dc_total * hf[iter][i]; } unsigned int cidx = cin[iter]; // thank you google ai :) for (int i = 0; i < hidd; i++) { float tanh_c = tanh(c0[iter][i]); float dc_total = dc0_next[i] + (dh0[i] * h0o[iter][i] * (1.f - tanh_c * tanh_c)); float df_raw = dc_total * c0[iprev][i] * (h0f[iter][i] * (1.f - h0f[iter][i])); float di_raw = dc_total * h0m[iter][i] * (h0i[iter][i] * (1.f - h0i[iter][i])); float dm_raw = dc_total * h0i[iter][i] * (1.f - h0m[iter][i] * h0m[iter][i]); float do_raw = dh0[i] * tanh_c * (h0o[iter][i] * (1.f - h0o[iter][i])); dh0f_raw[i] = df_raw; dh0i_raw[i] = di_raw; dh0m_raw[i] = dm_raw; dh0o_raw[i] = do_raw; h0fbias[i] += df_raw; h0ibias[i] += di_raw; h0mbias[i] += dm_raw; h0obias[i] += do_raw; dnnh0if[cidx][i] += df_raw; dnnh0ii[cidx][i] += di_raw; dnnh0im[cidx][i] += dm_raw; dnnh0io[cidx][i] += do_raw; for (unsigned int j = 0; j < hidd; j++) { dnnh0hf[j][i] += df_raw * h0[iprev][j]; // def. swapped dnnh0hi[j][i] += di_raw * h0[iprev][j]; dnnh0hm[j][i] += dm_raw * h0[iprev][j]; dnnh0ho[j][i] += do_raw * h0[iprev][j]; } dc0_next[i] = dc_total * h0f[iter][i]; } for (int i = 0; i < hidd; i++) { float sum0 = 0.f; float sum = 0.f; for (int j = 0; j < hidd; j++) { sum0 += nnh0hf[j][i] * dh0f_raw[j] + nnh0hi[j][i] * dh0i_raw[j] + nnh0hm[j][i] * dh0m_raw[j] + nnh0ho[j][i] * dh0o_raw[j]; sum += nnhhf[j][i] * dhf_raw[j] + nnhhi[j][i] * dhi_raw[j] + nnhhm[j][i] * dhm_raw[j] + nnhho[j][i] * dho_raw[j]; } dh0_next[i] = sum0; dh_next[i] = sum; } } memcpy(h0[wind], h0[winm], sizeof h0[wind]); memcpy(h[wind], h[winm], sizeof h[wind]); memcpy(c0[wind], c0[winm], sizeof c0[wind]); memcpy(c[wind], c[winm], sizeof c[wind]);

“great eyes!” to any comments, and thank you :slight_smile:

if anyone is amused by such things, “my c rnn also!” is famous on github, thankfully working quite well and hopefully not leading the world astray.
git hub. com/atomictraveller/rnn
(then i freaked out about how everything automatically turns links into sparkly webpage events, explaining link fragmentation to force text)

3

Hmm… for now, based on what can be said from this alone:

I would not take this as evidence that two 64/128-unit layers are too small yet.

abababab and the one-sentence overfit are useful smoke tests, but they may not exercise the window boundary, the full two-layer backward path, or the optimizer state in the same way as the complete text.

The first useful split is the loss scale.

If the error coefficient is ordinary mean negative log-likelihood using natural logs over 96 outputs, then zero logits give:

double random_nll = log(96.0);

/* 4.564348... */

So under that definition, 3.0 would already be better than uniform guessing.

If your coefficient is normalized or scaled differently, that is fine too; I would just establish its zero-logit baseline before reading too much into 3.0, 2.68, etc.

The second split is normalization versus clipping.

These are not equivalent:

double scale = 1.0;

if (grad_norm > 1.0) {
    scale = 1.0 / grad_norm;
}

/* Apply scale to all gradients. */

That is clipping: a gradient with norm 0.5 remains at 0.5.

By contrast:

scale = 1.0 / grad_norm;

always normalizes to 1.0, so a small gradient can be enlarged near a good solution. The usual global-norm clipping definition only scales gradients down when they exceed the limit.

A third split is the LSTM state at a BPTT boundary.

With updates disabled, these two paths should produce the same logits and final states:

/* Path A: one continuous forward pass. */
forward(sequence, 2 * T, h0, c0);

/* Path B: two windows. */
forward(sequence,     T, h0, c0);
forward(sequence + T, T, hT, cT);

Both h and c need to cross the window boundary. The forward state may continue while the backward history stops there; that is the usual distinction in truncated BPTT.

If the two paths differ before any weight update, I would fix that before changing hidden size or learning rate.

Finally, ADAM at 1e-8 has two very different branches:

learning_rate = 1e-8;  /* Extremely small update. */
epsilon       = 1e-8;  /* Common numerical stabilizer. */

If those basic invariants all pass, the next high-information test would be a tiny deterministic two-layer case, for example:

vocab  = 3
hidden = 2
steps  = 3
double precision
no clipping
no Adam

Then compare the analytical gradients against finite differences.

That does not require restructuring the main source into a permanent one-layer model; the existing variable-layer code can remain intact and just run a very small test configuration.

So my current branch order would be:

loss baseline
-> clipping vs unconditional normalization
-> h/c window equivalence
-> tiny two-layer gradient check
-> Adam state
-> only then BPTT depth and capacity

The capacity question still matters, but the sudden runaway behaviour sounds more informative about state, updates, or measurement than about raw storage size by itself.