How can I define a network like the following?
I want two or more LSTM networks (here one has 3 and other 4 states), some of these states could be related (equal letters here). I want to connect them to a partial feedforward layer and finally output some outputs (here 5 equal to distinct letters).
A --------------A
B ----------|--B
C -------|-----C
| |
B-----| |
C ---------|
D -----------D
E ---------- E
Could you please guide me, I am novice with Pytorch. I know Linear or Sequential classes, but they are all about fully connected networks.