- Author
- Casey Primozic
- Published
- 15 January 2023
- Kind
- Personal blog post with a hands-on experiment
- Original
- cprimozic.net/blog/reverse-engineering-a-small-neural-network/
- Subject
- Language models and training
What it is
Casey Primozic writes a personal homepage blog on small neural networks. The post describes training and then reverse engineering a network for addition. It aims to explain how the network computes the sum.
What it says
- Primozic trained small networks to add two 8-bit numbers, reaching a version with 422 parameters.
- The first layer used a custom activation called Ameo and later layers used tanh.
- Rounded weights doubled along each input, like a digital-to-analog converter.
- Neuron outputs were sine-like, with periods matching how binary digits switch while counting.
- Later layers pushed signals toward square waves near the targets of -1 and 1.
- Swapping the first activation for sin(x) worked in much the same way.
- The author allows that the result may be a fluke of this architecture.
What evidence it gives
The post shows the network layout, the rounded weights and plots of neuron outputs. The author validated accuracy on all input combinations during training. Larger networks reached near-perfect results in almost every run. At 422 parameters, perfect solutions became less common. No code is given in the post.
Interests and limits
The author uses a custom activation function from earlier work and trains small networks. The findings come from one setup. The author says the solution may be a fluke of the architecture. The post makes no test on larger models. The closing idea about big models is speculation. The author tempers it by citing The Bitter Lesson.