05.09.2022 Going Deep
This was another week of learnings and stumbling so although I only worked on one ticket… it took some time to get here.
As plan of explaining my workings, I have published this blog post on the things I wanted to try and why I think they would work.
There is no actual code change to the main pipeline as all the testing was done on notebooks first. I created a small sample (~1.5k) from my personal user_data which was then used as training data for a typical Deep Neural Net and Recurrent Neural net. This results still need some scrutiny but there is good results from DNN 88% but worse results in RNN 75%, which is either interesting or I’ve implemented something wrong.
I plan to merge this into the main pipeline in the coming release so that we can start using these new models. I am slightly afraid of letting a small raspberry pi run neural network training so in the initial phase, I will set the default if installed as an add_on to not train neural nets.
The states snapshots was also getting a bit out of hand as I originally set it up to take a snap every 5 secs. This turned out to be a huge mistake as the db grew too large and any queries took forever. I have now removed this and plan to leverage the data from the states db.
This release will be focused on making things to ‘production’. A large part of the training and model saving I would want to store them on AWS. Also, I want to remove putting db creds on the code as its pretty awful so I will add a small login page on my website to have people signup (free!) for a token.
I am also busy this Friday so the release will be the week after!