Unitary/Orthogonal matrices are known best solving gradient vanishing/explosion problems. This approach leads to new type of RNNs and has the potential to next level natural language models. It can also be applied to meta learning and reinforcement learning models.
Rotational Unit of Memory
Rumen Dangovski*, Li Jing*, Marin Soljačić
Transactions of the Association for Computational Linguistics
Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
Li Jing*, Yichen Shen*, Tena Dubček, John Peurifoy, Scott Skirlo, Yann LeCun, Max Tegmark, Marin Soljačić
International Conference on Machine Learning (ICML) 2017
Applying modern deep learning tools to frontier physics research with massive data has significant advantage. Unconscious reasoning plays an important role in human decision. This is the same for physical world. Going with data directly and building intuitive models surprisingly provide rich understanding in physics.