읽어야 할 Paper List

【Efficient ML】

- General

• (리뷰 完) Tan, Mingxing, and Quoc Le. "Efficientnet: Rethinking model scaling for convolutional neural  networks." International Conference on Machine Learning. PMLR, 2019.
• (리뷰 完) Howard, Andrew G., et al. "Mobilenets: Efficient convolutional neural networks for mobile vision applications."arXiv preprint arXiv:1704.04861(2017).

 

- Quantization

• (리뷰 完) Banner, Ron, et al. "Post-training 4-bit quantization of convolution networks for rapid-deployment."arXiv preprint arXiv:1810.05723(2018).
• Bhandare, Aishwarya, et al. "Efficient 8-bit quantization of transformer neural machine language translation model." arXiv preprint arXiv:1906.00532 (2019).
• Zafrir, Ofir, et al. "Q8bert: Quantized 8bit bert." arXiv preprint arXiv:1910.06188 (2019).
• Choi, Jungwook, et al. "Pact: Parameterized clipping activation for quantized neural networks." arXiv preprint arXiv:1805.06085 (2018).
• Bondarenko, Yelysei, Markus Nagel, and Tijmen Blankevoort. "Understanding and Overcoming the Challenges of Efficient Transformer Quantization." arXiv preprint arXiv:2109.12948 (2021).
• Oh, Sangyun, et al. "Automated Log-Scale Quantization for Low-Cost Deep Neural Networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

- Tensor Decomposition

• Ma, Xindian, et al. "A tensorized transformer for language modeling." Advances in Neural Information Processing Systems 32 (2019): 2232-2242.
• Yin, Miao, et al. "Towards Efficient Tensor Decomposition-Based DNN Model Compression With Optimization Framework." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

 

- Pruning

• Han, Song, et al. "Learning both weights and connections for efficient neural networks." arXiv preprint arXiv:1506.02626 (2015).
• Han, Song, Huizi Mao, and William J. Dally. "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding." arXiv preprint arXiv:1510.00149 (2015).
• Li, Hao, et al. "Pruning filters for efficient convnets." arXiv preprint arXiv:1608.08710 (2016).
• He, Yihui, et al. "Amc: Automl for model compression and acceleration on mobile devices." Proceedings of the European conference on computer vision (ECCV). 2018.
• Frankle, Jonathan, and Michael Carbin. "The lottery ticket hypothesis: Finding sparse, trainable neural networks." arXiv preprint arXiv:1803.03635 (2018).
• Liu, Zhuang, et al. "Rethinking the value of network pruning." arXiv preprint arXiv:1810.05270 (2018).

 

- Distillation

• Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 2.7 (2015).
• Romero, Adriana, et al. "Fitnets: Hints for thin deep nets." arXiv preprint arXiv:1412.6550 (2014).
• Zagoruyko, Sergey, and Nikos Komodakis. "Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer." arXiv preprint arXiv:1612.03928 (2016).
• Touvron, Hugo, et al. "Training data-efficient image transformers & distillation through attention." International Conference on Machine Learning. PMLR, 2021.

 

---

 

【Vision】

• Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).
• Tolstikhin, Ilya, et al. "Mlp-mixer: An all-mlp architecture for vision." arXiv preprint arXiv:2105.01601 (2021).
• Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." arXiv preprint arXiv:2103.14030 (2021).
• Arnab, Anurag, et al. "Vivit: A video vision transformer." arXiv preprint arXiv:2103.15691 (2021).
• Sachin, Mehta, et al. "MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer." arXiv preprint arXiv:2110.02178 (2021).  
• He, Kaiming, et al. "Masked Autoencoders Are Scalable Vision Learners." arXiv preprint arXiv:2111.06377 (2021).
• Bao, Hangbo, Li Dong, and Furu Wei. "BEiT: BERT Pre-Training of Image Transformers." arXiv preprint arXiv:2106.08254 (2021).
• Liu, Ze, et al. "Swin Transformer V2: Scaling Up Capacity and Resolution." arXiv preprint arXiv:2111.09883 (2021).
• Liu, Zhuang, et al. "A ConvNet for the 2020s." arXiv preprint arXiv:2201.03545 (2022).
• Baevski, Alexei, et al. "data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language."arXiv preprint arXiv:2202.03555(2022).
• Park, Namuk, and Songkuk Kim. "How Do Vision Transformers Work?." arXiv preprint arXiv:2202.06709 (2022).
• Liu, Ze, et al. "Video swin transformer." arXiv preprint arXiv:2106.13230 (2021).
• Xiao, Tete, et al. "Early convolutions help transformers see better." arXiv preprint arXiv:2106.14881 (2021).

 

---

 

【NLP】

• (리뷰 完) Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. "Neural machine translation by jointly learning to align and translate."arXiv preprint arXiv:1409.0473(2014).
• Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems. 2017.

 

---

 

【SNN】

• Lee, Jun Haeng, Tobi Delbruck, and Michael Pfeiffer. "Training deep spiking neural networks using backpropagation." Frontiers in neuroscience 10 (2016): 508.

 

---

 

【Hardware】

• Zhang, Wenqiang, et al. "Neuro-inspired computing chips." Nature Electronics 3.7 (2020): 371-382.
• Yao, Peng, et al. "Fully hardware-implemented memristor convolutional neural network." Nature 577.7792 (2020): 641-646.
• Chen, Huawei, et al. "Logic gates based on neuristors made from two-dimensional materials." Nature Electronics 4.6 (2021): 399-404.

 

---

 

【NAS】

• Gaier, Adam, and David Ha. "Weight agnostic neural networks." arXiv preprint arXiv:1906.04358 (2019).
• Knyazev, Boris, et al. "Parameter Prediction for Unseen Deep Architectures." Advances in Neural Information Processing Systems 34 (2021).

 

---

 

【RL】

• Rashid, Tabish, et al. "Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning." International Conference on Machine Learning. PMLR, 2018.

 

---

 

【Math】

• Vinyals, Oriol, Meire Fortunato, and Navdeep Jaitly. "Pointer networks." arXiv preprint arXiv:1506.03134 (2015).
• Blalock, Davis, and John Guttag. "Multiplying Matrices Without Multiplying." arXiv preprint arXiv:2106.10860 (2021).