What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Code

Yao Wan, Wei Zhao, Hongyu Zhang, Yulei Sui, Guandong Xu, Hai Jin

Type: Preprint

Publication Date: 2022-01-01

Citations: 15

DOI: https://doi.org/10.48550/arxiv.2202.06840

View Publication

Locations

arXiv (Cornell University) - View
DataCite API - View

Similar Works

Action	Title	Year	Authors
+ PDF Chat	What do pre-trained code models know about code?	2021	Anjan Karmakar Romain Robbes
+	What do pre-trained code models know about code?	2021	Anjan Karmakar Romain Robbes
+	What do pre-trained code models know about code?	2021	Anjan Karmakar Romain Robbes
+ PDF Chat	GraphCodeBERT: Pre-training Code Representations with Data Flow	2020	Daya Guo Shuo Ren Shuai Lu Zhangyin Feng Duyu Tang Shujie Liu Long Zhou Nan Duan A. Svyatkovskiy Shengyu Fu
+	GraphCodeBERT: Pre-training Code Representations with Data Flow	2020	Daya Guo Shuo Ren Shuai Lu Zhangyin Feng Duyu Tang Shujie Liu Long Zhou Nan Duan A. Svyatkovskiy Sheng‐Yu Fu
+	Are Code Pre-trained Models Powerful to Learn Code Syntax and Semantics?	2022	Wei Ma Mengjie Zhao Xiaofei Xie Qiang Hu Shangqing Liu Jie Zhang Wenhan Wang Yang Liu
+ PDF Chat	Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities	2024	Wei Ma Shangqing Liu Mengjie Zhao Xiaofei Xie Wenhan Wang Qiang Hu Jie Zhang Yang Liu
+	An Exploratory Study on Code Attention in BERT	2022	Rishab Sharma Fuxiang Chen Fatemeh H. Fard David Lo
+ PDF Chat	Diet code is healthy: simplifying programs for pre-trained models of code	2022	Zhaowei Zhang Hong‐yu Zhang Beijun Shen Xiaodong Gu
+	An exploratory study on code attention in BERT	2022	Rishab Sharma Fuxiang Chen Fatemeh H. Fard David Lo
+	Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code	2023	Ziyin Zhang Chaoyu Chen Bingchang Liu Cong Liao Zi Gong Hang Yu Jianguo Li Rui Wang
+	CLSEBERT: Contrastive Learning for Syntax Enhanced Code Pre-Trained Model.	2021	Xin Wang Yasheng Wang Pingyi Zhou Fei Mi Xiao Meng Yadao Wang Li Li Xiao Liu Hao Wu Jin Liu
+	CAT-probing: A Metric-based Approach to Interpret How Pre-trained Models for Programming Language Attend Code Structure	2022	Nuo Chen Qiushi Sun Renyu Zhu Xiang Li Xuesong Lu Ming Gao
+	CAT-probing: A Metric-based Approach to Interpret How Pre-trained Models for Programming Language Attend Code Structure	2022	Nuo Chen Qiushi Sun Renyu Zhu Xiang Li Xuesong Lu Ming Gao
+	Pre-trained Contextual Embedding of Source Code	2019	Aditya Kanade Petros Maniatis Gogul Balakrishnan Kensen Shi
+	How to get better embeddings with code pre-trained models? An empirical study	2023	Yu Zhao Lina Gong Haoxiang Zhang Yaoshen Yu Zhiqiu Huang
+	Structured Code Representations Enable Data-Efficient Adaptation of Code Language Models	2024	Mayank Agarwal Yikang Shen Bailin Wang Yoon Kim Chen Jie
+ PDF Chat	Looking into Black Box Code Language Models	2024	Muhammad Umair Haider Umar Farooq A. B. Siddique Mark Marron
+	Bridging Pre-trained Models and Downstream Tasks for Source Code Understanding	2021	Deze Wang Zhouyang Jia Shanshan Li Yue Yu Yun Xiong Wei Dong Xiangke Liao
+	TreeBERT: A Tree-Based Pre-Trained Model for Programming Language	2021	Xue Jiang Zhuoran Zheng Chen Lyu Liang Li Lei Lyu

Works That Cite This (10)

Action	Title	Year	Authors
+ PDF Chat	Diet code is healthy: simplifying programs for pre-trained models of code	2022	Zhaowei Zhang Hong‐yu Zhang Beijun Shen Xiaodong Gu
+ PDF Chat	CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training	2022	Xin Wang Yasheng Wang Yao Wan Jiawei Wang Pingyi Zhou Li Li Hao Wu Jin Liu
+	Is Model Attention Aligned with Human Attention? An Empirical Study on Large Language Models for Code Generation	2023	Bonan Kou Shengmai Chen Zhijie Wang Lei Ma Tianyi Zhang
+	CAT-probing: A Metric-based Approach to Interpret How Pre-trained Models for Programming Language Attend Code Structure	2022	Nuo Chen Qiushi Sun Renyu Zhu Xiang Li Xuesong Lu Ming Gao
+ PDF Chat	INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers	2023	Anjan Karmakar Romain Robbes
+	Towards Trustworthy AI Software Development Assistance	2023	Daniel Maninger Krishna Narasimhan Mira Mezini
+ PDF Chat	APIContext2Com: Code Comment Generation by Incorporating Pre-Defined API Documentation	2023	Ramin Shahbazi Fatemeh H. Fard
+	INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers	2023	Anjan Karmakar Romain Robbes
+	Which Syntactic Capabilities Are Statistically Learned by Masked Language Models for Code?	2024	Alejandro Velasco David N. Palacio Daniel Rodríguez-Cárdenas Denys Poshyvanyk
+	Which Syntactic Capabilities Are Statistically Learned by Masked Language Models for Code?	2024	Alejandro Velasco David N. Palacio Daniel Rodríguez-Cárdenas Denys Poshyvanyk

Works Cited by This (0)

Action	Title	Year	Authors