Building a module to generate a dataset for training the image-based source code generation task

In this study, a new mechanism for generating training data for a neural network for the task of image-based code generation is proposed. In order for a system to be able to perform the task assigned to it, it must be trained. The initial dataset that is provided with the pix2code system allows the system to be trained, but it relies on the data that is provided in the domain-specific dictionary. Expanding or changing words in the dictionary does not affect the data set in any way, which limits the flexibility of the system's application by not allowing for the rules that may apply to the enterprise to be taken into account. Some studies claim to have created their own dataset, but its lack of public access makes it difficult to assess the complexity of the images it contains. To solve this problem, within the framework of this study, a submodule was developed that allows, based on a modified dictionary of a domain-specific language, to create a custom training dataset consisting of an image-source code pair corresponding to this image. To test the functionality of the created dataset, the modified pix2code system performed training and was then able to predict the code on test examples.

1. Beltramelli T. pix2code: Generating Code from a Graphical User Interface Screenshot. In: EICS '18: Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems, 19–22 June 2018, Paris, France. New York: Association for Computing Machinery; 2018. https://doi.org/10.1145/3220134.3220135

2. Zhu Zh., Xue Zh., Yuan Z. Automatic Graphics Program Generation Using Attention-Based Hierarchical Decoder. In: Computer Vision – ACCV 2018: 14th Asian Conference on Computer Vision: Revised Selected Papers: Part VI, 02–06 December 2018, Perth, Australia. Cham: Springer; 2019. P. 181–196. https://doi.org/10.1007/978-3-030-20876-9_12

3. Liu Ya., Hu Q., Shu K. Improving pix2code Based BI-directional LSTM. In: 2018 IEEE International Conference on Automation, Electronics and Electrical Engineering (AUTEEE), 16–18 November 2018, Shenyang, China. IEEE; 2019. P. 220–223. https://doi.org/10.1109/AUTEEE.2018.8720784

4. Nikitin I.V. Using ResNet and Transformers Architectures in the Problem of Source Code Generation from an Image. Modeling, Optimization and Information Technology. 2025;13(2). (In Russ.). https://doi.org/10.26102/2310-6018/2025.49.2.002

5. Nikitin I.V. Assessing the Quality of the Result in the Problem of Source Code Generation from an Image. Modeling, Optimization and Information Technology. 2025;13(1). (In Russ.). https://doi.org/10.26102/2310-6018/2025.48.1.030

6. Chen W.-Yi., Podstreleny P., Cheng W.-H., Chen Yu.-Ya., Hua K.-L. Code Generation from a Graphical User Interface Via Attention-Based Encoder-Decoder Model. Multimedia Systems. 2022;28(1):121–130. https://doi.org/10.1007/s00530-021-00804-7

7. Zou D., Wu G. Automatic Code Generation for Android Applications Based on Improved Pix2code. Journal of Artificial Intelligence and Technology. 2024;4(4):325–331. https://doi.org/10.37965/jait.2024.0515

8. Nguyen T.A., Csallner Ch. Reverse Engineering Mobile Application User Interfaces with REMAUI (T). In: 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 09–13 November 2015, Lincoln, NE, USA. IEEE; 2016. P. 248–259. URL: https://doi.org/10.1109/ASE.2015.32

9. Paul D.Gh., Zhu H., Bayley I. Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review. In: 2024 IEEE International Conference on Artificial Intelligence Testing (AITest), 15–18 July 2024, Shanghai, China. IEEE; 2024. P. 87–94. https://doi.org/10.1109/AITest62860.2024.00019

10. Ling W., Blunsom Ph., Grefenstette E., et al. Latent Predictor Networks for Code Generation. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016: Volume 1: Long Papers, 07–12 August 2016, Berlin, Germany. The Association for Computer Linguistics; 2016. P. 599–609. https://doi.org/10.18653/v1/P16-1057