Deep Learning–Based 3D Segmentation and Spatial Mapping for Tropical Heritage Decay Diagnosis: A Framework from Semarang Sites
Abstract
The diagnosis of chromatic decay in tropical heritage buildings remains a critical yet underexplored challenge, constrained by the instability of surface conditions and the inadequacy of traditional visual documentation. This research proposes an advanced hybrid framework that fuses unsupervised hierarchical clustering with supervised Random Forest classification to detect and delineate chromatic deterioration from photogrammetric RGB point clouds of four heritage sites in Semarang, Indonesia. The unsupervised model, operating on HSV-transformed color data, autonomously identifies spectral irregularities across complex masonry surfaces, while the supervised pipeline refines decay categorization through expert-annotated UV texture maps. Validation against ground-truth datasets confirmed the robustness of both approaches: the unsupervised clustering achieved mean precision above 85% and an F1-score over 0.83, showing strong resilience to variable illumination and material heterogeneity, whereas the supervised method delivered higher class discrimination at the cost of intensive manual labeling. The comparative evaluation underscores the complementary strengths of both strategies—automation and scalability in clustering, interpretive accuracy in supervised learning. By mitigating spectral ambiguity and minimizing annotation dependency, the framework establishes a semi-automated protocol for decay mapping at architectural and urban scales. The study advances digital heritage diagnostics by providing a transferable, data-efficient methodology for preventive conservation in climate-sensitive, data-limited tropical contexts.
Keywords
Full Text:
PDFReferences
Adamopoulos, E., Rinaudo, F., & Ardissono, L. (2020). A Critical Comparison of 3D Digitization Techniques for Heritage Objects. ISPRS International Journal of Geo-Information, 10(1), 10. https://doi.org/10.3390/ijgi10010010
Ahan, M. M., Nouri, A. S., & Matzarakis, A. (2023). Investigating the Relationship of Outdoor Heat Stress upon Indoor Thermal Comfort and Qualitative Sleep Evaluation: The Case of Ankara. Atmosphere, 14(9). https://doi.org/10.3390/atmos14091407
Anagnostopoulos, T., Zaslavsky, A., Kolomvatsos, K., Medvedev, A., Amirian, P., Morley, J., & Hadjieftymiades, S. (2017). Challenges and Opportunities of Waste Management in IoT-Enabled Smart Cities: A Survey. IEEE Transactions on Sustainable Computing, 2(3), 275–289. https://doi.org/10.1109/TSUSC.2017.2691049
Arganda-Carreras, I., Kaynig, V., Rueden, C., Eliceiri, K. W., Schindelin, J., Cardona, A., & Sebastian Seung, H. (2017). Trainable Weka Segmentation: a machine learning tool for microscopy pixel classification. Bioinformatics, 33(15), 2424–2426. https://doi.org/10.1093/bioinformatics/btx180
Betsas, T., Georgopoulos, A., Doulamis, A., & Grussenmeyer, P. (2025). Deep Learning on 3D Semantic Segmentation: A Detailed Review. Remote Sensing, 17(2), 298. https://doi.org/10.3390/rs17020298
Boccarusso, L., D’Addona, D. M., Durante, M., Fazio, D. De, Capece Minutolo, F. M., & Langella, A. (2020). A study on the drilling process of hemp/epoxy composites by using different tools. Procedia CIRP, 88, 462–466. https://doi.org/10.1016/j.procir.2020.05.080
Bruno, S., Galantucci, R. A., & Musicco, A. (2023). Decay detection in historic buildings through image-based deep learning. VITRUVIO - International Journal of Architectural Technology and Sustainability, 8, 6–17. https://doi.org/10.4995/vitruvio-ijats.2023.18662
Busin, L., Vandenbroucke, N., & Macaire, L. (2009). Color Spaces and Image Segmentation (pp. 65–168). https://doi.org/10.1016/S1076-5670(07)00402-8
Camps-Valls, G., Fernández-Torres, M.-Á., Cohrs, K.-H., Höhl, A., Castelletti, A., Pacal, A., Robin, C., Martinuzzi, F., Papoutsis, I., Prapas, I., Pérez-Aracil, J., Weigel, K., Gonzalez-Calabuig, M., Reichstein, M., Rabel, M., Giuliani, M., Mahecha, M. D., Popescu, O.-I., Pellicer-Valero, O. J., … Williams, T. (2025). Artificial intelligence for modeling and understanding extreme weather and climate events. Nature Communications, 16(1), 1919. https://doi.org/10.1038/s41467-025-56573-8
Chiabrando, F., Lo Turco, M., & Rinaudo, F. (2017). MODELING THE DECAY IN AN HBIM STARTING FROM 3D POINT CLOUDS. A FOLLOWED APPROACH FOR CULTURAL HERITAGE KNOWLEDGE. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W5, 605–612. https://doi.org/10.5194/isprs-archives-XLII-2-W5-605-2017
De Fino, M., Bruno, S., & Fatiguso, F. (2022). Dissemination, assessment and management of historic buildings by thematic virtual tours and 3D models. Virtual Archaeology Review, 13(26), 88–102. https://doi.org/10.4995/var.2022.15426
De Fino, M., Galantucci, R. A., & Fatiguso, F. (2023). Condition Assessment of Heritage Buildings via Photogrammetry: A Scoping Review from the Perspective of Decision Makers. Heritage, 6(11), 7031–7067. https://doi.org/10.3390/heritage6110367
De Fino, M., Scioti, A., Rubino, R., Pierucci, A., & Fatiguso, F. (2018). ‘Augmented diagnostics’ for the architectural heritage. International Journal of Heritage Architecture: Studies, Repairs and Maintence, 2(2), 248–260. https://doi.org/10.2495/ha-v2-n2-248-260
El Mankibi, M., Cantin, R., & Zoubir, A. (2015). Contribution to the thermal renovation of old buildings: Numerical and Experimental approach for characterizing a double window. Energy Procedia, 78(0), 2470–2475. https://doi.org/10.1016/j.egypro.2015.11.231
Forte, M. (2012). Cyber-Archaeology: Notes on the simulation of the past. Virtual Archaeology Review, 2(4), 7. https://doi.org/10.4995/var.2011.4543
Gadd, G. M., Fomina, M., & Pinzari, F. (2024). Fungal biodeterioration and preservation of cultural heritage, artwork, and historical artifacts: extremophily and adaptation. Microbiology and Molecular Biology Reviews, 88(1). https://doi.org/10.1128/mmbr.00200-22
Galantucci, R. A., Musicco, A., Verdoscia, C., & Fatiguso, F. (2025). Machine Learning for the Semi-Automatic 3D Decay Segmentation and Mapping of Heritage Assets. International Journal of Architectural Heritage, 19(3), 389–407. https://doi.org/10.1080/15583058.2023.2287152
Gbran, H. (2024a). From Physical Models to Innovations: Technology Advances in Architectural and Civil Engineering. Arsitektura, 22(1), 91. https://doi.org/10.20961/arst.v22i1.81909
Gbran, H. (2024b). Preserving History in a Modern Setting “An Adaptive Redesign of Lawang Sewu for Sustainable Development and Architectural Heritage Conservation.” International Journal of Architecture and Urbanism, Vol. 07, N. https://doi.org/10.32734/ijau.v7i3.13748
Gbran, H. (2024c). Strategies for using sustainable urban drainage systems in the architectural design of civil infrastructure projects in Jakarta. International Journal of Architecture and Urbanism, 08(03), 358–370. https://doi.org/10.32734/ijau.v8i3.17178
Gbran, H., & Ratih Sari, S. (2024). Studying the visual impact of modern construction on historic cityscapes: a case study of Lawang Sewu building, Indonesia. Contexto, 18(28), 15–34. https://doi.org/10.29105/contexto18.28-410
Gbran, H., Rukayah, S., & Suprapti, A. (2025). Smart Heritage Management and Economic Sustainability Through AI and IoT : A Digital Strategy for Lawang Sewu in Indonesia. JESH: Journal of Social, Economics, and Humanities, 3(1). https://doi.org/10.30595/jesh.v3i1.320
Gbran, H., Rukayah, S., Suprapti, A., & Pandelaki, E. (2025a). AI-IoT for tropical heritage conservation: scoping review and case framework Lawang Sewu. Proceedings of the Institution of Civil Engineers - Engineering History and Heritage. https://doi.org/10.1680/jenhh.25.00023
Gbran, H., Rukayah, S., Suprapti, A., & Pandelaki, E. E. (2025b). Innovative Strategies for Optimizing Energy Efficiency and Thermal Comfort in Heritage Architecture: The Case of Lawang Sewu. Nternational Journal of Energy Technology, 16(2), 90–101. https://www.researchtrend.net/ijet/pdf/Innovative-Strategies-for-Optimizing-Energy-Efficiency-and-Thermal-Comfort-in-Heritage-Architecture-The-Case-of-Lawang-Sewu-Hassan-Gbran-14.pdf
Gbran, H., & Sari, S. R. (2023). The Visual Impact of Modern Constructions on the Old Cities in Indonesia: The Lawang Sewu Building in Semarang. ISVS E-Journal, 10(2), 71–90. https://isvshome.com/pdf/ISVS_10-2/ISVSej_10.2.5_Hasan.pdf
Gherardini, F., Santachiara, M., & Leali, F. (2019). Enhancing heritage fruition through 3D virtual models and augmented reality: an application to Roman artefacts. Virtual Archaeology Review, 10(21), 67. https://doi.org/10.4995/var.2019.11918
Golovkina, A. G., Karpukhin, O. R., Kravchenko, A. V., Khairullina, E. M., Tumkin, I. I., & Kalinichev, A. V. (2024). Digital color analysis and machine learning for ballpoint pen ink clustering and aging investigation. Forensic Science International, 364, 112236. https://doi.org/10.1016/j.forsciint.2024.112236
Gonizzi Barsanti, S., Guidi, G., & De Luca, L. (2017). SEGMENTATION of 3D MODELS for CULTURAL HERITAGE STRUCTURAL ANALYSIS - SOME CRITICAL ISSUES. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 4(2W2), 115–122. https://doi.org/10.5194/isprs-annals-IV-2-W2-115-2017
Grilli, E., Menna, F., & Remondino, F. (2017). A review of point cloud segmentation and classification algorithms. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W3, 339–344. https://doi.org/10.5194/isprs-archives-XLII-2-W3-339-2017
Grilli, E., & Remondino, F. (2019). Classification of 3D Digital Heritage. Remote Sensing, 11(7), 847. https://doi.org/10.3390/rs11070847
Guerra, M. G., & Galantucci, R. A. (2020). Standard quantification and measurement of damages through features characterization of surface imperfections on 3D models: An application on architectural heritages. Procedia CIRP, 88(July 2019), 515–520. https://doi.org/10.1016/j.procir.2020.05.089
Gupta, P., Nguyen, T. N., Gonzalez, C., & Woolley, A. W. (2025). Fostering Collective Intelligence in Human–AI Collaboration: Laying the Groundwork for COHUMAIN. Topics in Cognitive Science, 17(2), 189–216. https://doi.org/10.1111/tops.12679
Hackel, T., Wegner, J. D., & Schindler, K. (2016). Fast Semantic Segmentation of 3D Point Clouds With Strongly Varying Density. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, III–3(July), 177–184. https://doi.org/10.5194/isprsannals-iii-3-177-2016
Icomos, F., Roland, P., Training, T., International, E., Committee, S., Icomos, S., Committees, N., Plan, I. S., Committee, I. E., Ethical, I., & Statement, C. (2002). Ethical Commitment Statement for ICOMOS Members ( Revision , November 2002 , Madrid ). November, 1–6.
Jadhav, A. (2025). Hybrid Approaches in Fraud Detection : Combining Supervised and Unsupervised Learning. 11(3), 3–4. https://ijsret.com/wp-content/uploads/IJSRET_V11_issue3_1116.pdf
Jiang, Z., Xia, Q., Wang, Z., Zhu, K., Su, Q., Wang, J., Huang, Y., Wu, B., & Hong, Y. (2025). Cultural Heritage Color Regeneration: Interactive Genetic Algorithm Optimization Based on Color Network and Harmony Models. Applied Sciences, 15(4), 1720. https://doi.org/10.3390/app15041720
Krull, A., Buchholz, T.-O., & Jug, F. (2019). Noise2Void - Learning Denoising From Single Noisy Images. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2124–2132. https://doi.org/10.1109/CVPR.2019.00223
Ladiana, D., & Di Sivo, M. (2019). Programmed conservation of historical and architectural heritage. tools for optimising a process based on knowledge and information. International Journal of Design and Nature and Ecodynamics, 14(3), 229–240. https://doi.org/10.2495/DNE-V14-N3-229-240
Lerario, A., & Varasano, A. (2020). An IoT Smart Infrastructure for S. Domenico Church in Matera’s “Sassi’’: A Multiscale Perspective to Built Heritage Conservation. Sustainability, 12(16), 6553. https://doi.org/10.3390/su12166553
Lercari, N. (2019). Monitoring earthen archaeological heritage using multi-temporal terrestrial laser scanning and surface change detection. Journal of Cultural Heritage, 39, 152–165. https://doi.org/10.1016/j.culher.2019.04.005
Letellier, R. (2016). Recording, Documentation and Information Management for the Conservation of Heritage Places. In Recording, Documentation and Information Management for the Conservation of Heritage Places. https://doi.org/10.4324/9781315793917
Li, X., Yang, H., Chen, C., Zhao, G., & Ni, J. (2024). Deterioration identification of stone cultural heritage based on hyperspectral image texture features. Journal of Cultural Heritage, 69, 57–66. https://doi.org/10.1016/j.culher.2024.07.011
Li, Z., Kamnitsas, K., & Glocker, B. (2021). Analyzing Overfitting Under Class Imbalance in Neural Networks for Image Segmentation. IEEE Transactions on Medical Imaging, 40(3), 1065–1077. https://doi.org/10.1109/TMI.2020.3046692
Michele Russo, Eleonora Grilli, Fabio Remondino, Simone Teruggi, & Francesco Fassi. (2021a). Machine Learning for Cultural Heritage Classification. In Representation Challenges. Augmented Reality and Artificial Intelligence in Cultural Heritage and Innovative Design Domain. FrancoAngeli srl. https://doi.org/10.3280/oa-686.33
Michele Russo, Eleonora Grilli, Fabio Remondino, Simone Teruggi, & Francesco Fassi. (2021b). Machine Learning for Cultural Heritage Classification. In Representation Challenges. Augmented Reality and Artificial Intelligence in Cultural Heritage and Innovative Design Domain. FrancoAngeli srl. https://doi.org/10.3280/oa-686.33
Microimaging, M. (2022). Contrast and Feature Enhancement in Fiji. Guide to Brightness/Contrast in Fiji
Mobarak, M. H., Islam, M. A., Hossain, N., Al Mahmud, M. Z., Rayhan, M. T., Nishi, N. J., & Chowdhury, M. A. (2023). Recent advances of additive manufacturing in implant fabrication – A review. Applied Surface Science Advances, 18, 100462. https://doi.org/10.1016/j.apsadv.2023.100462
Murtagh, F., & Legendre, P. (2015). Ward’s Hierarchical Agglomerative Clustering Method: Which Algorithms Implement Ward’s Criterion? Journal of Classification, 31(3), 274–295. https://doi.org/10.1007/s00357-014-9161-z
Onaka, T. (2009). a Study of the Documentation Process for Conservation of Architectural Heritage Sites : Illustrated By Examples From Egyptand Belgium Abstract : 22nd CIPA Symposium, October 11-15, 2009, Kyoto, Japan, 1–6. http://cipa.icomos.org/fileadmin/template/doc/KYOTO/188.pdf
Pang, B., Yang, J., Xia, T., Zhang, A., Zhang, K., Xu, Q., & Wang, F. (2025). Automated heritage building component recognition and modelling based on local features. Journal of Cultural Heritage, 71, 252–264. https://doi.org/10.1016/j.culher.2024.12.006
Patankar, N., Dixit, S., Bhamare, A., Darpel, A., & Raina, R. (2021). Customer Segmentation Using Machine Learning. Advances in Parallel Computing, 39(May 2023), 239–244. https://doi.org/10.3233/APC210200
Pigawati, M. A. dan B. (2017). Pingembangan Kawasan Vihara Budhajaya Wattugong Sebagai Objek Wisata Di Kota Semarang. Jurusan Perencanaan Wilayah Dan Kota, Fakultas Teknik, Universitas Diponegoro, Semarang.
Prasetyo, M. H., Lihawa, F., & Baderan, D. W. K. (2024). Potensi Model Sistem Dinamik dalam Sistem Pengelolaan Sampah Perkotaan.
Razia Sulthana, A., Jaiswal, A., Supraja, P., & Sairamesh, L. (2023). Customer Segmentation using Machine Learning. 2023 3rd International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies, ICAECT 2023, 9(6), 956–965. https://doi.org/10.1109/ICAECT57570.2023.10117924
Revol-Muller, C., Grenier, T., Rose, J.-L., Pacureanu, A., Peyrin, F., & Odet, C. (2014). Region Growing: When Simplicity Meets Theory – Region Growing Revisited in Feature Space and Variational Framework (pp. 426–444). https://doi.org/10.1007/978-3-642-38241-3_29
Rosina, E., & Scazzosi, L. (2019). the Conservation and Enhancement of Built.
Sánchez-Aparicio, L. J., Del Pozo, S., Ramos, L. F., Arce, A., & Fernandes, F. M. (2018). Heritage site preservation with combined radiometric and geometric analysis of TLS data. Automation in Construction, 85(September 2017), 24–39. https://doi.org/10.1016/j.autcon.2017.09.023
Sánchez-Aparicio, L. J., Del Pozo, S., Ramos, L. F., & Fernandes, F. M. (2019). Geometric and Radiometric Analysis of TLS Point Clouds to Diagnose the Conservation State of Historical Constructions. Case of Study in the Master Gate of San Francisco, Almeida, Portugal (pp. 465–473). https://doi.org/10.1007/978-3-319-99441-3_50
Sanchez, T. W., Brenman, M., & Ye, X. (2024). The Ethical Concerns of Artificial Intelligence in Urban Planning. Journal of the American Planning Association, 91(2), 294–307. https://doi.org/10.1080/01944363.2024.2355305
Sardiyarso, E. S., Puspitasari, P., Walaretina, R., & Lakawa, A. R. (2023). The reading of borobudur temple reliefs: Virtue architecture. In AIP Conference Proceedings (Vol. 2706). https://doi.org/10.1063/5.0120648
Sudikno, A., & Surjono, S. (2017). Pelestarian Eks Pusat Kota Kolonial Lama Semarang. Prosiding Seminar Nasional Perencanaan Wilayah Dan Kota, Universitas Brawijaya, March.
Widiatmika, K. P. (2015). VOLUNTARY NATIONAL REVIEW FOR IMPLEMENTATION OF NEW URBAN AGENDA. Etika Jurnalisme Pada Koran Kuning : Sebuah Studi Mengenai Koran Lampu Hijau, 16(2), 39–55.
Y. Boffill, H. Blanco, I. L. &L. V. (2020). Structural Assessment of Ancient Masonries before and during a Rehabilitation Process. International Journal of Architectural Heritage. https://doi.org/10.1080/15583058.2018.1539784
Yang, H., Xu, Z., Shi, Y., Tang, W., Liu, C., Yunusa-Kaltungo, A., & Cui, H. (2023). Multi-objective optimization designs of phase change material-enhanced building using the integration of the Stacking model and NSGA-III algorithm. Journal of Energy Storage, 68, 107807. https://doi.org/10.1016/j.est.2023.107807
DOI: https://doi.org/10.17509/jare.v8i2.87742
Refbacks
- There are currently no refbacks.
Copyright (c) 2026 Journal of Architectural Research and Education

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.






