Deep Learning–Based 3D Segmentation and Spatial Mapping for Tropical Heritage Decay Diagnosis: A Framework from Semarang Sites

Hassan Gbran, Siti Rukayah, Atik Suprapti

Abstract


The diagnosis of chromatic decay in tropical heritage buildings remains a critical yet underexplored challenge, constrained by the instability of surface conditions and the inadequacy of traditional visual documentation. This research proposes an advanced hybrid framework that fuses unsupervised hierarchical clustering with supervised Random Forest classification to detect and delineate chromatic deterioration from photogrammetric RGB point clouds of four heritage sites in Semarang, Indonesia. The unsupervised model, operating on HSV-transformed color data, autonomously identifies spectral irregularities across complex masonry surfaces, while the supervised pipeline refines decay categorization through expert-annotated UV texture maps. Validation against ground-truth datasets confirmed the robustness of both approaches: the unsupervised clustering achieved mean precision above 85% and an F1-score over 0.83, showing strong resilience to variable illumination and material heterogeneity, whereas the supervised method delivered higher class discrimination at the cost of intensive manual labeling. The comparative evaluation underscores the complementary strengths of both strategies—automation and scalability in clustering, interpretive accuracy in supervised learning. By mitigating spectral ambiguity and minimizing annotation dependency, the framework establishes a semi-automated protocol for decay mapping at architectural and urban scales. The study advances digital heritage diagnostics by providing a transferable, data-efficient methodology for preventive conservation in climate-sensitive, data-limited tropical contexts.


Keywords


Heritage Decay; Lawang Sewu Deep Learning; Tropical Heritage; Chromatic Segmentation; Unsupervised Classification

Full Text:

PDF

References


Adamopoulos, E., Rinaudo, F., & Ardissono, L. (2020). A Critical Comparison of 3D Digitization Techniques for Heritage Objects. ISPRS International Journal of Geo-Information, 10(1), 10. https://doi.org/10.3390/ijgi10010010

Ahan, M. M., Nouri, A. S., & Matzarakis, A. (2023). Investigating the Relationship of Outdoor Heat Stress upon Indoor Thermal Comfort and Qualitative Sleep Evaluation: The Case of Ankara. Atmosphere, 14(9). https://doi.org/10.3390/atmos14091407

Anagnostopoulos, T., Zaslavsky, A., Kolomvatsos, K., Medvedev, A., Amirian, P., Morley, J., & Hadjieftymiades, S. (2017). Challenges and Opportunities of Waste Management in IoT-Enabled Smart Cities: A Survey. IEEE Transactions on Sustainable Computing, 2(3), 275–289. https://doi.org/10.1109/TSUSC.2017.2691049

Arganda-Carreras, I., Kaynig, V., Rueden, C., Eliceiri, K. W., Schindelin, J., Cardona, A., & Sebastian Seung, H. (2017). Trainable Weka Segmentation: a machine learning tool for microscopy pixel classification. Bioinformatics, 33(15), 2424–2426. https://doi.org/10.1093/bioinformatics/btx180

Betsas, T., Georgopoulos, A., Doulamis, A., & Grussenmeyer, P. (2025). Deep Learning on 3D Semantic Segmentation: A Detailed Review. Remote Sensing, 17(2), 298. https://doi.org/10.3390/rs17020298

Boccarusso, L., D’Addona, D. M., Durante, M., Fazio, D. De, Capece Minutolo, F. M., & Langella, A. (2020). A study on the drilling process of hemp/epoxy composites by using different tools. Procedia CIRP, 88, 462–466. https://doi.org/10.1016/j.procir.2020.05.080

Bruno, S., Galantucci, R. A., & Musicco, A. (2023). Decay detection in historic buildings through image-based deep learning. VITRUVIO - International Journal of Architectural Technology and Sustainability, 8, 6–17. https://doi.org/10.4995/vitruvio-ijats.2023.18662

Busin, L., Vandenbroucke, N., & Macaire, L. (2009). Color Spaces and Image Segmentation (pp. 65–168). https://doi.org/10.1016/S1076-5670(07)00402-8

Camps-Valls, G., Fernández-Torres, M.-Á., Cohrs, K.-H., Höhl, A., Castelletti, A., Pacal, A., Robin, C., Martinuzzi, F., Papoutsis, I., Prapas, I., Pérez-Aracil, J., Weigel, K., Gonzalez-Calabuig, M., Reichstein, M., Rabel, M., Giuliani, M., Mahecha, M. D., Popescu, O.-I., Pellicer-Valero, O. J., … Williams, T. (2025). Artificial intelligence for modeling and understanding extreme weather and climate events. Nature Communications, 16(1), 1919. https://doi.org/10.1038/s41467-025-56573-8

Chiabrando, F., Lo Turco, M., & Rinaudo, F. (2017). MODELING THE DECAY IN AN HBIM STARTING FROM 3D POINT CLOUDS. A FOLLOWED APPROACH FOR CULTURAL HERITAGE KNOWLEDGE. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W5, 605–612. https://doi.org/10.5194/isprs-archives-XLII-2-W5-605-2017

De Fino, M., Bruno, S., & Fatiguso, F. (2022). Dissemination, assessment and management of historic buildings by thematic virtual tours and 3D models. Virtual Archaeology Review, 13(26), 88–102. https://doi.org/10.4995/var.2022.15426

De Fino, M., Galantucci, R. A., & Fatiguso, F. (2023). Condition Assessment of Heritage Buildings via Photogrammetry: A Scoping Review from the Perspective of Decision Makers. Heritage, 6(11), 7031–7067. https://doi.org/10.3390/heritage6110367

De Fino, M., Scioti, A., Rubino, R., Pierucci, A., & Fatiguso, F. (2018). ‘Augmented diagnostics’ for the architectural heritage. International Journal of Heritage Architecture: Studies, Repairs and Maintence, 2(2), 248–260. https://doi.org/10.2495/ha-v2-n2-248-260

El Mankibi, M., Cantin, R., & Zoubir, A. (2015). Contribution to the thermal renovation of old buildings: Numerical and Experimental approach for characterizing a double window. Energy Procedia, 78(0), 2470–2475. https://doi.org/10.1016/j.egypro.2015.11.231

Forte, M. (2012). Cyber-Archaeology: Notes on the simulation of the past. Virtual Archaeology Review, 2(4), 7. https://doi.org/10.4995/var.2011.4543

Gadd, G. M., Fomina, M., & Pinzari, F. (2024). Fungal biodeterioration and preservation of cultural heritage, artwork, and historical artifacts: extremophily and adaptation. Microbiology and Molecular Biology Reviews, 88(1). https://doi.org/10.1128/mmbr.00200-22

Galantucci, R. A., Musicco, A., Verdoscia, C., & Fatiguso, F. (2025). Machine Learning for the Semi-Automatic 3D Decay Segmentation and Mapping of Heritage Assets. International Journal of Architectural Heritage, 19(3), 389–407. https://doi.org/10.1080/15583058.2023.2287152

Gbran, H. (2024a). From Physical Models to Innovations: Technology Advances in Architectural and Civil Engineering. Arsitektura, 22(1), 91. https://doi.org/10.20961/arst.v22i1.81909

Gbran, H. (2024b). Preserving History in a Modern Setting “An Adaptive Redesign of Lawang Sewu for Sustainable Development and Architectural Heritage Conservation.” International Journal of Architecture and Urbanism, Vol. 07, N. https://doi.org/10.32734/ijau.v7i3.13748

Gbran, H. (2024c). Strategies for using sustainable urban drainage systems in the architectural design of civil infrastructure projects in Jakarta. International Journal of Architecture and Urbanism, 08(03), 358–370. https://doi.org/10.32734/ijau.v8i3.17178

Gbran, H., & Ratih Sari, S. (2024). Studying the visual impact of modern construction on historic cityscapes: a case study of Lawang Sewu building, Indonesia. Contexto, 18(28), 15–34. https://doi.org/10.29105/contexto18.28-410

Gbran, H., Rukayah, S., & Suprapti, A. (2025). Smart Heritage Management and Economic Sustainability Through AI and IoT : A Digital Strategy for Lawang Sewu in Indonesia. JESH: Journal of Social, Economics, and Humanities, 3(1). https://doi.org/10.30595/jesh.v3i1.320

Gbran, H., Rukayah, S., Suprapti, A., & Pandelaki, E. (2025a). AI-IoT for tropical heritage conservation: scoping review and case framework Lawang Sewu. Proceedings of the Institution of Civil Engineers - Engineering History and Heritage. https://doi.org/10.1680/jenhh.25.00023

Gbran, H., Rukayah, S., Suprapti, A., & Pandelaki, E. E. (2025b). Innovative Strategies for Optimizing Energy Efficiency and Thermal Comfort in Heritage Architecture: The Case of Lawang Sewu. Nternational Journal of Energy Technology, 16(2), 90–101. https://www.researchtrend.net/ijet/pdf/Innovative-Strategies-for-Optimizing-Energy-Efficiency-and-Thermal-Comfort-in-Heritage-Architecture-The-Case-of-Lawang-Sewu-Hassan-Gbran-14.pdf

Gbran, H., & Sari, S. R. (2023). The Visual Impact of Modern Constructions on the Old Cities in Indonesia: The Lawang Sewu Building in Semarang. ISVS E-Journal, 10(2), 71–90. https://isvshome.com/pdf/ISVS_10-2/ISVSej_10.2.5_Hasan.pdf

Gherardini, F., Santachiara, M., & Leali, F. (2019). Enhancing heritage fruition through 3D virtual models and augmented reality: an application to Roman artefacts. Virtual Archaeology Review, 10(21), 67. https://doi.org/10.4995/var.2019.11918

Golovkina, A. G., Karpukhin, O. R., Kravchenko, A. V., Khairullina, E. M., Tumkin, I. I., & Kalinichev, A. V. (2024). Digital color analysis and machine learning for ballpoint pen ink clustering and aging investigation. Forensic Science International, 364, 112236. https://doi.org/10.1016/j.forsciint.2024.112236

Gonizzi Barsanti, S., Guidi, G., & De Luca, L. (2017). SEGMENTATION of 3D MODELS for CULTURAL HERITAGE STRUCTURAL ANALYSIS - SOME CRITICAL ISSUES. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 4(2W2), 115–122. https://doi.org/10.5194/isprs-annals-IV-2-W2-115-2017

Grilli, E., Menna, F., & Remondino, F. (2017). A review of point cloud segmentation and classification algorithms. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W3, 339–344. https://doi.org/10.5194/isprs-archives-XLII-2-W3-339-2017

Grilli, E., & Remondino, F. (2019). Classification of 3D Digital Heritage. Remote Sensing, 11(7), 847. https://doi.org/10.3390/rs11070847

Guerra, M. G., & Galantucci, R. A. (2020). Standard quantification and measurement of damages through features characterization of surface imperfections on 3D models: An application on architectural heritages. Procedia CIRP, 88(July 2019), 515–520. https://doi.org/10.1016/j.procir.2020.05.089

Gupta, P., Nguyen, T. N., Gonzalez, C., & Woolley, A. W. (2025). Fostering Collective Intelligence in Human–AI Collaboration: Laying the Groundwork for COHUMAIN. Topics in Cognitive Science, 17(2), 189–216. https://doi.org/10.1111/tops.12679

Hackel, T., Wegner, J. D., & Schindler, K. (2016). Fast Semantic Segmentation of 3D Point Clouds With Strongly Varying Density. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, III–3(July), 177–184. https://doi.org/10.5194/isprsannals-iii-3-177-2016

Icomos, F., Roland, P., Training, T., International, E., Committee, S., Icomos, S., Committees, N., Plan, I. S., Committee, I. E., Ethical, I., & Statement, C. (2002). Ethical Commitment Statement for ICOMOS Members ( Revision , November 2002 , Madrid ). November, 1–6.

Jadhav, A. (2025). Hybrid Approaches in Fraud Detection : Combining Supervised and Unsupervised Learning. 11(3), 3–4. https://ijsret.com/wp-content/uploads/IJSRET_V11_issue3_1116.pdf

Jiang, Z., Xia, Q., Wang, Z., Zhu, K., Su, Q., Wang, J., Huang, Y., Wu, B., & Hong, Y. (2025). Cultural Heritage Color Regeneration: Interactive Genetic Algorithm Optimization Based on Color Network and Harmony Models. Applied Sciences, 15(4), 1720. https://doi.org/10.3390/app15041720

Krull, A., Buchholz, T.-O., & Jug, F. (2019). Noise2Void - Learning Denoising From Single Noisy Images. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2124–2132. https://doi.org/10.1109/CVPR.2019.00223

Ladiana, D., & Di Sivo, M. (2019). Programmed conservation of historical and architectural heritage. tools for optimising a process based on knowledge and information. International Journal of Design and Nature and Ecodynamics, 14(3), 229–240. https://doi.org/10.2495/DNE-V14-N3-229-240

Lerario, A., & Varasano, A. (2020). An IoT Smart Infrastructure for S. Domenico Church in Matera’s “Sassi’’: A Multiscale Perspective to Built Heritage Conservation. Sustainability, 12(16), 6553. https://doi.org/10.3390/su12166553

Lercari, N. (2019). Monitoring earthen archaeological heritage using multi-temporal terrestrial laser scanning and surface change detection. Journal of Cultural Heritage, 39, 152–165. https://doi.org/10.1016/j.culher.2019.04.005

Letellier, R. (2016). Recording, Documentation and Information Management for the Conservation of Heritage Places. In Recording, Documentation and Information Management for the Conservation of Heritage Places. https://doi.org/10.4324/9781315793917

Li, X., Yang, H., Chen, C., Zhao, G., & Ni, J. (2024). Deterioration identification of stone cultural heritage based on hyperspectral image texture features. Journal of Cultural Heritage, 69, 57–66. https://doi.org/10.1016/j.culher.2024.07.011

Li, Z., Kamnitsas, K., & Glocker, B. (2021). Analyzing Overfitting Under Class Imbalance in Neural Networks for Image Segmentation. IEEE Transactions on Medical Imaging, 40(3), 1065–1077. https://doi.org/10.1109/TMI.2020.3046692

Michele Russo, Eleonora Grilli, Fabio Remondino, Simone Teruggi, & Francesco Fassi. (2021a). Machine Learning for Cultural Heritage Classification. In Representation Challenges. Augmented Reality and Artificial Intelligence in Cultural Heritage and Innovative Design Domain. FrancoAngeli srl. https://doi.org/10.3280/oa-686.33

Michele Russo, Eleonora Grilli, Fabio Remondino, Simone Teruggi, & Francesco Fassi. (2021b). Machine Learning for Cultural Heritage Classification. In Representation Challenges. Augmented Reality and Artificial Intelligence in Cultural Heritage and Innovative Design Domain. FrancoAngeli srl. https://doi.org/10.3280/oa-686.33

Microimaging, M. (2022). Contrast and Feature Enhancement in Fiji. Guide to Brightness/Contrast in Fiji

Mobarak, M. H., Islam, M. A., Hossain, N., Al Mahmud, M. Z., Rayhan, M. T., Nishi, N. J., & Chowdhury, M. A. (2023). Recent advances of additive manufacturing in implant fabrication – A review. Applied Surface Science Advances, 18, 100462. https://doi.org/10.1016/j.apsadv.2023.100462

Murtagh, F., & Legendre, P. (2015). Ward’s Hierarchical Agglomerative Clustering Method: Which Algorithms Implement Ward’s Criterion? Journal of Classification, 31(3), 274–295. https://doi.org/10.1007/s00357-014-9161-z

Onaka, T. (2009). a Study of the Documentation Process for Conservation of Architectural Heritage Sites : Illustrated By Examples From Egyptand Belgium Abstract : 22nd CIPA Symposium, October 11-15, 2009, Kyoto, Japan, 1–6. http://cipa.icomos.org/fileadmin/template/doc/KYOTO/188.pdf

Pang, B., Yang, J., Xia, T., Zhang, A., Zhang, K., Xu, Q., & Wang, F. (2025). Automated heritage building component recognition and modelling based on local features. Journal of Cultural Heritage, 71, 252–264. https://doi.org/10.1016/j.culher.2024.12.006

Patankar, N., Dixit, S., Bhamare, A., Darpel, A., & Raina, R. (2021). Customer Segmentation Using Machine Learning. Advances in Parallel Computing, 39(May 2023), 239–244. https://doi.org/10.3233/APC210200

Pigawati, M. A. dan B. (2017). Pingembangan Kawasan Vihara Budhajaya Wattugong Sebagai Objek Wisata Di Kota Semarang. Jurusan Perencanaan Wilayah Dan Kota, Fakultas Teknik, Universitas Diponegoro, Semarang.

Prasetyo, M. H., Lihawa, F., & Baderan, D. W. K. (2024). Potensi Model Sistem Dinamik dalam Sistem Pengelolaan Sampah Perkotaan.

Razia Sulthana, A., Jaiswal, A., Supraja, P., & Sairamesh, L. (2023). Customer Segmentation using Machine Learning. 2023 3rd International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies, ICAECT 2023, 9(6), 956–965. https://doi.org/10.1109/ICAECT57570.2023.10117924

Revol-Muller, C., Grenier, T., Rose, J.-L., Pacureanu, A., Peyrin, F., & Odet, C. (2014). Region Growing: When Simplicity Meets Theory – Region Growing Revisited in Feature Space and Variational Framework (pp. 426–444). https://doi.org/10.1007/978-3-642-38241-3_29

Rosina, E., & Scazzosi, L. (2019). the Conservation and Enhancement of Built.

Sánchez-Aparicio, L. J., Del Pozo, S., Ramos, L. F., Arce, A., & Fernandes, F. M. (2018). Heritage site preservation with combined radiometric and geometric analysis of TLS data. Automation in Construction, 85(September 2017), 24–39. https://doi.org/10.1016/j.autcon.2017.09.023

Sánchez-Aparicio, L. J., Del Pozo, S., Ramos, L. F., & Fernandes, F. M. (2019). Geometric and Radiometric Analysis of TLS Point Clouds to Diagnose the Conservation State of Historical Constructions. Case of Study in the Master Gate of San Francisco, Almeida, Portugal (pp. 465–473). https://doi.org/10.1007/978-3-319-99441-3_50

Sanchez, T. W., Brenman, M., & Ye, X. (2024). The Ethical Concerns of Artificial Intelligence in Urban Planning. Journal of the American Planning Association, 91(2), 294–307. https://doi.org/10.1080/01944363.2024.2355305

Sardiyarso, E. S., Puspitasari, P., Walaretina, R., & Lakawa, A. R. (2023). The reading of borobudur temple reliefs: Virtue architecture. In AIP Conference Proceedings (Vol. 2706). https://doi.org/10.1063/5.0120648

Sudikno, A., & Surjono, S. (2017). Pelestarian Eks Pusat Kota Kolonial Lama Semarang. Prosiding Seminar Nasional Perencanaan Wilayah Dan Kota, Universitas Brawijaya, March.

Widiatmika, K. P. (2015). VOLUNTARY NATIONAL REVIEW FOR IMPLEMENTATION OF NEW URBAN AGENDA. Etika Jurnalisme Pada Koran Kuning : Sebuah Studi Mengenai Koran Lampu Hijau, 16(2), 39–55.

Y. Boffill, H. Blanco, I. L. &L. V. (2020). Structural Assessment of Ancient Masonries before and during a Rehabilitation Process. International Journal of Architectural Heritage. https://doi.org/10.1080/15583058.2018.1539784

Yang, H., Xu, Z., Shi, Y., Tang, W., Liu, C., Yunusa-Kaltungo, A., & Cui, H. (2023). Multi-objective optimization designs of phase change material-enhanced building using the integration of the Stacking model and NSGA-III algorithm. Journal of Energy Storage, 68, 107807. https://doi.org/10.1016/j.est.2023.107807




DOI: https://doi.org/10.17509/jare.v8i2.87742

Refbacks

  • There are currently no refbacks.


Copyright (c) 2026 Journal of Architectural Research and Education

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

View My Stats

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.