Determination of Mango Fruit Maturity on the Tree Based on Digital Image Processing and Artificial Neural Networks

Until now


INTRODUCTION
Mango is a seasonal fruit that is widely known by the Indonesian community.It originated from India and had many varieties that are popular among Indonesians.Mango trees thrive in lowland areas with a hot climate.However, there are also varieties that can grow in regions with elevations of up to 600 meters above sea level.
Probolinggo, Situbondo, and Pasuruan in East Java are among the largest mango-producing regencies in Java Island.According to the Ministry of Agriculture (2015), mango production in Indonesia increased from 2012 to 2014.In 2012, the production reached 76,547 tons, followed by 98,958 tons in 2013, and 102,820 tons in 2014 (Sanjaya and Rosadi, 2018).
In the case of mango ripeness, sometimes a mango can still taste sour even when it is fully ripe, and vice versa (Hossain et al., 2021).Therefore, there is a need for technology to determine the level of ripeness of mangoes (Kurniawan and Junaidi, 2022).According to the Webster dictionary, an image is a representation, resemblance, or imitation of an object.For example, a photograph represents a person captured by a camera, an X-ray image represents a person's body part, and so on (Shi et al., 2020).There is a lot of information that can be extracted from an image because it provides visual information (Adenugraha et al., 2022).In general, digital image processing is a process that aims to manipulate and analyze images with the help of computers.
Previous research conducted by (Javanmardi et al., 2021) focused on image processing to determine the ripeness of grapes on the vine by combining the Radial Symmetry Transform algorithm and k-Nearest Neighbors, which achieved an accuracy of 90.2%.Therefore, it is highly possible to use digital image processing methods to analyze the ripeness level of fruits other than grapes (Javanmardi et al., 2021).Hence, this research will attempt to apply digital image processing to determine the ripeness level of mangoes that are still on the tree.

RESEARCH METHODS
Figure 1.Key steps of digital image processing

Digital Image Processing
The first step in digital image processing is the image acquisition process, which involves capturing or obtaining the image.This can be done using sensors such as cameras, scanners, and other devices.The next step is the preprocessing stage, where the image undergoes adjustments, such as resizing or enhancing its quality before being used for specific purposes (Zeb et al., 2007).The subsequent step is segmentation, which involves dividing the image into its constituent parts.This process aims to separate the desired objects from the rest of the image.
Further observation and analysis (representation and description) are then performed to verify that the regions within the boundaries represent the intended objects (Zhang and Lu, 2004).This step is necessary because the output of the segmentation process provides the boundaries between the object of interest and other objects in the image (Sutisna et al., 2020).The final step is recognition and interpretation.Recognition involves labeling the objects based on the information provided by their descriptors.Interpretation aims to extract meaningful information and understanding from the labeled objects.These steps collectively contribute to the analysis and interpretation of digital images as shown in Figure 2a-2c (Syarifah et al., 2022).

Color Model and Color Features
In computer applications, colors are formed on the display interface using a specific color model.One widely used example of a color model is RGB (Alcaruban et al., 2018).The RGB color model belongs to the additive color model, where colors are formed by combining the light energy of three primary colors: red, green, and blue (Latuconsina, 2021).Different levels of intensity in each primary color component when combined will result in different colors (Nurraharjo, 2012).
The current computer screens typically have 256 levels of intensity (0-255) for each RGB component, allowing for a wide range of color gradations.In addition to the RGB color model, the HSV color model is also commonly used in digital image processing (Himmah et al., 2020).The HSV color model represents color using three components: hue (H), saturation (S), and value (V) (Salsabila et al., 2021).The HSV color model is a nonlinear model that approximates human visual color perception.
Features are distinctive characteristics that differentiate one object from another.In an image, there are features extracted from the patterns of colors, which can be used to differentiate and compare one image with another (Laia et al., 2023).Several color features can be derived from statistical characteristics of a group of RGB color component values or HSV color component values in an image (Alcahruban et al., 2018).Examples of such features include the mean values of the red, green, blue, hue, saturation, value, and grayscale components.In this study, the features used in the segmentation process are the mean values of R, G, B, and grayscale (Sari and Purnama, 2018).During the classification process (maturation determination), the features used are the mean values of R and V.These feature values are extracted from cropped images containing mango pixels.An example of a cropped image with its corresponding extracted feature values as shown in Figure 3a-3c

Backpropagation Classification
Backpropagation is one of the algorithms used in artificial neural networks, specifically in supervised learning models (Chen and Zhong, 2009).The concept of backpropagation involves a multi-layer network that aims to minimize the error in the network's output (Paola and Schowengerdt, 1995).In the classification process using backpropagation, there are several steps involved.It starts with placing each object's features into the input layer neurons, corresponding to indices 1 and 2 (mean R and mean V).Then, an activation process is performed using a binary sigmoid activation function for each neuron in the hidden layer, followed by calculating the activation values for each neuron in the output layer.The calculated results are then used for thresholding.If the value is greater than or equal to 0.5, it is considered as 1.If the value is less than 0.5, it is considered as 0. Since there are only 2 neurons in the output layer, the possible output combinations [out1, out2]  Figure 4. Flowchart of the process in the maturity determination system using Backpropagation

RESULTS AND DISCUSSION
To manage and develop a computer system for classifying the ripeness level of mangoes, a dataset is required to identify the existing issues.The dataset should consist of captured images of mangoes at different ripeness levels, ranging from unripe to overripe.In this research, a dataset of 52 mangoes with varying ripeness levels was used, including 11 unripe mangoes, 12 medium-ripe mangoes, 11 ripe mangoes, and 10 overripe mangoes.This data was then processed to be learned by the artificial neural network system.
Identifying the ripeness level of mangoes manually is actually easy to implement, as it can be visually observed from the color and through touch by assessing the mango's texture.However, implementing manual identification for a large quantity of mangoes is considered less effective as shown in Figure 5a-5d.In this research, we used the following features: Red Value, standard deviation, skewness, entropy, and kurtosis as inputs.The obtained values are then normalized and grouped into 4 ripeness levels with the codes 1 (unripe), 2 (medium-ripe), 3 (ripe), and 4 (overripe).The primary parameter used to determine the ripeness level of the mango is the Red value (RGB representation).This parameter can be logically classified and facilitates the learning process of the system.
To achieve an accurate system, several experiments need to be conducted with different percentages of training and testing data.This research performed 3 experimental scenarios with different percentages of training and testing data: 80:20, 60:40, and 50:50.These scenarios were based on previous studies (Karo et al., 2022) that yielded the best model with those compositions.Each data processing experiment was tested 10 times.The expected prediction results should be 1 , 1, 2, 2, 2, 3, 3, 3, 4, and 4 in sequential order.The neuron architecture design parameters used were 2-2-1, with a learning rate of 0.1.
The activation function for the hidden layer was sigmoid, and purelin was used for the output layer.Table 1 provides an example of the testing results conducted.

Testing MSE Epoch Regression False error
Testing Result

T T T T T T T T T T
After simulation, in the first experiment with an 8:2 ratio of training data to test data, an accuracy rate of 72% was obtained.In the second experiment, data processing was conducted with a 6:4 ratio of training data to test data.After testing, an accuracy rate of 60.90% was achieved.In the third experiment, data processing was performed with a 5:5 ratio of training data to test data, resulting in an accuracy rate of 53.75%.
From the above testing, the data training and test data process with a 5:5 ratio could not be implemented because the percentage of accuracy in the test results was almost equal to the percentage of incorrect test results.In the data training and test data process with a 6:4 ratio, it could still be implemented as the percentage of accuracy in the test results was higher than the percentage of incorrect test results.The sample data used in the 6:4 ratio consisted of 30 samples, including 6 raw mangoes, 7 ripening mangoes, 11 mature mangoes, and 6 overripe mangoes.
The best level of accuracy was achieved with an 8:2 ratio of test data to training data, with a percentage of 72%.This is because more training data was processed in this experiment compared to the previous ones.

Figure 2a .
Figure 2a.An example image of a fruiting mango tree with raw