ORIGINAL RESEARCH published: 21 May 2020 doi: 10.3389/fgene.2020.00513 Estimating Conformational Traits in Dairy Cattle With DeepAPS: A Two-Step Deep Learning Automated Phenotyping and Segmentation Approach Jessica Nye 1* † , Laura M. Zingaretti 1* † and Miguel Pérez-Enciso 1,2* 1 Centre for Research in Agricultural Genomics (CRAG), CSIC-IRTA-UAB-UB Consortium, Barcelona, Spain, 2 ICREA, Barcelona, Spain Edited by: Assessing conformation features in an accurate and rapid manner remains a challenge Guilherme J. M. Rosa, in the dairy industry. While recent developments in computer vision has greatly improved University of Wisconsin-Madison, automated background removal, these methods have not been fully translated to United States biological studies. Here, we present a composite method (DeepAPS) that combines Reviewed by: Shogo Tsuruta, two readily available algorithms in order to create a precise mask for an animal University of Georgia, United States image. This method performs accurately when compared with manual classification Christian Maltecca, North Carolina State University, of proportion of coat color with an adjusted R2 = 0.926. Using the output mask, United States we are able to automatically extract useful phenotypic information for 14 additional *Correspondence: morphological features. Using pedigree and image information from a web catalog Jessica Nye (www.semex.com), we estimated high heritabilities (ranging from h2 = 0.18–0.82), Jessica.Nye@gmail.com Laura M. Zingaretti indicating that meaningful biological information has been extracted automatically from m.lau.zingaretti@gmail.com imaging data. This method can be applied to other datasets and requires only a Miguel Pérez-Enciso miguel.perez@uab.es minimal number of image annotations (∼50) to train this partially supervised machine- † These authors have contributed learning approach. DeepAPS allows for the rapid and accurate quantification of multiple equally to this work phenotypic measurements while minimizing study cost. The pipeline is available at https://github.com/lauzingaretti/deepaps. Specialty section: This article was submitted to Keywords: image analysis, morphology, phenomics, image mask, deep learning, dairy cattle Livestock Genomics, a section of the journal Frontiers in Genetics INTRODUCTION Received: 05 December 2019 Accepted: 27 April 2020 Breeding programs depend on large-scale, accurate phenotyping, which is also critical for genomic Published: 21 May 2020 dissection of complex traits. While the genome of an organism can be characterized, e.g., with Citation: high density genotyping arrays, the “phenome” is much more complex and can never be fully Nye J, Zingaretti LM and described, as it varies over time and changes with the environment (Houle et al., 2010). The Pérez-Enciso M (2020) Estimating cost of genotyping continues to drop, but there is still a need for improvements in obtaining Conformational Traits in Dairy Cattle high-performance phenotypes at a lower cost (Tardieu et al., 2017). In cattle, the number of With DeepAPS: A Two-Step Deep Learning Automated Phenotyping phenotypes recorded in traditional breeding schemes is relatively small, because its recording is and Segmentation Approach. expensive. For instance, yearly milk yield is usually inferred by extrapolation using a few lactation Front. Genet. 11:513. measurements, whereas actual milk production can now be measured individually and daily using doi: 10.3389/fgene.2020.00513 automated milking robots. Frontiers in Genetics | www.frontiersin.org 1 May 2020 | Volume 11 | Article 513 Nye et al. DeepAPS: An Authomatic Phenotyping Approach In addition to milk yield, dairy cattle breeders are interested Here, we show how automatically parsed web-based in conformational traits. These metrics are not only relevant catalog datasets can be converted into useful information aesthetically but can also have an important influence on by automatically inferring genetic parameters of several an animal’s breeding value. Body conformation is associated morphological measurements in dairy cattle. We combined web with dairy performance (Guliński et al., 2005) and longevity, scraping, deep learning, and statistical techniques in order to which strongly contributes to lifetime milk production (Sawa achieve our objective. The proposed methodology is a mixture et al., 2013). Milk production is positively correlated with between a supervised deep learning approach, Mask R-CNN udder size (Mingoas et al., 2017). The highest negative (He et al., 2017) and an unsupervised algorithm (Kanezaki, economic impact for dairy farmers is caused by lameness 2018) which can achieve highly precise automatic image either due to leg malformations or injury (Sogstad et al., 2006; segmentation. After removing the background, phenotypic Green et al., 2010). Extracting the detailed conformational information, including coat color and body conformational traits phenotypes which may impact progeny success are likewise can be easily quantified. Lastly, we demonstrate the potential time consuming and costly to collect, and in the absence applications of this method in other datasets. We assert that our of quantitative tools, farmers often evaluate morphometric work could constitute a good proxy for using inexpensive and measurements qualitatively. non-invasive computer vision techniques into the dairy cattle The emergence of modern sensor technologies, such as breeding programs. Unmanned Aerial Vehicles (UAV) combined with simple digital cameras (Kefauver et al., 2017), mass spectroscopy, robotics, and hyper-spectral images (Fahlgren et al., 2015), among others, have MATERIALS AND METHODS revolutionized breeding programs, mainly in plants, allowing for non-invasive evaluation of multiple complex traits. Although Image Collection in animal breeding their application is more scarce, modern Images of bulls were collected through web-scraping using livestock farming is beginning to benefit from access to these the python library Beautiful Soup (Richardson, 2007). Images inexpensive sensor tools. Now, it is possible to remotely monitor from sire catalogs of six Artificial Insemination companies behavior (Guzhva et al., 2016; Foris et al., 2019; Zehner et al., were collected. We additionally automatically collected bull 2019) and animal welfare (Beer et al., 2016), assess movement images from one semen provider1 and those of identified (Chapinal et al., 2011), measure body confirmation (Van Hertem familial relationships (daughters, dams, granddams, and great et al., 2013; Song et al., 2018), quantify individual food intake granddams) where possible. We downloaded a total of 1,819 (Braun et al., 2014; Beer et al., 2016; Foris et al., 2019), maintain images. These images ranged in size between 339–879 pixels and an optimum environment (Chen and Chen, 2019), or decrease 257–672 pixels for width and height, respectively. The animals are instances of stillbirths (Palombi et al., 2013; Ouellet et al., 2016). Holstein with patched black and white bodies, but some images These automated measurements rely on temperature (Palombi are red Holstein. Individuals ranged in color from all white, all et al., 2013; Ouellet et al., 2016; Chen and Chen, 2019), pressure black, all brown, to a mixture of the colors. The images were (Braun et al., 2014; Beer et al., 2016), movement (Chapinal et al., flipped so that all animals faced the right side of the image using 2011), and visual (Van Hertem et al., 2013; Guzhva et al., 2016; ImageMagick version 7.0.9-0 convert -flop function. The animals Song et al., 2018; Foris et al., 2019; Zehner et al., 2019) sensors. are standing in front of dynamic backgrounds including forest, As several remote monitoring schemes are based on digital field, snow, water, and straw. All images contained only one images or video, automated image analysis techniques are animal, and sometimes contained a person or an arm. urgently needed to quantify traits of interest (Zhang et al., 2018). Applying image analysis to breeding programs is not Automated Segmentation new, however many of these methods largely depend on One of the most challenging tasks in computer vision is instance time consuming image-by-image processing facilitated by the segmentation, i.e., the identification of boundaries of objects at researcher (as in Hayes et al., 2010; Cortes et al., 2017; the pixel level (Kanezaki, 2018), whereas object classification, i.e., Rosero et al., 2019). The few automated resources currently to determine if an object belongs to certain class is relatively implemented for cattle analyses require complicated set-ups and simpler. R-CNN (Girshick et al., 2014), a deep learning approach, costly equipment (Chapinal et al., 2011; Song et al., 2018). This is as well as Fast R-CNN (Girshick, 2015), Faster R-CNN (Ren not surprising as accurately quantifying phenotypic information et al., 2015), or Mask R-CNN (He et al., 2017) are widely used is one of the most challenging aspects in biology (Houle et al., to solve this task. Although these methods are efficient, they 2011; Boggess et al., 2013; Rahaman et al., 2015). are not accurate enough for some purposes since the obtained The availability of new algorithms based on machine learning segmentation often removes parts of the object of interest or has revolutionized computer vision, impacting a wide range contains parts of the background. of fields that rely on computers to analyze images, with the We applied a two-step procedure to automatically segment the potential to optimize herd care and improve animal and plant animal’s profile as accurately as possible. The composite method breeding program outcomes (Song et al., 2018; Foris et al., 2019; begins by using Mask R-CNN (He et al., 2017), which has three Zehner et al., 2019). These recent advances have led to precise outputs for each candidate object in an input image (Figure 1A): object detection and semantic segmentation in complex images (Girshick et al., 2014; Han et al., 2018; Gu et al., 2018). 1 www.semex.com Frontiers in Genetics | www.frontiersin.org 2 May 2020 | Volume 11 | Article 513 Nye et al. DeepAPS: An Authomatic Phenotyping Approach a class label (say “cow”), bounding box offset or region of interest (RoI), and the object mask consisting of an approximate layout of a spatial object. As in the original Mask R-CNN, we used the annotated image database common objects in context (COCO; Lin et al., 2014)2 to train the algorithm, and select the class codes for cow. In short, Mask R-CNN is a deep learning algorithm that consists of two steps: first, it proposes regions within the image that may contain objects of interest and, second, generates a mask for every detected object. The latter step consists of a binary classification of pixels, either a pixel belongs to the object or to the background. For more details about this method readers can consult, e.g., https://towardsdatascience.com/computer- vision-instance-segmentation-with-mask-r-cnn-7983502fcad1, https://engineering.matterport.com/splash-of-color-instance- segmentation-with-mask-r-cnn-and-tensorflow-7c761e238b46 or should refer to He et al. (2017). Figure 1B shows the applied mask predicted by Mask R-CNN, this mask removes the majority of the background, but also removes parts of the cow’s body making it necessary for the development of our two-step composite method. We used the implementation of Mask R-CNN in https://github.com/matterport/Mask_RCNN. After the RoI and class labels are extracted, we select only the RoI for our desired object (i.e., the bull or cow). This allows us to remove some of the background and obtain a smaller, less noisy image. As explained above, the Mask R-CNN segmentation was not accurate enough for our purposes (Figure 1B). Therefore, we passed the RoI and predicted mask to a modified version of the unsupervised image segmentation algorithm from Kanezaki (2018). We used the code available at https://github. com/kanezaki/pytorch-unsupervised-segmentation. The original algorithm relies on separating pixels from each other and grouping them into distinct clusters based on color and texture. FIGURE 1 | Example input and outputs. (A) Original input image. (B) Mask The underlying assumptions of this model are that: (1) pixels R-CNN applied mask. (C) DeepAPS raw output. (D) Final output of DeepAPS of similar features should be clustered together, (2) spatially after all applied filters. (E) Final DeepAPS mask applied to input image. (F) Outline extraction of original input image. (G) Extracted landmark continuous pixels should be clustered together, and (3) the coordinates. (H) Manual color segmentation. Image from Semex. number of clusters should be large. This is achieved by applying a linear classifier which groups pixels into different clusters based on their features. The difference between the original algorithm and ours is we do not try to maximize the total number of (Figure 1C). However, the unsupervised clustering still can clusters, but instead we merely improve upon the mask generated confound the foreground and the background. We then apply an by Mask R-CNN based on pixel identity. This makes more additional filter to the mask, median blur function from OpenCV effective the algorithm to run, since the algorithm applied to (Bradski, 2000), removing small islands that have been mislabeled the whole original image was not completely satisfactory. This during the clustering step (Figure 1D). We lastly apply the mask proceeds by self-training the network through back propagation, by coloring all pixels predicted to be in the background by a solid by alternating between two stages: (1) forward super pixel color (Figure 1E). refinement, and (2) backward gradient descent. Much like any To extract the proportion of and average color(s) from supervised approach this is achieved by calculating the cross- each cluster, we apply k-means using the scikit-learn library entropy loss between network and cluster labels, then back (Pedregosa et al., 2011). To measure anatomical features, propagating the error rates used to update the convolutional filter we extract only the outline of the desired object from parameters. Backpropagation is a popular and clever method the mask (Figure 1F) using the edge detection algorithm used in deep learning. It allows computing the gradient of the loss developed by Canny (1986) and implemented in OpenCV function very efficiently by using the chain rule for derivatives, (Bradski, 2000). After extracting the edge, we apply one which greatly simplifies optimization in complex models. more filter to remove any islands that may remain using the After refinement through the unsupervised algorithm, we remove_small_objects function from the morphology package obtained a relatively precise mask for our input image available from scikit-learn (Pedregosa et al., 2011). Now that the input image has been reduced down to just the object 2 http://cocodataset.org outline, we can take advantage of common conformational Frontiers in Genetics | www.frontiersin.org 3 May 2020 | Volume 11 | Article 513 Nye et al. DeepAPS: An Authomatic Phenotyping Approach FIGURE 2 | Example input and outputs. (A) Original input image. (B) Mask R-CNN applied mask. (C) DeepAPS raw output. (D) Final output of DeepAPS after all applied filters. features of the underlying data, and extract pixel coordinates. image processing software GIMP3 , and the background For example, we extracted the coordinate of the pixel closest was manually changed from the colored cluster to to the bottom left corner which corresponds to the back white (Figure 1H). To extract the color clusters, we foot of the cow. We proceeded in this way to extract 13 calculated the proportion of color clusters in each image coordinates from each animal (Figure 1G). We then calculate the by using k-means as above, and manually matched each distance in pixels between various points, effectively extracting color cluster to the original picture and removed the body confirmations automatically. The 14 conformational proportion of background. traits are described in Supplementary Figure S1. Code for the whole pipeline is available at https://github.com/ Genetic Parameters lauzingaretti/deepaps. To calculate heritabilities for the measured phenotypes, we extracted pedigree information and constructed a relationship Manual Segmentation matrix for each bull whenever possible. This was done by To check how accurate the automated segmentation was, automatic web scraping in the sire catalog website, where we we manually segmented N = 481 images that were not identified bull id, any relative type (i.e., daughter, dam, granddam, of Semex origin. We used Kanezaki’s demo.py program and great granddam), and their images. From the list of bull and (2018) in python3.6 (van Rossum, 1995) using default parameters. The output images were opened in the 3 https://www.gimp.org/ Frontiers in Genetics | www.frontiersin.org 4 May 2020 | Volume 11 | Article 513 Nye et al. DeepAPS: An Authomatic Phenotyping Approach FIGURE 3 | (A) Correlation (adjusted R2 = 0.926) between manual and automated color segmentation of 481 images. (B) Example input image. (C) Applied DeepAPS output mask. (D) Manual color segmentation. Image from Semex. relatives’ ids, we computed the standard numerator relationship R-CNN using the starting weights of the COCO database matrix, which contains the genetic relationships assuming an (Lin et al., 2014). The model was trained for 20 epochs and infinitesimal model. Bayesian estimates of heritability were default parameters. Using either the COCO or custom model, calculated with the R 3.5.2. (R Core Team, 2013) package BGLR DeepAPS was applied and the composite mask was visually (Perez and de los Campos, 2014) using default priors. One assessed for accuracy. thousand Gibbs iterations were performed. Our sample sizes were N = 1,338 for proportion of white and N = 1,062 for morphological characteristics. The difference in sample size is RESULTS due to removing any image with a missing coordinate. We first visually compared the masks generated by the three Application to Other Datasets methods that were applied to our entire dataset of 1,819 To assess the applicability to other datasets, we chose images (Figure 2A). When we used the supervised algorithm two other objects that had been annotated in the COCO Mask R-CNN and applied the mask to the input images database (Lin et al., 2014), horse and giraffe, as well as two (Figure 2B), we observed in all cases parts of the cow body objects that had not been annotated, butterfly and duck. were removed along with the background (i.e., tail, nose, We downloaded 50 images from the internet that had the ear, and hoof). These masks are not satisfactorily precise license set to “labeled for non-commercial reuse” for horse to extract morphological measurements. The unsupervised and giraffe and 100 images for butterfly and duck. For the segmentation by back propagation (Figure 2C) often unannotated objects we annotated 50 of the images using separates the precise border between cow and background, VGG Image Annotator (VIA; Dutta and Zissermann, 2019). but that this method on its own is not automated. Each These annotations were used to train a model in Mask output image would still need to be processed separately Frontiers in Genetics | www.frontiersin.org 5 May 2020 | Volume 11 | Article 513 Nye et al. DeepAPS: An Authomatic Phenotyping Approach FIGURE 4 | (A) posterior h2 distribution of the white coat color, (B) posterior h2 distribution of the gait, (C) posterior h2 distribution of the chest depth, (D) posterior h2 distribution of the back height, (E) posterior h2 distribution of the back deviation, (F) posterior h2 distribution of the front height, (G) posterior h2 distribution of the back leg height, (H) posterior h2 distribution of the front leg height, (I) posterior h2 distribution of the cow length, (J) posterior h2 distribution of the face length, (K) posterior h2 distribution of the head length, (L) posterior h2 distribution of the head width, (M) posterior h2 distribution of FIGURE 5 | Application of DeepAPS method to four additional datasets. the neck width, (N) posterior h2 distribution of the triangle body area, (O) (A) Horses and (B) Giraffes trained using the COCO database. (C) Butterflies posterior h2 distribution of the polygon body area. and (D) Ducks trained using 50 custom annotations. in order to match which body parts were grouped into using 1,338 images of related animals, in which we had partial each color cluster. DeepAPS (Figure 2D) across our input information about great granddam, granddam, dam, bull, and dataset produces a more accurate mask than Mask R-CNN daughter relationships. Our relationship matrix consists of and a fully automated mask, which the unsupervised 689 families, with an average of 2.6 individuals per family. approach fails to do. Figure 4 shows the 15 posterior distributions of the heritability In order to assess how accurately we were able to extract the calculations and lists average values. Coat color proportion has true coat color percentage from each image, we compared manual the highest calculated heritability h2 = 0.82, followed by body area and automated color segmentation. Our test set consists of 481 (triangle) h2 = 0.43, body area (polygon) h2 = 0.38, and cow body manually annotated images. After removing the background, length h2 = 0.34. These values are similar to previously published we clustered each bull into one- or two-color components and results (Hayes et al., 2010; Pritchard et al., 2013). These high extracted the percentage of dark and light colors in the coats. The heritability measurements indicate foremost that the meaningful automated method reports a highly accurate color segmentation genetic information can be quickly and easily extracted from with an adjusted R2 = 0.926 (Figure 3A) when compared to imaging and pedigree data available online. manual segmentation (Figures 3B–D). The images that fall out as To assess whether this method is robust to the type and outliers belong to one of two groups, the majority of the outliers quality of the underlying data, we downloaded images from have small image sizes (less than 400 × 400 pixels), and therefore the internet of horse, giraffe, butterfly, and duck. These images the quality was not sufficient to accurately separate the body into were randomly collected, and we had no control over quality, two color classes, the second group were bulls with a two-toned size, lighting, or background. We also wanted to test how many body color, in which the legs were of a different color than the input annotations are required to produce a robust mask using body. In these cases, the algorithm has difficulty in separating the DeepAPS. Because the two-step method uses back propagation dark-colored legs from the dark background. in order to refine the predicted mask generated from the machine Because the mask recovered after using this composite learning algorithm, we hypothesized that fewer annotations method is so precise, we could extract coordinates of 13 points would be needed. Therefore, we annotated 50 images for the located around the outline of the cow body (Figure 1G and butterfly and duck datasets, as they were not pre-annotated in the Supplementary Figure S1) which allowed for measurements of COCO database. We found that overall, our composite method 14 body conformation distances (see Supplementary Figure S2 preforms accurately (Figure 5). The masks generated from the for phenotypic distributions). Next, we estimated heritability thousands of annotations from the COCO dataset were precise Frontiers in Genetics | www.frontiersin.org 6 May 2020 | Volume 11 | Article 513 Nye et al. DeepAPS: An Authomatic Phenotyping Approach (Figures 5A,B), while those based on only 50 annotations were underlying information. Hayes et al., 2010) estimated heritability still far more accurate than using any currently available method of coat color percentage by manual quantification and reported a (Figures 5C,D). These results together indicate this method is heritability of h2 = 0.74 in N = 327 bulls; remarkably, we found robust to input data and can still preform reliably despite being similar estimates (h2 = 0.81), even if our pedigree information trained by few instances, making it a promising tool for automatic was quite incomplete. The reported heritability of back leg height morphological analyses. is nearly identical to previous reports (h2 = 0.22 vs. 0.21; Pritchard et al., 2013). Nevertheless, estimates of two other reported conformational heritabilities were somewhat lower: chest depth DISCUSSION h2 = 0.28 vs. 0.37 and height h2 = 0.27 vs. 0.42 (Pritchard et al., 2013); perhaps because actual metrics analyzed here are not In recent decades, there have been vast improvements in exactly those used in previous studies and because we cannot molecular and statistical methods applied to animal and plant obtain absolute values (e.g., height in meters), since there is not a breeding. While modern livestock studies typically involve the common scale across images. In all, this proof of concept shows analysis of entire genomes and/or vast number of polymorphic how genetic parameters could be estimated using solely data sites (Börner et al., 2012; Wiggans et al., 2017; Yin and König, that are already available on the web. For practical applications, 2019), high throughput phenotyping is lagging, especially in more accurate estimates suitable for breeding programs could be animal breeding. Often, phenotypic variation is explored today obtained, e.g., combining SNP genotyping data with automatic in the same manner as it was done decades ago, using simple image analyses from larger datasets. quantifications such as length, number, categorical classifications, While imaging data is fast and simple to collect as well as etc (Houle et al., 2010, 2011; Cole et al., 2011). Phenomics is inexpensive to store, the most burdensome stage of image analysis extremely important in breeding programs in particular, as the is the generation of image annotations. We found that this desired outcome is a change in a phenotype. As phenotypes method is able to leverage the publicly available COCO database are formed by a complex process involving multiple genes, is and apply it to new and different problem sets. Allowing for the dependent on the environment, and dynamic overtime, collecting creation of an accurate object mask based only on a training set of multiple descriptive statistics can make relating genotype to 50 instances (Figure 5), which is remarkably low for any machine phenotype more feasible and, importantly, more meaningful. learning approach. Images are among the easiest to collect data and are This method has the potential to allow for imaging data to underutilized. Here we combine two of the state-of-the-art be easily and quickly applied to high-throughput studies, which image analysis tools, the supervised Mask R-CNN (He et al., can be highly useful and improve extant breeding programs. We 2017) and unsupervised segmentation (Kanezaki, 2018) in order provide a combined deep learning algorithm that results in highly to automatically extract phenotypic measurements accurately. accurate segmentation of animal profiles, which is necessary Not only can we create a precision mask but can cluster and for further processing in applications related to conformational segment the underlying colors and automatically measure body measurements. Nevertheless, we are well aware that much confirmation. Accurate image segmentation remains the most work remains to be done in the area. For instance, software challenging part of computer vision. The ability of DeepAPS to to accurately quantify a number of additional conformational separate the animal from multiple background types at the pixel features, such as udder metrics or movement, using different level out preforms, for our purposes, the available algorithms angle pictures or videos should be developed. Software should currently published (Kanezaki, 2018; He et al., 2017). also be optimized for speed and be able to analyze high- The validity and speed of this method allows for multiple resolution pictures. quantitative morphological traits to be implemented in breeding programs. Despite the success of ongoing dairy breeding programs (Wiggans et al., 2017), including more and accurately DATA AVAILABILITY STATEMENT quantified measurements has the potential to result in further improvements (Goddard, 2009; Gonzalez-Recio et al., 2014). The datasets generated for this study are available on request to Furthermore, this method uses standard side-view stud images the corresponding author. which are inexpensive to generate and store. Our presented method eliminates the high cost of phenotype collection while maintaining quality and can contribute to lowering the cost of AUTHOR CONTRIBUTIONS conformational measurement collections. MP-E conceived and supervised the research. JN and LZ Our analyses were performed on images scrubbed from performed the research and wrote the code. JN drafted the the internet. As such, we had no control over backgrounds, manuscript with contributions from the other authors. lighting, image size, or quality. Despite the dynamic input data on which we tested DeepAPS, we were able to produce high quality masks and phenotypic measurements in most cases FUNDING (Figure 4). Furthermore, the heritability rates we calculated from over 1,000 images of related individuals broadly agree with LZ was supported by a Ph.D. grant from the Ministry of published results, indicating that our method accurately captures Economy and Science (MINECO, Spain), by the MINECO Frontiers in Genetics | www.frontiersin.org 7 May 2020 | Volume 11 | Article 513 Nye et al. DeepAPS: An Authomatic Phenotyping Approach grant AGL2016-78709-R to MP-E and from the EU through the SUPPLEMENTARY MATERIAL BFU2016-77236-P (MINECO/AEI/FEDER, EU) and the “Centro de Excelencia Severo Ochoa 2016-2019” award SEV-2015-0533. The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fgene. 2020.00513/full#supplementary-material ACKNOWLEDGMENTS FIGURE S1 | Description of the 14 extracted conformational traits. We thank comments and suggestions from José Antonio Jiménez FIGURE S2 | Phenotypic distributions for the 14 measured anatomical features (CONAFE, Valdemoro, Spain) and Michael Louis from Semex. (N = 1,062). REFERENCES Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., et al. (2018). Recent advances in convolutional neural networks. Pat. Recognit. 77, 354–377. Beer, G., Alsaaod, M., Starke, M., Schuepbach-Regula, G., Müller, H., Kohler, P., Guliński, P., Młynek, K., Litwiñczuk, Z., and Dobrogowska, E. (2005). Heritabilities et al. (2016). Use of extended characteristics of locomotion and feeding behavior of genetic and phenotypic correlations between condition score and production for automated identification of lame dairy cows. PLoS One 11:e0155796. doi: and conformation traits in Black-and-White cows. Anim. Sci. Pap. Rep. 23, 10.1371/journal.pone.0155796 33–41. Boggess, M. V., Lippolis, J. D., Hurkman, W. J., Fagerquist, C. K., Briggs, S. P., Guzhva, O., Ardö, H., Herlin, A., Nilsson, M., Åström, K., and Bergsten, C. Gomes, A. V., et al. (2013). The need for agriculture phenotyping: “Moving (2016). Feasibility study for the implementation of an automatic system for the from genotype to phenotype”. J. Proteo. 93, 20–39. doi: 10.1016/j.jprot.2013. detection of social interactions in the waiting area of automatic milking stations 03.021 by using a video surveillance system. Comput. Elect. Agric. 127, 506–509. Börner, V., Teuscher, F., and Reinsch, N. (2012). Optimum multistage genomic Han, J., Zhang, D., Cheng, G., Liu, N., and Xu, D. (2018). Advanced Deep-Learning selection in dairy cattle. J. Dairy Sci. 95, 2097–2107. doi: 10.3168/jds.2011-4381 techniques for salient and category-specific object detection: a survey. IEEE Bradski, G. (2000). The OpenCV library. Dr. Dobbs J. 120, 122–125. Signal Process. 35, 84–100. Braun, U., Tschoner, T., and Hässig, M. (2014). Evaluation of eating and Hayes, B. J., Pryce, J., Chamerlain, A. J., Bowman, P. J., and Goddard, M. E. (2010). rumination behavior using a noseband pressure sensor in cows during the Genetic architecture of complex traits and accuracy of genomic prediction: coat peripartum period. BMC Vet. Res. 10:195. doi: 10.1186/s12917-014-0195-6 colour, milk-fat percentage, and type in Holstein Cattle as contrasting model Canny, J. (1986). A computational approach to edge detection. IEEE T. Pattern traits. PLoS Genet. 6:e1001139. doi: 10.1371/journal.pgen.1001139 Anal. 8, 679–698. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017). Mask R-CNN. IEEE Int. Chapinal, N., de Passillé, A. M., Pastell, M., Hänninen, L., Munksgaard, L., and Conf. Comput. Vis. 2017, 2980–2988. Rushen, J. (2011). Measurement of acceleration while walking as an automated Houle, D., Govindaraju, D. R., and Omholt, S. (2010). Phenomics: the next method for gait assessment in dairy cattle. J. Dairy Sci. 94, 2895–2901. doi: challenge. Nat. Rev. Genet. 11, 855–866. 10.3168/jds.2010-3882 Houle, D., Pélabon, C., and Wagner, G. P. (2011). Measurement and meaning in Chen, C. S., and Chen, W. C. (2019). Research and development of automatic biology. Q. Rev. Biol. 86, 3–34. monitoring system for livestock farms. Appl. Sci. 9:1132. Kanezaki, A. (2018). Unsupervised image segmentation by backpropagation. IEEE Cole, J. B., Wiggans, G. R., Ma, L., Sonstegard, T. S., Lawlor, T. J. Jr., Crooker, Int. Conf. Comput. Vis. 1543–1547. B. A., et al. (2011). Genome-wide association analysis of thirty-one production, Kefauver, S. C., Vincente, R., Vergara-Díaz, O., Fernandez-Gallego, J. A., Kerfal, S., health, reproduction and body conformation traits in contemporary U.S. Lopez, A., et al. (2017). Comparative UAV and field phenotyping to assess yield Holstein cows. BMC Genomics 12:408. doi: 10.1186/1471-2164-12-408 and nitrogen use efficiency in hybrid and conventional barley. Front. Plant Sci. Cortes, D. F. M., Catarina, R. S., Barros, G. B. D. A., Arêdes, F. A. S., da Silveira, 8:1733. doi: 10.3389/fpls.2017.01733 S. F., Ferreguetti, G. A., et al. (2017). Model-assisted phenotyping by digital Lin, T. Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., et al. (2014). images in papaya breeding programs. Sci. Agric. 74, 294–302. “Microsoft COCO: common objects in context,” Computer Vision – ECCV Dutta, A., and Zissermann, A. (2019). The VIA Annotation Software for Images, 2014, eds D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars (Cham: Springer), Audio, and Video. New York, NY: ACM. 740–755. Fahlgren, N., Gehan, M. A., and Baxter, I. (2015). Lights, camera, action: high- Mingoas, K. J. P., Awah-Ndukum, J., Dakyang, H., and Zoli, P. A. (2017). Effects throughput plant phenotyping is ready for a close-up. Curr. Opin. Plant Biol. of body conformation and udder morphology on milk yield of zebu cows in 24, 93–99. doi: 10.1016/j.pbi.2015.02.006 North region of Cameroon. Vet. World 10, 901–905. doi: 10.14202/vetworld. Foris, B., Thompson, A. J., von Keyserlingk, M. A. G., Melzer, N., and Weary, 2017.901-905 D. M. (2019). Automatic detection of feeding- and drinking-related agonistic Ouellet, V., Vasseur, E., Heuwieser, W., Burfeind, O., Maldague, X., and behavior and dominance in dairy cows. J. Dairy Sci. 102, 9176–9186. doi: Charbonneau, É (2016). Evaluation of calving indicators measured 10.3168/jds.2019-16697 by automated monitoring devices to predict the onset of calving in Girshick, R. G. (2015). Fast R-CNN. IEEE Comput. Soc. Conf. Comput. Vis. 2015, Holstein dairy cows. J. Dairy Sci. 99, 1539–1548. doi: 10.3168/jds.2015- 1440–1448. 10057 Girshick, R. G., Donahue, J., Darrell, T., and Malik, J. (2014). “Rich feature Palombi, C., Paolucci, M., Stradaioli, G., Corubolo, M., Pascolo, P. B., and Monaci, hierarchies for accurate object detection and semantic segmentation,” in M. (2013). Evaluation of remote monitoring of parturition in dairy cattle as Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern a new tool for calving management. BMC Vet. Res. 9:191. doi: 10.1186/1746- Recognition, Columbus, OH, 580–587. 6148-9-191 Goddard, M. (2009). Genomic selection: prediction of accuracy and maximization Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., of long-term response. Genetica 136, 245–257. doi: 10.1007/s10709-008-9308-0 et al. (2011). Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, Gonzalez-Recio, O., Coffey, M. P., and Pryce, J. E. (2014). On the value of the 2825–2830. phenotypes in the genomic era. J. Dairy Sci. 97, 7905–7915. doi: 10.3168/jds. Perez, P., and de los Campos, G. (2014). Genome-wide regression and prediction 2019-102-6-5764 with the BGLR statistical package. Genetics 198, 483–495. doi: 10.1534/genetics. Green, L. E., Borkert, J., Monti, G., and Tadich, N. (2010). Assocications between 114.164442 lesion-specific lameness and the milk yield of 1,635 dairy cows from seven herds Pritchard, T., Coffey, M., Mrode, R., and Wall, E. (2013). Genetic parameters for in the Xth region of Chile and implications for management of lame dairy cows production, health, fertility and longevity traits in dairy cows. Animal 7, 34–46. worldwide. Anim. Welfare 19, 419–427. doi: 10.1017/S1751731112001401 Frontiers in Genetics | www.frontiersin.org 8 May 2020 | Volume 11 | Article 513 Nye et al. DeepAPS: An Authomatic Phenotyping Approach R Core Team (2013). R: A Language and Environment for Statistical Computing. contour extraction from natural barn background in side-view images. Comput. Vienna: R Foundation for Statistical Computing. Electron. Agr. 91, 65–74. Rahaman, M. M., Chen, D., Gillani, Z., Klukas, C., and Chen, M. (2015). Advanced van Rossum, G. (1995). Centrum voor Wiskunde en Informatica. Python tutorial, phenotyping and phenotype data analysis for the study of plant growth and technical report CS-R9526. Amsterdam: University of Amsterdam. development. Front. Plant Sci. 6:619. doi: 10.3389/fpls.2015.00619 Wiggans, G. R., Cole, J. B., Hubbard, S. M., and Sonstegard, T. S. (2017). Genomic Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster R-CNN: towards real- selection in dairy cattle: the USDA experience. Annu. Rev. Anim. Biosci. 5, time object detection with region proposal networks. NeuralPS 39, 91–99. doi: 309–327. doi: 10.1146/annurev-animal-021815-111422 10.1109/TPAMI.2016.2577031 Yin, T., and König, S. (2019). Genome-wide associations and detection of potential Richardson, L. (2007). Beautiful Soup Documentation. Available online at: http: candidate genes for direct genetic and maternal genetic effects influencing dairy //citebay.com/how-to-cite/beautiful-soup/ cattle body weight at different ages. Genet. Select. Evol. 51:4. doi: 10.1186/ Rosero, A., Granada, L., Pérez, J. L., Rosero, D., Burgos-Paz, W., Martínez, R., s12711-018-0444-4 et al. (2019). Morphometric and colurimetric tools to dissect morphological Zehner, N., Niederhauser, J. J., Schick, M., and Umstatter, C. (2019). Development diversity: an application in sweet potato. Genet. Resour. Crop Evol. 66, and validation of a predictive model for calving time based on sensor 1257–1278. measurements of ingestive behavior in dairy cows. Comput. Electron. Agric. 161, Sawa, A., Bogucki, M., Krêñel-Czopek, S., and Neja, W. (2013). Relationship 62–71. between conformational traits and lifetime production efficiency of cows. ISRN Zhang, S., Huang, W., and Zhang, C. (2018). Three-channel convolutional neural Vet. Sci. 2013:124690. networks for vegetable leaf disease recognition. Cogn. Syst. Res. 53, 31–41. Sogstad, ÅM., Østerås, O., and Fjeldaas, T. (2006). Bovine claw and limb disorders related to reproductive performance and production diseases. J. Dairy Sci. 89, Conflict of Interest: The authors declare that the research was conducted in the 2519–2528. doi: 10.3168/jds.S0022-0302(06)72327-X absence of any commercial or financial relationships that could be construed as a Song, X., Bokkers, E. A. M., van der Tol, P. P. J., Groot Koerkamp, P. W. G., and potential conflict of interest. van Mourik, S. (2018). Automated body weight prediction of dairy cows using 3-dimensional vision. J. Dairy Sci. 101, 4448–4459. doi: 10.3168/jds.2017-13094 Copyright © 2020 Nye, Zingaretti and Pérez-Enciso. This is an open-access article Tardieu, F., Cabrera-Bosquet, L., Pridmore, T., and Bennett, M. (2017). Plant distributed under the terms of the Creative Commons Attribution License (CC BY). phenomics, from sensors to knowledge. Curr. Biol. 27, R770–R783. doi: 10. The use, distribution or reproduction in other forums is permitted, provided the 1016/j.cub.2017.05.055 original author(s) and the copyright owner(s) are credited and that the original Van Hertem, T., Alchanatis, V., Antler, A., Maltz, E., Halachmi, I., Schlageter- publication in this journal is cited, in accordance with accepted academic practice. No Tello, A., et al. (2013). Comparison of segmentation algorithms for cow use, distribution or reproduction is permitted which does not comply with these terms. Frontiers in Genetics | www.frontiersin.org 9 May 2020 | Volume 11 | Article 513