Journal of Vision (2022) 22(6):8, 1–45 1 Contrast sensitivity functions in autoencoders Image Processing Lab, Parc Cientific, Universitat de Qiang Li Valéncia, Spain Computer Vision Center, Universitat Autónoma de Alex Gomez-Villa Barcelona, Spain Instituto de Óptica, Spanish National Research Council Marcelo Bertalmío (CSIC), Spain Image Processing Lab, Parc Cientific, Universitat de Jesús Malo Valéncia, Spain Three decades ago, Atick et al. suggested that human frequency sensitivity may emerge from the Introduction enhancement required for a more efficient analysis of retinal images. Here we reassess the relevance of The human contrast sensitivity function (CSF) low-level vision tasks in the explanation of the contrast characterizes the psychophysical response to visual sensitivity functions (CSFs) in light of 1) the current gratings of different frequency (Campbell & Robson, trend of using artificial neural networks for studying 1968). Filter characterizations in the Fourier domain vision, and 2) the current knowledge of retinal image are complete only for linear, shift-invariant systems. representations. As a first contribution, we show that a Human vision certainly is more complicated than that, very popular type of convolutional neural networks however, this simple measure of the bandwidth of the (CNNs), called autoencoders, may develop human-like system is still of paramount significance in biological CSFs in the spatiotemporal and chromatic dimensions vision: the CSF filter is an image-computable model when trained to perform some basic low-level vision that roughly describes the kind of visual information tasks (like retinal noise and optical blur removal), but that is available for humans (Watson & Ahumada, not others (like chromatic) adaptation or pure 2016). Moreover, although it is defined for threshold reconstruction after simple bottlenecks). As an conditions, there are many examples that illustrate illustrative example, the best CNN (in the considered set the relevance of the CSF in more general situations of simple architectures for enhancement of the retinal signal) reproduces the CSFs with a root mean square (Watson et al., 1986; Watson & Malo, 2002; Watson & error of 11% of the maximum sensitivity. As a second Ahumada, 2005), so it has shaped image engineering contribution, we provide experimental evidence of the over decades (Mannos & Sakrison, 1974; Hunt, 1975; fact that, for some functional goals (at low abstraction Wallace, 1992; Taubman & Marcellin, 2001). This level), deeper CNNs that are better in reaching the theoretical and practical relevance motivated the quantitative goal are actually worse in replicating measurement of CSFs, not only for spatial gratings human-like phenomena (such as the CSFs). This (Campbell & Robson, 1968), but also for moving low-level result (for the explored networks) is not gratings (Kelly, 1979), chromatic gratings (Mullen, necessarily in contradiction with other works that report 1985), spatiotemporal chromatic gratings (Díez-Ajenjo advantages of deeper nets in modeling higher level et al., 2011), at different luminance levels (Wuerger vision goals. However, in line with a growing body of et al., 2020), and for alternative basis of the image space literature, our results suggests another word of caution (Malo et al., 1997). about CNNs in vision science because the use of simplified units or unrealistic architectures in goal Principled explanations of the human CSFs optimization may be a limitation for the modeling and understanding of human vision. Of course, the psychophysical CSFs have physiological roots in the spatiotemporal bandwidths of the center-surround cells tuned to achromatic and chromatic stimuli (Enroth-Cugell & Robson, 1966; Citation: Li, Q., Gomez-Villa, A., Bertalmío, M., & Malo, J. (2022). Contrast sensitivity functions in autoencoders. Journal of Vision, 22(6):8, 1–45, https://doi.org/10.1167/jov.22.6.8. https://doi.org/10.1167/jov.22.6.8 Received February 28, 2021; published May 19, 2022 ISSN 1534-7362 Copyright 2022 The Authors This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 2 de Valois & Pease, 1971; Ingling & Martinez-Uriegas, can be optimized with automatic differentiation to fulfill 1983; Martinez-Uriegas, 1994; Cai et al., 1997; Reid & whatever computational goal under consideration. As Shapley, 1992, 2002). However, the physiological basis a result, deep learning models are becoming standard of psychophysical phenomena does not explain the in visual neuroscience (Kriegeskorte, 2015; Yamins & functional role (or goal) of the underlying computation DiCarlo, 2016; Cadena et al., 2019). (Marr & Poggio, 1976; Marr, 1982). The discussion According to the above, the study of the CSF of about the goal of certain mechanism relies on deriving artificial neural networks is interesting for two reasons: the biological behavior from a computational principle. 1) CNNs are flexible and easily optimizable tools that In the specific case of the CSFs, the classical work of may allow us to investigate principled explanations of Atick et al. (Atick et al., 1992; Atick & Redlich, 1992; the human CSFs with more generality than the classical Atick, 2011) derived the spatiochromatic CSFs from methods considered, and 2) given the widespread use of the maximization of the information transferred from CNNs in computer vision and their recent use in visual the input to the response of the system that, under neuroscience, the eventual emergence of human-like certain conditions, is equivalent to optimal deblurring sensitivities in these artificial systems has intrinsic and denoising of the retinal signals. These classical interest. explanations were based on clever observations about Very recently, two groups have reported the 2nd order properties of natural images, but relied complementary results on the emergence of CSFs in on linear filtering models. As a result, the consideration deep networks: first, in order to explain the human-like of more flexible (nonlinear) models could lead to nature of some of the brightness and color illusions a better fulfillment of the computational goal and, in CNNs trained for low-level visual tasks found in eventually to better explanations of the CSFs. A step Gomez-Villa et al. (2019), a novel eigenanalysis of the forward in a more general (nonlinear) derivation of networks was proposed (Gomez-Villa et al., 2020). these phenomena from low-level principles was given This analysis revealed the emergence of human-like by Karklin and Simoncelli (2011), where they obtained chromatic channels and achromatic and chromatic sensors with center-surround receptive fields optimizing CSFs in these channels. Then Akbarinia et al. (2021) the information transferred by a linear+nonlinear have found that networks trained for high-level visual layer of neurons with noisy inputs. However, this tasks, such as classification, also may develop an work did not consider the chromatic or the temporal achromatic CSF, in this case not explicitly imposing dimensions of the problem, and no explicit comparison low-level constraints. with the psychophysical CSFs was done. Similarly, (Lindsey et al., 2019) also reproduced center-surround Contributions and scope of this work sensors close to the retina when training anatomically constrained artificial neural nets (in this case, training • First, following Atick et al. (1992), Atick and for a higher level task, such as object recognition). Redlich (1992), Atick (2011), Karklin and Again, these center-surround cells eventually would Simoncelli (2011), we reconsider principled induce CSFs, but this was not analyzed in that paper. explanations of the CSFs from low-level visual tasks in light of new available methods: i) the Emergence of CSFs in artificial neural networks current tools from deep-learning, and ii) the current knowledge of retinal image representations. Automatic differentiation (Baydin et al., 2018) has We check the emergence of spatiotemporal simplified the search of computational principles in chromatic CSFs in a wider range of low-level vision science because it allows the optimization of (goal/architecture) situations with more realistic complex models according to different goals without inputs. the burden of obtaining the analytical derivatives Regarding the retinal input, we use recent models of the goals w.r.t. the model parameters. Automatic of the human modulation transfer function (MTF) differentiation is at the core of the current explosion (Watson, 2013), and recent calibrated estimations of of deep learning (Goodfellow et al., 2016). A full the noise in the cones (Esteve et al., 2020) obtained analytical description of the derivatives of realistic via the retina models implemented in ISETbio nonlinearities in visual neuroscience is certainly possible (Cottaris et al., 2019, 2020). In this way, here we (Martinez et al., 2018), but the widespread availability generate realistic spatiotemporal noisy inputs to the of deep-learning tools for simplified neurons makes the visual pathway in a plausible representation: the exploration of these artificial architectures much easier. cones of (Stockman & Sharpe, 2000) tuned to long, Conventional convolutional neural networks (CNNs) medium, and short wavelengths (LMS cones). are too simplistic from the neuroscience perspective,1 Regarding the deep learning tools, we use but the freedom to combine multiple of such simplified spatiotemporal extensions of the convolutional layers in any possible way may compensate this autoencoders used in our analysis of color illusions shortcoming. In the end, one has a flexible system that in CNNs (Gomez-Villa et al., 2019, 2020). We Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 3 elaborate on the proper determination of the CSF The structure of this article is as follows. We used for convolutional autoencoders: instead of the estimating contrast sensitivity in autoencoders that linear characterization of the autoencoder used in extends the theory proposed in Gomez-Villa et al. Gomez-Villa et al. (2020), which hides its nonlinear (2020) to obtain the CSFs of autoencoders with an nature into a single matrix, here we stimulate the analysis of the energy (or standard deviation) of the networks with gratings of different contrast. In input and the output gratings. Experiments describe this way, the changes of the attenuation functions the considered low-level visual tasks (compensation describe the nonlinearities of the system. of biodistortions, chromatic adaptation, and signal Regarding the architectures, in this work we focus reconstruction after bottlenecks) and the setting of the on autoencoders that reconstruct the signal in the numerical experiments. Results show the main empirical input domain as opposed to the consideration of findings of the work: the emergence of human-like more general architectures that encode the images CSFs in the spatiotemporal and chromatic dimensions into more abstract representations to achieve higher in shallow CNN autoencoders trained to minimize the level tasks, such as classification. This limitation distortion introduced by the optics of the eye and the in scope is reasonable if one wants to model early noise in the cones. Finally, we discuss the implications vision stages like the lateral geniculate nucleus of the empirical results: on the one hand, statements (LGN), which do not imply change of domain and about the goal or organization principles are difficult may function according to error minimization and to separate from the implementation because the final signal enhancement principles (Martinez-Otero behavior very much depends on the algorithmic level (or et al., 2014). If the CSFs are related to the selected architecture). On the other hand, special care response of LGN neurons, as is usually assumed, has to be taken in using deep models in low-level vision autoencoders seem a reasonable computational science: their ability for function approximation may framework to use. make them excel in the performance of a sensible score, Regarding the tasks, in this low-level context but without the appropriate architecture constraints, with autoencoder tools, we consider different visual this does not guarantee the similarity with humans. tasks which may be implemented as early as in Appendix A provides details of the implementation the retina-LGN path: a) the enhancement of the of the models. Appendix B describes the image/video retinal signal (related to information maximization) datasets to train the models and the sinusoids used to when the input is subject to different degrees of probe the networks. Appendix C illustrates the proper degradation owing to different pupil diameters training and convergence of all the considered CNNs in or different plausible levels of retinal noise, b) all experiments. It shows the learning curves and explicit the compensation of changes in the spectral examples of the responses (reconstructed signals in test) illumination of the scene in a reasonable range of for all the considered goal/architecture scenarios. color temperature, and c) the reconstruction of the signal when some information may be lost in eventual bottlenecks. Methods: Estimating contrast • Second, here we provide experimental evidence of “deeper CNNs are not necessarily better” (in sensitivity in autoencoders representing this abstraction level). The bigger generality of flexible CNN models over fixed linear Here we consider different linear characterizations of models is obvious, but one may ask: do more the autoencoders including the eigenanalysis proposed flexible architectures necessarily lead to more in Gomez-Villa et al. (2020). That theory is extended human CSFs? or does better accuracy in the goal with the explicit consideration of the image acquisition imply more human CSFs and masking behavior? process in the human eye, which leads us to propose Consistently with previous results in low-level tasks a procedure to estimate the autoencoder CSF that is (Flachot et al., 2020; Gomez-Villa et al., 2020), our more connected to the definition of the CSF in human CSF results presented also seem to favor shallow observers. networks (in the explored range of architectures). Our findings at this low level of abstraction complement other results where deeper Autoencoders architectures actually imply closer resemblance to Autoencoders are artificial networks that transform human behavior (Yamins et al., 2014; Cichy et al., the signal into an inner representation through 2016; Cadena et al., 2019; Lindsey et al., 2019). an encoder, and a decoder transforms this inner But this is not contradictory because they refer representation back in the input domain. to different abstraction levels (high-level object recognition vs. our low-level color constancy and Nθ (x) error minimization goals). x −→ y (1) Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 4 In Equation 1, we do not made explicit the encoding case of humans the attenuation factor, CSF( f ), has to and decoding operations, that is, x and y are in the be obtained from contrast thresholds because there is image space. In this work, we will not make any no access to the output. However, for autoencoders, the assumption on the nature of the inner representation of computation of the output is straightforward. If the the autoencoder. This is because the basic goal function degradation of the acquisition is taken into account, in autoencoders (reconstruction error) is defined in the the sinusoids, b f , used to simulate the measurement of image domain, shared by input and response. Moreover, the CSF have to undergo the degradation as well, and with the appropriate stimuli, the CSF characterization we should consider an eye+network system, S: can be defined in this image domain. Following Gomez-Villa et al. (2019, 2020), we focus on convolutional autoencoders. We discuss and explore different computational goals, but, for now, let’s consider that the parameters θ are trained to (2) compensate the blur and noise introduced in the signal Therefore, one could check the attenuation factor by by the image acquisition process. In this context, given comparing the standard deviation of output and input: a clean image, xc , the input to the neural system would be a distorted version: x = H · xc + nr , where H is a σ (Sθ (b f )) σ (Nθ (H · b f + nr )) blurring operator related to the optics of the eye and CSF( f ) = = (3) nr is the noise associated with the response of the σ (b f ) σ (b f ) LMS photodetectors at the retina. Both H and nr are Note that the CSF ratio in Equation 3 (which uses unknown to the neural system. The goal of the network degraded sinusoids to probe the network) is different at this early stage is inferring xc from x. Accurate from checking the Fourier response of the network, models of LGN cells show that this may be one of the where one would use clean sinusoids at the input: goals of the biological processing after retinal detection (Martinez-Otero et al., 2014). σ (Nθ (b f )) In the supervised learning setting of artificial NF(f ) = (4) neural networks, the parameters θ of the network σ (b f ) are selected so that the average reconstruction error The relation of Equation 3 with the regular ε = |xc − Nθ (x)|2 is minimized over a set of training determination of the CSF in humans is illustrated in images (Goodfellow et al., 2016). In our case, we refer Figure 1. Of course, the ratio in Equation 3 should be to the average reconstruction error as εLMS because computed for low-contrast sinusoids to keep parallelism the input signal is expressed in the LMS cone space with human CSF and keep the (eventually) nonlinear (Stockman & Sharpe, 2000). Of course, supervised autoencoder in the low-energy range. For chromatic learning and parameter updates using backpropagation sinusoids the deviations have to be computed separately may not be biologically plausible (Lillicrap et al., 2020). over the achromatic, red–green, and blue–yellow color However, our initial aim here is looking for statistical channels (Mullen, 1985). In this work we use a classical explanations of human frequency sensitivity and hence opponent color space (Hurvich & Jameson, 1957) to artificial neural networks can be seen as convenient generate achromatic and purely chromatic gratings and tools to optimize the selected goal. With this focus to decompose the corresponding responses. on the goal, the specific learning algorithm is not as Of course, plain attenuation for sinusoids in important as ensuring that the final network actually Equation 3 may not provide a full description of the fulfills the goal. We will see that the situation may not action of nonlinear systems. In principle, it is not be that simple because networks optimizing the same obvious why we should perform the analysis in a specific goal with equivalent performance may display human or basis. Therefore, one should check to what extent waves non-human CSFs, depending on their architecture. are indeed eigenfunctions of the system. A way to test this point is linearizing the response of Filter definition of the CSF and linearized the autoencoders in the low-contrast regime and check autoencoders that it is shift invariant. Using a Taylor expansion, the response for low-contrast images can be approximated The CSF describes the linear response of human by the Jacobian around the origin (the zero-contrast viewers for low-contrast sinusoids (Campbell & Robson, image, 0, which is just a flat gray patch): 1968; Kelly, 1979; Mullen, 1985). In that linear setting, y = Sθ (x) the CSF describes an input-output mapping where an input sinusoid of frequency f , the basis function b f , ylow = Sθ (0 + xlow ) leads to an output, y f , with attenuated contrast (or ≈ Sθ (0) + ∇x Sθ (0) · xlow (5) attenuated standard deviation, σ ). The output standard deviation is given by, σ (y f ) = CSF( f ) σ (b f ). In the ylow ≈ ∇x Sθ (0) · xlow Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 5 Figure 1. Definition of the CSF as a frequency-dependent attenuation factor in a system to develop low-level vision tasks. The diagram illustrates transforms in the visual signal from the input stimulus (A), the degraded signal owing to optical blur and retinal noise (B), the process of the early neural path where the output is still in a spatial LMS representation (C) modeled here by autoencoders, and additional mechanisms that compute a decision on the visibility (D). In the conventional view of the CSF as a filter, the process from A to C is assumed to be linear and (in humans) the visibility of gratings is assumed to be based on the amplitude of the response at the point C. In human psychophysics (with no access to C) the observer makes visibility decisions, and the attenuation factors are determined from the thresholds. When dealing with artificial systems we do have access to the response in C so we do not need to model the decision mechanism and we can simply estimate the CSF from the ratio in Equation 3. where we assumed that the response for zero-contrast to the CSF. A direct comparison of the eigenvalue images is zero. If the behavior of the system at this spectrum with the CSF may not be simple because low-energy regime is shift invariant, the Jacobian the eigenfunctions may differ from Fourier sinusoids. matrix can be diagonalized as ∇x Sθ (0) = B · λ · B−1 , Examples of this include isotropic systems (with with extended oscillatory basis functions in the columns a constant sensitivity for certain | f | independent of B (and rows of B−1 ). Fourier basis and cosine of orientation). In this case, the eigenbasis may be basis are examples of extended (nonlocal) oscillatory not sinusoids, but arbitrary linear combinations functions that diagonalize shift invariant systems. The of sinusoids of the same frequency and different reason for this result is equivalent to the emergence of orientation. cosine basis when computing the principal components Nevertheless, if the linearized version of the system of stationary signals (shift-invariant autocorrelation) (the Jacobian at 0) is shift invariant, which can be seen (Clarke, 1981). As a result, the slope of the response from a convolutional structure in the Jacobian matrix, for low-contrast sinusoids (the CSF) will be related to oscillatory waves are eigenfunctions of the system, and the eigendecomposition of the Jacobian of the system hence Equation 3 may provide a good description of at 0. Let’s compute the response for a sinusoid in this the behavior of the system. Taylor/Fourier setting to see the relation. A basis function b f with specific frequency f is orthogonal to all rows (sinusoids) in B−1 except that of the same Alternative linear characterizations of the  frequency, that is, B−1 · b f = δ f f . And this delta selects autoencoders the corresponding column (of frequency f ) among all the columns in the matrix B: A two-dimensional (2D) cartoon of the impact of the degradation and restoration processes in the probability y f = Sθ (b f ) ≈ ∇x Sθ (0) · b f density function (PDF) of the signal can illustrate ≈ B · λ · B−1 · b f alternative characterizations of the neural networks  optimized to enhance the retinal signal (see Figure 2). ≈ B · λ · δf f In this diagram, two-pixel natural scenes (left) follow ≈ λf bf (6) a PDF obtained from independent Student t-sources mixed by a matrix that introduces strong correlation So the slope of the response for basis functions of between the luminance of the pixels. This kind of frequency f is λ f (the corresponding eigenvalue of the two-pixel representations is common to describe the Jacobian of the autoencoder). As a result, for systems statistics of natural images (Simoncelli & Olshausen, with shift invariance in the low-contrast regime, the 2001), and mixtures of sparse components is a widely eigenvalues of the linear approximation of the system accepted model for natural scenes (Hyvärinen et al., (eigenvalues of the Jacobian) are conceptually similar 2009; van den Oord & Schrauwen, 2014; Malo, 2020), Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 6 Figure 2. Degradation of the signal PDF and linear and nonlinear strategies to compensate the degradation. The axes of the plots represent the luminance in each of the photodetectors of two-pixel images as in Simoncelli and Olshausen (2001). (A) The PDF of natural scenes: marginal heavy tailed distributions of oscillatory functions mixed to have strong correlation in the pixel domain. Optical blur (second panel) implies contraction of the high frequencies. Additive retinal noise implies the convolution by the PDF of the noise leading to the PDF in (B). The solutions to the restoration problem at point C may include i) nonlinear transforms such as the one represented by the PDF at the right-top, and ii) linear transforms as the one at the right-bottom. and appropriate enough for this illustration. In this • The optimal linear solution: The matrix M that diagram, the low-frequency direction corresponds better relates the input x with the desired output with the main diagonal (where the two pixels have the xc . This is the M that minimizes the expected value same luminance) and the high-frequency direction E{|xc − M · x|2 }. Assuming a representative set is orthogonal (for images where one of the pixels is of N clean/distorted pairs stacked in the matrices brighter than the other). The zero-contrast image is at X c = [xc(1) xc(2) . . . xc(N ) ] and X = [x(1) x(2) . . . x(N ) ], the crossing point of the frequency axes. the optimal solution in Euclidean terms is given by Optical blur implies the attenuation of high- the pseudoinverse: frequency versus low-frequency components and hence the contraction of the dataset as shown in the second M = Xc · X† (7) panel. Assuming linear and noisy photoreceptors, the PDF of the retinal response results from the • Globally linearized network: The matrix M that convolution of the PDF of the blurred images with the better describes the nonlinear behavior of the PDF of the noise (the function with circular support). network over the whole set of natural images. This The result (third panel) is the input to the autoencoder, is the M that minimizes E{|S(x, θ ) − M · x|2 }. whose goal is recovering the distribution at the first Assuming a representative set of N input/output panel. Linear solutions are limited to global scaling of pairs stacked in the matrices X = [x(1) x(2) . . . x(N ) ], the domain (for instance, by inverting the contraction and Y = [y(1) y(2) . . . y(N ) ], the solution is given by introduced by the blur), whereas nonlinear solutions the pseudoinverse: may twist the domain in arbitrary ways. In this setting, the computation of the CSF according M = Y · X† (8) to Equation 3 means putting low-contrast sinusoids (e.g., the samples highlighted in green in the third • Locally linearized network at 0: The matrix M panel) through the system, and checking the amplitude that better describes the nonlinear behavior of the of the output (green dots at the panels at the right) over network for low-contrast images. This is the M the directions of the input. This nonlinear example that minimizes E{|∇x S(0, θ ) · xlow − M · xlow |2 }. Of illustrates the fact that the behavior can be contrast course, this could be empirically approximated by dependent (see the different twist in the concentric M = Y low · X †low , but in this case the obvious exact contours). This graphical view illustrates the difference solution is: between three possible linear characterizations, with y = M · x: M = ∇x Sθ (0) (9) Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 7 Although the optimal linear solution (or the optimal averaging over the realizations, and 2) by using relatively linear network) is a convenient reference to describe high-contrast sinusoids so that the effect of the residual the problem, the other two options are different noise is negligible. characterizations of the autoencoder. Equation 8 In this work (for computational convenience) we summarizes the behavior of the network in a single used the second approach: we probed the models matrix, and Equation 9 is a description only valid with sinusoids with contrasts in the range [0.07, 0.6]. around 0, and hence more closely connected to the The lower limit is certainly higher than the minimum low-contrast regime of the CSF. The eigenanalysis absolute threshold of the standard spatial observer cited for ∇x S in Equation 6 can be applied for the three (which is approximately 0.005) (Watson & Malo, 2002; matrix characterizations, but it is important to note the Watson & Ahumada, 2005). Nevertheless, we choose differences between them. this range for two reasons: first, 0.07 is the average of The Jacobian of cascades of linear+nonlinear layers the threshold achromatic contrasts in the standard (as in autoencoders based on CNNs) can be obtained spatial observer, and second, we empirically checked either analytically,2 or it can be obtained via automatic that the effect of the noise was negligible above this differentiation or alternative methods based on system value. identification (Berardino et al., 2017). However, these procedures are tedious, so in Gomez-Villa et al. (2020), we took the more straightforward approach represented by Equation 8. Experiments The different linear characterizations considered in this section and the diagram in Figure 2 illustrate The Introduction raised questions on the role that the behavior of a nonlinear autoencoder for of low-level vision goals to explain the CSFs, the high contrasts may be substantially different from emergence of the CSFs in autoencoders working the threshold behavior. Therefore, the attenuation to solve these goals, and the eventual advantage of of sinusoids by the linearized system (by the matrix progressively more flexible models in explaining the Equation 8) will be compared with the result of CSFs. To address these issues in the more general Equation 3. spatiotemporal chromatic case, 1) we perform two extensive experiments (one with images, and one with video) to compensate biologically sensible degradation Limitation of the proposed CSF definition in of the retinal signal (compensation of biodistortion), autoencoders using a range of CNN architectures of different depth or flexibility, 2) we consider alternative low-level To maximize the equivalence to human CSFs, the functional goals such as chromatic adaptation and proposed procedure (the ratio in Equation 3, which the compensation of the effect of bottlenecks, 3) we compares the signals at points C and A in Figure 1) consider different levels of biodistortion, chromatic implies the consideration of the retinal degradation shifts in different directions, and bottlenecks with process. This consideration of the retinal noise will be different restrictions, and 4) we consider the consistency shown to improve similarity with human CSFs in the of the results under changes in the statistics of the experiment 3, but it comes at a cost. Note that, even if signal. In this section, we describe the experimental the role of the autoencoder is compensating the retinal setting of these simulations. noise, complete removal is not possible. Therefore, there is some residual distortion in the response after the autoencoder. As a result, the standard deviation in Functional goals the numerator of the proposed Equation 3 not only measures the contrast of the output grating, but also Compensation of retinal biodistortion (biological measures the energy of the residual noise. In this way, blur and noise) consists of overcoming the degradation when the contrast of the sinusoids b f is very small, as introduced in the acquisition of the visual signal. expected in threshold conditions, the standard deviation Specifically, the top panel in Figure 3 shows how a maybe measuring more the residual noise than the natural scene is degraded at the output of the retina contrast of the output. The limitation of Equation 3 is according to the variations of the eye MTF for different that it has to be applied to sinusoids of relatively high pupil diameters (from top to bottom, d = 2 mm, contrasts so that the energy of the response coming d = 4 mm and d = 6 mm), and a sensible range of from the sinusoid is bigger than the energy of the Poisson retinal noise levels (from left to right, Fano response coming from the noise. factors F = 0.25, F = 0.5 and F = 1). Variations of One can overcome this limitation in two ways: 1) the MTF have been simulated with the expression by computing the response many times for different in Watson (2013), and the noise in LMS sensors has noise evaluations and cancelling the residual noise by been estimated in the discrete representation of the Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 8 Figure 3. Functional goals. Possible low-level goals of the autoencoders are compensating the following distortions in the visual input. Top panel: different levels of retinal biodistortion. Bottom panel (first row) Changes in the spectral illumination. Bottom panel (second row) Changes in illumination + retinal biodegradation. An alternative low-level goal is the reconstruction of the signal in presence of bottlenecks (as in the architectures considered below, Figure 4 right). Note: This selfie of the corresponding author was sent to the first author to test the models on a copyright-free image. input digital image as in Esteve et al. (2020). In that integration of the photocurrents over space/time work, noise was obtained by stimulating the ISETBio reveals the effective Poisson nature of the noise (in the retinal model (Cottaris et al., 2019, 2020) with flat original LMS units) and allows the estimation of the stimuli of controlled size and tristimulus values over effective Fano factor in the original discrete grid of short and long exposure times. Cartesian resampling the input image. In that way, we can easily generate of the random cone mosaic of the retinal model and calibrated noisy retinal images by adding this effective Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 9 Poisson noise in the LMS representation of the digital reconstruction of the signal in presence of bottlenecks image. The illustrations in Figure 3 come from the is a sensible low-level goal to explore. transformation of the LMS tristimulus images into the RGB digital counts for proper display. Compensation of bottlenecks and biodistortion As stated, errors in the acquisition front end Chromatic adaptation (biodistortion) do exist, so their consideration together with bottleneck compensation makes the goal more Consists of the compensation of the deviations of realistic. the signal induced by the change of illuminant. The All in all, we explored nine levels of biodegradation, bottom panel shows how the image of a natural scene chromatic adaptation in the blueish and the reddish changes under changes in the shape of the spectral directions (T = 8600 K and T = 4400 K, respectively), radiance of the illuminant. The change of illuminant and the combination of the central biodistortion with in a digital image was simulated in this way: each pixel the considered chromatic deviations. We considered of the image was associated with a reflectance chosen a pure reconstruction task with the eight bottleneck from a large database of natural reflectances so that configurations in Figure 4, and the compensation of under an equienergetic illuminant led to the tristimulus these bottlenecks was also combined with the central values of the pixel. Then, a black-body radiator, which biodistortion case. The optical/retinal degradation simulates natural ambient light along the day, was used in movies was applied in a frame-by-frame basis. to generate spectra of the same energy but different No experiments involving chromatic adaptation or shape. From there, we could get versions of the scene bottlenecks were done in movies, but only in natural under arbitrary color temperatures. This process is and cartoon images. straightforward using the functionalities and databases The above computational goals are all measured of Colorlab (Malo & Luque, 2002). Of course, this in distortion terms, or how well the deviations εLMS process is just an approximation because it disregards were compensated. However, even within this low the (unknown) geometry of the scene and assumes a abstraction level, other computational goals could be flat Lambertian world. Nevertheless, as illustrated in considered together with the distortion, as for instance the examples in Figure 3, it does a good qualitative the information or the energy of the signal. In the job to generate controlled samples to check chromatic experiments we restrict ourselves to the considered cases adaptation in large image databases. of distortion minimization and purely architectural bottlenecks. The discussion suggests how the goals Compensation of chromatic shifts and biodistortions considered here could be related or combined with other kind of goals or more general (energy or information) The reason to consider this combination is that bottlenecks. pure chromatic shifts with no additional distortion is not a realistic input for the visual pathway: the image acquisition front-end does exist and, hence, what we Architectures called biodistortion has to be taken into account. Such combination of distortions is illustrated by the second In this work, we consider 2D CNNs that act on row of the bottom panel in Figure 3. Note that in spatiochromatic signals and three-dimensional (3D) the examples involving chromatic deviations (bottom CNNs that act on spatiotemporal signals (color images panel) we introduced a flat-reflectance frame to help the and color videos). models to cope with the chromatic adaptation3 . The set of explored architectures is shown in Figure 4. These are variations of the basic toy networks studied in Gomez-Villa et al. (2019, 2020): autoencoders Compensation of bottlenecks (pure reconstruction) with convolutional layers made of 8 feature maps with Consists of recovering the input after the signal has kernels of spatial size 5 × 5 and sigmoids or rectified gone through a bottleneck. Examples of bottlenecks linear units (ReLU) as activation functions. From that include the restriction of the spatial resolution or the starting point, here we consider a range of nets of restriction of the number of features (or channels) increasing depth and flexibility: from the linear network in the representation. Figure 4 (right panel) shows in Equation 7 (as a convenient baseline reference of one an illustrative range of architectures: from cases that layer with no flexibility), and CNNs with two layers to expand the number of features (no bottleneck) to a eight layers, both for the 2D and 3D cases. Moreover, variety of cases that introduce local pooling, reduce we also consider a range of architectures with different the number of features, or try to compensate the effect bottlenecks, in this case, only 2D. of spatial undersampling by increasing the number of Of course, the range of possible architectures is features. Bottlenecks may imply severe information loss virtually infinite and an exhaustive exploration of the if the representation is not optimized. Therefore, pure architecture space is out of the scope of this work. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 10 Figure 4. Architectures. (Left) Range of architectures of different depth following the basic structure of the nets studied in Gomez-Villa et al. (2019, 2020): three channels at the input and at the last (output) layer. In this work the input is in an LMS color representation. The rest of the layers have eight features with no undersampling or bottleneck, as represented by the eight blue lines of the same length. These architectures of increasing depth are used to study the compensation of the biodistortion in images and videos. The best of these architectures in terms of CSFs (which turns out to be the two layers example) is used with images to explore chromatic adaptation. (Right) Range of architectures used to illustrate the effect of bottlenecks. In this case, the inner layer in the four layer architecture at the left is systematically expanded (in A) or contracted (from C to G, either in the number of features or in the spatial resolution. See numbers and indications of pooling/upsampling and corresponding length of the layers-) to generate a range of bottlenecks. Finally, an illustrative U-Net with no residual connections is also considered in the experiments with bottlenecks. However, note that the considered set of architectures of generic CNNs for denoising (Zhang et al., 2017; of progressive flexibility and constraints is appropriate Soh & Cho, 2021) and deblurring (Tao et al., 2018), for the aim of this work for two reasons: 1) these which gave excellent results with arbitrary Gaussian architectures do a good job in fulfilling the goal so noise and blur in Gomez-Villa et al. (2020), is not they are good examples to reason about systems that satisfactory with biological distortion. In particular, work according to the considered function, and 2) generic enhancement algorithms did not produce they display a range of flexibility and accuracy in the better results than the considered simple architectures goal which is appropriate to illustrate the proposed (specifically trained for this biodistortion). questions (eventual emergence of the CSF and other Of course, this does not mean that the toy models nonlinearities, and qualitative effect in the CSF of used here are better than the state of the art, nor that increased flexibility and improvements in the goal state-of-the-art models are intrinsically unable to deal accuracy). with this biological degradation. One could certainly The first point (the considered toy models do a fine-tune these deep architectures for the biodistortion reasonably good job in fulfilling the goal) is a technical and then get a better result than with the considered set issue that is demonstrated by the performance tables of architectures, but that is not the goal of this work. shown below and by the specific learning curves and The relevant argument in favor of the considered (toy) reconstructions included in Appendix A. However, architectures for our purposes here is this: the fact that to put this quantitative performance in context, it generic blind restoration CNNs need to be retrained is interesting to note that the retinal biodistortion to get better results than the proposed models means is not an easy task to solve for general-purpose that these simple models can be considered as good state-of-the-art image restoration CNNs. In particular, (enough) examples of systems actually fulfilling the following Gomez-Villa et al. (2020), on top of the goal. described toy networks, the computation of the CSFs Regarding the second point (the considered set of of cutting-edge deeper models designed for restoration architectures is good enough to illustrate interesting could be an illustrative limit to consider. However, we questions), consider that i) according to the results found that the combination of representative examples presented below (Results, Tables 1 and 4) the toy Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 11 nonlinear models decrease by up to 35% and 48% the where the scaled attenuation factors of the model are error of the optimal linear solution in images and video, related to the raw attenuation factors of the model as: respectively, and ii) the best nonlinear model reduces the error of the shallower nonlinear model by 21% and CSFscaled ( f ) = αCSF · CSFraw (α f · f ) (11) 12% in images and video respectively. In summary, the considered set of architectures (progressively deeper CNNs and a range of bottlenecks) In the following, we report the scaled CSFs together does a reasonable job in optimizing the goals, and it is with the scaling factors that minimize the distance with wide enough to illustrate changes in the achievement of human CSFs. the goals. As a result, the considered set of architectures It is important to mention that the relative scaling is appropriate to address the questions raised in the between the CSFs in the three chromatic channels is a introduction. characteristic feature of a network (or system) and it See Appendix A for implementation details. Data and should not be modified. Therefore, the same factors in code are available at http://isp.uv.es/code/visioncolor/ Equation 11 are applied to the three CSFs. With these autoencoderCSF.html considerations, the CSFs reported below represent See Appendix B for details on the databases to the closest approximation the models may give to the generate the training stimuli and the stimuli used to human CSFs, and hence the comparison between them probe networks. is fair. The magnitude of the RMSE errors has to be understood in reference to the maximum value of Assessing the quality of the CSF results the human sensitivity. As a convenient example to have in mind, RMSE – 22 corresponds with an The CSFs defined for the autoencoders may be average deviation of 10% of the scale of the human subject to two arbitrary scale factors. On the one hand, spatiotemporal CSF at every frequency and chromatic the response of the network could be multiplied by an channel. This is because the maximum sensitivity is arbitrary global scale factor and hence, the numerator approximately 200 for stationary gratings and about in Equation 3 (and the CSF amplitude) would be 220 for moving gratings (Watson & Malo, 2002; Kelly, multiplied by this scale factor as well. We refer to this 1979). global scale factor in the amplitude as αCSF . On the other hand, the sampling assumptions (or assumptions on the extent of the signal, or the viewing distance) List of experiments introduced in the description of the stimuli are arbitrary and they imply an arbitrary scaling in the frequency The empirical exploration of the considered axis of our Fourier domains. We refer to this scale architectures consists of six experiments. Experiments factor on frequency as α f . 1 through 5 deal with spatiochromatic stimuli and 2D The factor on amplitude is not a major problem: networks, and experiment 6 deals with spatiotemporal one network and a modified version with its outputs chromatic stimuli and 3D networks. As stated, the multiplied by αCSF are equivalent and their quality computational goals are measured by the Euclidean should be rated the same. The factor on frequency does distance between the reconstructed image and the not reduce the validity of the results either as long as original image, referred to as εLMS . The similarity it is moderate. Note that using the MTF expressions with the human behavior is measured in terms of the in Watson (2013), if the filter corresponding with a Euclidean distance between the model CSFs and the pupil of 3.5 mm is modified by applying α f = 0.75 or human CSFs, that is, the RMSE defined in Equation 10. α f = 4.5, the resulting MTF is similar to what would have been obtained with d = 2 mm or d = 6 mm, • Experiment 1: Spatiochromatic CSFs from respectively. Therefore, as changes in the MTF (the only biodistortion compensation by a range of element where the scaling in frequency matters) are architectures. This experiment is focused on the plausible if α f ∈ [0.75, 4.5], one should also discount central degradation shown in the first panel of moderate variations of this factor when assessing the Figure 3 (d = 4 mm, F = 0.5) and analyzes in quality of the CSFs. detail the CSFs for nine architectures: the optimal The similarity between the model and the human linear network, and eight CNN architectures with CSFs will be measured by the Euclidean distance two, four, six, and eight layers with either sigmoid between the CSF vectors, averaging over the frequency, or ReLU activations, all optimized according f , and the chromatic channels, c (achromatic, red–green to this distortion–compensation goal. Once and yellow–blue), which will be referred to as: the architectures are properly trained (using 20   12 ·103 images of the ImageNet database cited in  RMSE = (CSFscaled c ( f ) − CSFchuman ( f ))2 , (10) Appendix B, 18 ·103 for training and 2 ·103 for f,c validation), we get the numerical performance Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 12 of the models in the independent test set of 103 way, Equation 4, in which we simply put the clean images. The sizes of the train/validation/test gratings through the considered network. This will sets are the same in all experiments with images, stress the difference in the obtained CSFs when experiments 1 to 5. Throughout all the experiments, considering realistic spatial degradations or not. the performance is expressed as the average εLMS • Experiment 4: Consistency of the human/non-human of the reconstruction in LMS space over 20 batches CSFs under change in signal statistics. Here we of 50 randomly chosen images per batch. The reconsider the chromatic adaptation and the standard deviation over these 20 computations is degradation–compensation goals of experiment 3 also reported. The learning curves (train/validation) now using stimuli of (apparently) quite different and the reconstructions of one representative test spatiochromatic statistics: the images from the Pink image are given in Appendix C. Then, the CSFs Panther cartoons. All the other settings remain the (attenuation factors) of the trained models are same as in experiment 3. computed according to the method described in • Experiment 5: CSFs from bottleneck–compensation the Methods: Estimating contrast sensitivity in and biodistortion compensation. This experiment autoencoders for gratings of different contrasts. shows the CSFs of the systems that emerge from The eventual variation of the attenuation reveals imposing pure reconstruction of the signal in the nonlinear nature of the contrast response presence of bottlenecks in the network (the eight for gratings. In experiment 1, we also show the examples in Figure 4, right). Pure reconstruction is CSFs of the linear network and the linearized compared with the compensation of biodistortion versions of the nonlinear networks introduced in in the same architectures. Given the similarity the Methods: Estimating contrast sensitivity in between activation options found in experiment 1, autoencoders. From the results of experiment 1, here we just explore the ReLU case. one of the nonlinear models is chosen as having • Experiment 6: Spatiotemporal chromatic CSFs representative resemblance with human behavior from biodistortion compensation by a range of in terms of the CSFs (two layers with ReLU architectures. Here we check the fundamental activation). Experiments 2, 3, and 4 further explore findings of experiment 1 for spatiotemporal the behavior of this specific model in a number of chromatic gratings on 3D networks optimized for conditions. degradation–compensation. Given the similarity • Experiment 2: Consistency of the CSFs from between activation options found in experiment 1, biodistortion compensation over a range of here we just explore the sigmoid case. Therefore, distortion levels. This experiment is focused on the we explored five architectures: the linear one and representative architecture selected after experiment two, four, six, and eight layers with sigmoid. In this 1, and checks its CSFs when trained for the nine spatiotemporal case we used 22 ·103 video patches different degradation levels considered in the first in the learning (20 ·103 for training and 2 ·103 for panel of Figure 3. validation), and 3 ·103 for test. • Experiment 3: CSFs from chromatic adaptation and biodistortion compensation. This experiment checks the CSFs of the representative architecture selected after experiment 1, when it is trained Results for i) the biodegradation compensation alone ii) the degradation compensation together Results in all the experiments have two parts: 1) with compensation of a bluish illuminant, iii) the perception part, with the CSFs and the contrast the degradation compensation together with responses of the networks, and 2) the technical part, compensation of a reddish illuminant, iv) pure with evidences of the convergence of the models, compensation of a bluish illuminant, and v) pure numerical performance in reconstruction, and visual compensation of a reddish illuminant. In the examples of the performance in reconstructing images. illustration of Figure 3, these correspond with The main text is focused on the perception part, while the five distorted versions closer to the clean all the technical material is given in the Appendix C. image under equienergetic illuminant. As stated, the purely chromatic deviations are not realistic because they disregard the optics and retinal noise. Experiment 1: Spatiochromatic CSFs from However, they represent an illustrative reference. biodistortion compensation In the same vein, as a convenient reference, in this experiment we compute the CSF in two ways: a) Figure 5 shows the achromatic and chromatic CSFs the proposed (realistic) way, Equation 3, by putting of the considered models (the linear solution and the clean gratings through the retinal degradation the eight CNNs) together with the human CSFs for before entering the network, and b) the idealized convenient reference. The human data come from Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 13 Comput. goal CSFs Comput. goal CSFs εLMS RMSE εLMS RMSE Distortion 15.5 ± 0.2 Linear net 13.1 ± 0.1 24.4 CNNs Sigmoid ReLU Sigmoid ReLU Sigmoid ReLU Sigmoid ReLU nonlinear nonlinear nonlinear nonlinear linearized linearized linearized linearized 2 Layers 10.3 ± 0.1 10.8 ± 0.1 26.9 24.5 ± 0.3 12.5 ± 0.1 12.6 ± 0.2 24.1 22.8 4 Layers 8.9 ± 0.1 9.1 ± 0.1 26.6 28.5 12.5 ± 0.1 12.5 ± 0.1 23.2 23.1 6 Layers 8.7 ± 0.1 8.5 ± 0.1 29.7 ± 0.7 33.1 12.5 ± 0.2 12.5 ± 0.2 23.2 23.1 8 Layers 8.9 ± 0.2 8.7 ± 0.1 31.2 31.6 12.6 ± 0.1 12.7 ± 0.1 27.0 27.4 Table 1. Experiment 1: Emergence of CSFs from biodistortion compensation (computational goal and error in CSFs). The achievement of the computational goal is described by εLMS (error of the reconstructed signal in LMS) for batches of images of the independent test set (averages and standard deviations from 20 realizations with 50 images/batch). The distance between the CSFs of the networks and the human CSFs is measured by the RMSE between the functions, Equation 10. Uncertainty of the RMSE was estimated only in two cases (two layer ReLU and six layer sigmoid) and is represented here by the standard error of the corresponding means. It is interesting to note that the optimal linear solution (computed from the train set) has worse performance in the test set than the linearized versions of the networks. Numbers in bold style refer to nonlinear networks and numbers in regular style refer to linearized networks. the achromatic standard spatial observer (Watson & In this regard, the shallow networks (two layers) display Malo, 2002; Watson & Ahumada, 2005) and from a consistent decay of the gain (attenuation factor) with the measurements in Mullen (1985). The plots for the contrast. This decay has an impact on the contrast nonlinear models show the attenuation factors (CSFs) response curves for gratings. The contrast response for gratings of different contrast (dark to light colors curves describe the evolution of the amplitude of the mean lower to higher contrasts). response to a grating as a function of the contrast of These plots include the RMSE measure of the the grating. In humans, contrast response curves are difference of the artificial CSFs with the human CSFs. increasing saturating functions both for achromatic The insets also show the optimal values of the arbitrary gratings (Legge & Foley, 1980; Legge, 1981) and for scaling factors (α f , αCSF ) applied to the axes of the chromatic gratings (Martinez-Uriegas, 1997). The decay raw CSFs of the network to minimize the distance found in two layer CNNs implies a saturation of the with the human CSFs. Because these optimal scaling contrast response curves for these shallow CNNs, in line factors values were found exhaustively in all cases, the with human behavior. Figure 6 shows representative comparison of the final CSFs and RMSE values is fair. examples of these response curves: Although the Results show the emergence of a band-pass sensitivity two layer network (top row) consistently displays in the achromatic channel and low-pass sensitivities saturating behavior for every frequency, the deeper net in the chromatic channels. The bandwidth of the (bottom row) shows non-human (linear or expansion) chromatic channels is always substantially narrower responses. than the achromatic bandwidth. These properties are Finally, Figure 7 shows the CSFs corresponding qualitatively in line with human behavior. with the linearized versions of the nonlinear CNNs, Shallower networks (either ReLU or sigmoid) Equation 8. Of course, the linear approximations have display a greater resemblance with human CSFs. In contrast-independent behavior and hence the same particular, deeper nets introduce substantial distortion CSF for all contrasts. The global linear approximations in the chromatic channels: note that the red–green of the nonlinear models improve the resemblance of channel is over attenuated (particularly for the eight the CSFs with human behavior: the linearized shallow layer architectures but also in the six layer cases). The nets are closer to humans than the linear model, and RMSE scores summarize these differences and show linearization corrects over attenuation of the chromatic that shallower nets (two and four layers) provide better channels in the six layer models. However this increased explanations of the CSFs than deeper nets (six and similarity with human CSF comes at the cost of a eight layers). significant drop in the performance (see the increase Interestingly, the optimal linear solution (a single in εLMS error in Table 1). The linearization leads to dense layer with identity activation) is the one that rigid models that disregard the differences between the better reproduces the CSFs. However, by its linear original nonlinear models and behave more similarly. nature, it cannot include contrast-dependent behavior. In any case, linearization does not overcome the Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 14 Figure 5. Experiment 1 (spatiochromatic CSFs from the compensation of biodistortion). Attenuation factors for gratings of different contrast for a range of CNN autoencoders trained for biodistortion compensation. Achromatic, blue, and red lines refer to the CSFs of the achromatic, red–green, and blue–yellow channels, respectively. Dark to light colors refer to progressively higher contrasts (evenly spaced in the range [0.07–0.6]). The human CSFs (top left) and the CSFs of the optimal linear solution (bottom left) are also shown as a convenient reference. RMSE measures describe the differences between the model and the human CSFs. The plots also display the values for the scaling factors in frequency and amplitude described in the text, Equation 11. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 15 Figure 6. Experiment 1 (Illustrative contrast responses from the compensation of biodistortion). Representative examples of nonlinear responses for achromatic and chromatic sinusoids found. Saturation in these responses comes from the decay in the attenuation factors with contrast in Figure 5. Similarly, expansion comes from the increase in the attenuation factors. The linear behavior at the low-contrast regime has been plotted with dashed line as useful reference to highlight the nonlinear behavior. It is interesting to note that the saturating or expansive nature of the final contrast nonlinearity is a collective behavior that is not trivially attached to the specific (saturating or expansive) nonlinearities in individual neurons. overattenuation of the red–green channel in the eight (Table 1) so they are fulfilling the goal. And second, layer models. the learning curves in the Appendix C (Figure Table 1 shows that while deeper networks are 14) show that all models achieved a plateau in the significantly better at fulfilling the computational goal training thus indicating proper convergence. Moreover, (as expected from their increased flexibility), they are the asymptotic values achieved in the learning are worse than shallow nets in reproducing the human consistent with the εLMS in test shown in Table 1. behavior (as seen in Figures 5–7). As stated, RMSE errors in Table 1 have to be Progressive improvement in the goal for increasing interpreted in terms of the scale of the human CSF. For depth is numerically substantial (and also visible in the example, the best and the worst CNNs (RMSE = 24.4 reconstructed signals in Figure 14 in Appendix A) from and RMSE = 33.1, respectively) have average deviations two, four, to six layers, and the numerical performance of 11% and 15% with regard to the maximum human stays (statistically) the same for eight layers. For this sensitivity. Of course, a single figure of merit averaged last case, there are chromatic issues in line with what over frequencies and chromatic channels may hide was been found in the CSFs: the colorfulness of the an uneven distribution of the errors. For instance, reconstruction in Figure 14 is related to the relative gain consider the specific six layer sigmoid CSF shown of the chromatic channels. In particular, the consistent in Figure 5, which displays a clear over attenuation underestimation of the red–green CSF by the eight of the red–green channel. In that case, if the global layer CNNs (either using ReLU or sigmoid activation) RMSE = 30.2 is broken down into its chromatic leads to low-saturation images. Interestingly, this effect components we have 29.0, 35.0, and 26.8 for the is also visible in the reconstructed images coming from achromatic, red–green, and yellow–blue errors, which the linearized CNNs (Figure 15) and is consistent clearly point out that the biggest problem is in the with the strong attenuation of the RG channel in the red–green sensitivity. The same is true for the average linearized eight layer architectures in Figure 7. over spatial frequencies: the global description does It is important to stress that the deviations in the not stress the discrepancy in the low frequencies of chromatic CSFs in deep models do not come from the achromatic channel. That is why the (necessarily not fulfilling the goal or having poor convergence in limited) description in the tables comes together the training. First, all models (even the linear one) do with the explicit plots of the three CSFs for different reduce the error of the original retinal degradation contrasts. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 16 Figure 7. Experiment 1 (spatiochromatic CSFs for linearized networks in biodistortion compensation). The human CSFs and the CSF of the optimal linear network are also included as reference. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 17 Another important technical issue is the consistency increases with contrast, which is equivalent to contrast of the CSFs over random initialization. This is easy response curves that are not saturating. to check by training a number of times the same architecture for the same computational task and over the same set of stimuli but from different initial Experiment 3: CSFs from chromatic adaptation values of the model parameters. Given the intensive versus biodistortion compensation computation required,4 we checked this variability only in two illustrative models: one with reasonably Figure 9 (top row) shows the CSFs emerging human-like behavior (two layer ReLU), and another when the representative shallow network with with less-human CSFs (six layer sigmoid). In these human-like behavior in experiment 1 is trained two models, we retrained them 20 additional times for a range of alternative low-level tasks, some and recomputed the corresponding CSFs (results not involving compensation of the retinal degradation plotted). In the six layer case, all the explored seeds (first, second, and third cases), and some others only lead to a flat red–green CSF of too low sensitivity (i.e., involving chromatic adaptation (fourth and fifth). The a non-human behavior), and in some cases even the corresponding learning curves for the models and visual blue–yellow sensitivity was strongly attenuated too. On examples of reconstruction are given in the Appendix A the contrary, the two layer case systematically leads to (Figure 18). Table 2 (top) summarizes the numerical better CSFs, as summarized by the RMSE in Table 1, performance of the models in this experiment (εLMS for where the uncertainty is represented by the standard the computational goal, and RMSE for the CSFs). error of the mean. The shape of the sensitivities is First, lets focus on the case where the determination pretty consistent in both cases, always better for the of the CSF faithfully follows Equation 3 and hence two layer case. At the same time, and not surprisingly, we have a realistic eye+retina degradation (solid the six layer architectures systematically led to lower lines). Results show that only the cases where the task εLMS error. Only 1 of the 42 realizations (21 per model) involves the biodegradation imply a clear difference in led to a clear outlier (RMSE = 43.8 in the six layer bandwidth between the achromatic and the chromatic case) and even for this CSF-outlier the distortion channels. In the cases where there is only chromatic εLMS was not off the distribution. According to the adaptation, the three CSFs are wider and of similar observed consistency, the remaining 49 configurations bandwidth. This behavior is clearly non-human, as of task/architecture in the work were studied using a confirmed by the RMSE measures in the fourth and single random initialization of the parameters. fifth panels at the right. The next experiments explore the consistency of the Second, this difference is more clear in the idealized human-like behavior found in shallow autoencoders in cases, Equation 4, where clean sinusoids are used to a number scenarios. According to the results found in determine the CSFs (dashed lines). In this situation, experiment 1, we select the two layer ReLU autoencoder the CSFs of the purely chromatic goals are wider and as a representative example of shallow architecture with flatter indicating that the networks are not performing reasonable human-like behavior (RMSE of 11% of the any particular spatial modification in any chromatic maximum sensitivity) so we focus on this architecture channel. As a result, the RMSE values for the chromatic in experiments 2 through 4. adaptation cases (light style numbers below the frequency axis) substantially increase indicating poorer description of human CSFs. In this regard, the errors Experiment 2: Consistency of CSFs over a range for the cases in which the task involves biodegradation of biodistortions are lower, but they are even lower if the CSF is measured considering the realistic degradation in the Figure 8 shows the CSFs obtained when training input. the two layer ReLU net to compensate a range of In summary, the results show two trends. On the one retinal degradations (as described by the different pupil hand, human-like features emerge in the CSFs if the diameters and Fano factors). Learning curves that show degradation–compensation task is considered, but they the good convergence of the models and representative do not if only chromatic adaptation is considered. On visual examples of the reconstructions are given in the the other hand, the CSFs are closer to human in RMSE Appendix C (Figures 16 and 17). if the determination takes the retinal degradation into The results in Figure 8 show that band pass / account in the sinusoids. low-pass channels with distinct bandwidths consistently Finally, there is an interesting human chromatic appear in all cases, and the RMSE with human CSF feature that is well-captured by all the CNN models (25 ± 2), mean and standard deviation, stays in the low that were trained for chromatic adaptation: all of range of the values found in experiment 1. them display a sort of Von-Kries modification of the Regarding the evolution of the CSFs with contrast, red–green and yellow–blue channels. Note that, when it is important to note that for some conditions (low the red illuminant has to be compensated (third and blur and high noise) the gain in the achromatic CSF fifth cases), the red–green CSF is attenuated while the Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 18 Figure 8. Experiment 2: Consistency of human-like result over a range of retinal degradations. Spatiochromatic CSFs of the two layer ReLU net for a range of retinal degradations. The RMSE distortion over the nine retinal conditions is 25 ± 2 (mean and standard deviation). The line style conventions and meaning of the numbers is the same as in experiment 1. blue–yellow CSF is boosted, and the other way around (as opposed to regular photographic images). The in in the compensation of a bluish illuminant (second corresponding learning curves for the models and visual and fourth cases, where the blue–yellow channel is examples of reconstruction are given in the Appendix C attenuated). (Figure 19). Table 2 (bottom) summarizes the numerical performance of the models in this experiment. The parallelism in the results of experiments 3 and Experiment 4: Consistency of spatiochromatic 4 confirms the robustness of the behaviors shown in CSFs under changes of signal statistics experiment 3 to certain changes in signal statistics. Note that this parallelism does not mean that the CSFs Figure 9 (bottom row) shows the CSFs emerging are independent of the signal statistics. It just means when the representative shallow network with that they are invariant to this change of statistics. It human-like behavior in experiment 1 is trained for is important to remark that the low-level statistics the range of low-level tasks considered in experiment of these (apparently different) sources may not be 3 optimizing the performances over cartoon images that different. Colors of the Pink Panther images are Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 19 Figure 9. Experiment 3 (top row): CSFs from biodistortion compensation versus chromatic adaptation in natural images. From left to right, we consider the CSFs that emerge from (a) the compensation of biodegradation (left plot), (b) combinations of chromatic adaptation and degradation compensation (second and third plots), and (c) pure chromatic adaptation panels (fourth and fifth plots at the right). The cases in the columns first-to-third involve retinal degradation (with or without color adaptation) while the cases in the fourth and fifth columns only involve chromatic adaptation. For the sake of clarity, only low contrast results are shown. Solid lines → Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 20 ← correspond with CSFs determined using Equation 3 (where the input to the network includes realistic acquisition process). Dashed lines correspond with CSFs determined using Equation 4 (where the network is probed with clean sinusoids). The RMSE values in bold correspond with the CSFs determined in realistic conditions (curves in bold). The RMSE values displayed below the frequency axis correspond to the CSFs determined with clean sinusoids (dashed lines). In the cases involving chromatic adaptation the input sinusoids were color shifted according to the corresponding change in the illuminant. Experiment 4 (bottom row): Consistency of the results for different signal statistics (cartoon images). The computational goals are the same as above. The only difference is that the models are trained with cartoon images (from the Pink Panther show) as opposed to regular photographic images from ImageNet (used in experiments 1–3). Natural images Comput. goals Degradat. only Degradat. + blue adapt. Degradat. + red adapt. Blue adapt. Red adapt. Original εLMS 12.1 ± 0.2 18.1 ± 0.2 15.2 ± 0.3 13.2 ± 0.2 10.2 ± 0.3 εLMS after 2-layers ReLU 9.79 ± 0.08 8.7 ± 0.1 8.9 ± 0.1 1.92 ± 0.05 1.01 ± 0.01 RMSE of CSFs CSFrealist. 28.4 27.3 29.3 34.9 42.6 CSFsimplist. 32.4 33.2 40.2 47.2 Cartoon Images Comput. Goals Degradat. Only Degradat. + Blue Adapt. Degradat. + Red Adapt. Blue Adapt. Red Adapt. Original εLMS 12.9 ± 0.2 18.1 ± 0.1 15.6 ± 0.2 13.4 ± 0.3 9.1 ± 0.3 εLMS after 2-layers ReLU 10.14 ± 0.08 9.5 ± 0.1 9.2 ± 0.1 1.34 ± 0.02 0.837 ± 0.06 RMSE of CSFs CSFrealist. 26.0 26.6 29.3 35.3 42.7 CSFsimplist. 37.7 33.1 41.4 47.9 Table 2. Experiment 3 (top). Compensation of biodegradation versus chromatic adaptation in natural images. Performance in the goals and eventual human-like CSFs are described by εLMS and RMSE, respectively. CSFrealist. and CSFsimplist. refer to the way the CSF is computed (taking into account or neglecting the retinal degradation in the sinusoidal stimuli). Experiment 4 (bottom). Consistency under change of image statistics. The considered goals and magnitudes have the same meaning as in experiment 3. The only difference is in the scenes used to train the models. certainly more saturated, but beyond this obvious in Figure 4 (right), when considering two different fact, other differences may be subtle. In particular, functional goals: 1) pure reconstruction of the input we took precautions to get frames from a 5-hour signal, that is, compensation of the information loss compilation where backgrounds around the whole imposed by the bottleneck, and 2) compensation of chromaticity range appear not to bias the chromatic the bottleneck together with compensation of the CSFs. Regarding the spatial content, note that there biodistortion. Table 3 summarizes the distortions are plenty of edges of arbitrary orientations and also in the CSFs, RMSE, and the performance in the low-frequency transitions and shadows in the cartoons. reconstruction, εLMS . More radical modifications of spatial information (e.g., The Appendix C, Figure 20, confirms that these edit the cartoon images to make them isoluminant; i.e., architectures converged to a plateau of εLMS . Moreover, zero contrast in the achromatic channel) could lead to consistently with the data in Table 3, Figures 20 and substantial variation of the CSFs, but the goal of this 21 show that these systems achieve the computational illustration is to point out the robustness of the result goals to an extent that depends on the severity of the more than look for its limits. bottleneck in a very intuitive fashion (see comments in Appendix C). More interesting is what happens to the emerging Experiment 5: CSFs from bottleneck CSFs in Figure 10. In the absence of a bottleneck, compensation versus biodistortion pure reconstruction leads to wide filters equal in compensation the three chromatic channels, a clearly non-human result with RMSE approximately 40 (architectures Figure 10 shows the CSFs of the systems that emerge A and B). Similarly to pure chromatic adaptation, from the architectures with bottlenecks considered unconstrained pure reconstruction induces no spatial Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 21 Figure 10. Experiment 5: CSFs from architectures with bottlenecks. Notation of the architectures with bottlenecks (letters in blue) refers to the diagrams shown in Figure 4. (Top row) CSFs emerging from bottleneck compensation and biodistortion degradation in progressively more restrictive bottlenecks in a four-later architecture. (Middle row) CSFs emerging from pure reconstruction of the signal in the same architectures. (Bottom row) CSFs emerging in U-Nets from biodistortion compensation (left) or pure reconstruction (right). Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 22 No bottleneck Bottlenecks A B C D E F G U-Net Bio-Distort. εLMS 9.1 ± 0.3 9.2 ± 0.2 9.6 ± 0.1 9.8 ± 0.2 10.3 ± 0.4 11.2 ± 0.2 15.6 ± 0.5 12.3 ± 0.3 RMSE 25.4 25.5 24.6 25.8 23.8 30.5 33.0 25.3 Pure Recons. εLMS 1.2 ± 0.1 2.2 ± 0.1 4.8 ± 0.3 3.0 ± 0.5 12.8 ± 0.7 5.6 ± 0.8 11.9 ± 0.5 12.2 ± 0.6 RMSE 39.2 39.2 30.4 36.3 29.3 42.4 34.9 33.2 Table 3. Experiment 5: Compensation of biodistortion versus pure reconstruction in networks with bottlenecks. The bottlenecks in the architectures A through G and U-Net are described in Figure 4 (right). Performance in the reconstruction goals is measured by the εLMS error (in a test set) and the quality of the CSFs is given by the RMSE error. The considered biodistortion was the central case in Figure 3 with an original level of εLMS = 15.5, which is reduced to the values reported in the first row after the application of the models. In the pure reconstruction case the original retinal distortion is εLMS = 0 so the errors reported in the 3rd row come from poor reconstruction or an incomplete compensation of the bottleneck. selectivity and hence small similarity with human vision. Experimental human CSFs for achromatic moving Mild bottlenecks restricting the number of features gratings (Kelly, 1979), and for chromatic moving and/or the spatial resolution do introduce differences gratings (Díez-Ajenjo et al., 2011) are also included as in the bandwidth of the achromatic/chromatic a useful reference. The learning curves for the models channels, but the shape of the filters is far from and visual examples of reconstructions are given in human (RMSE ≥ 30 in architectures C and D). the Appendix C (Figure 22). Table 3 summarizes Then, more severe bottlenecks (architectures E–G the numerical performance of the models in this and U-Net) quickly leads to over-attenuation of experiment. one or both chromatic CSFs (and hence non-human The CSF results show that the main feature of behavior with a RMSE of approximately 35 in the spatiotemporal human window of visibility (its these architectures for reconstruction). On the diamond shape), with smaller spatial bandwidth other hand, the very same architectures trained for for higher temporal frequencies (or speeds) (Kelly, the compensation of biodistortion lead to more 1979; Watson, 2013) is reproduced by all the models human-like CSFs. See the band-pass/low-pass shape as well as the substantially lower bandwidth of the of the achromatic/chromatic CSFs and the RMSE of chromatic channels, focused on very low spatiotemporal approximately 25, except for architectures F and G that frequencies. The error of the best net is a RMSE of overattenuate the chromatic CSFs but still preserve the 17% of the maximum sensitivity. band-pass nature of the achromatic channel. Better Consistently with the results found in images preservation of chromatic CSFs by the systems tuned (experiment 1), resemblance with human CSFs is bigger to compensate the biodistortion is visually confirmed in shallower models (linear, two layers with a RMSE by the reconstructions of a representative image in of approximately 17% or 18%, respectively) than in Appendix C, Figure 21. deeper models (six layers, and eight layers with a RMSE In summary, pure reconstruction with the explored of approximately 22%) despite the performance of bottlenecks induces a difference between the relative the deeper models in the goal is substantially better bandwidths of the achromatic and chromatic CSFs. than the performance of the linear or the two layer However, the results become closer to human (both in model. The major differences are in the scaling of the the shape of the filters and in RMSE) when considering chromatic CSFs: note that deeper models over attenuate the compensation of the biodegradation of the retinal the chromatic patterns. The RMSE measures confirm signal. And this resemblance remains even if the system the superiority of the shallower solutions. For instance, is not constrained by a bottleneck. note that the over attenuation of the red–green channel in the CNNs implies that the greenish hue of the background in the visual example of Figure 22 fades Experiment 6: Spatiochromatic–temporal CSFs away, while it does not in the linear solution (which has from biodistortion compensation obvious problems in other respects). The linear solution cannot display a contrast Figure 11 shows the attenuation factors found for dependent behavior, but the two layer architecture low-contrast moving sinusoids (both achromatic and displays a consistent decay of the gain with contrast chromatic) in the plane ( fx , ft ) for a range of 3D that is in line with the saturating nature of contrast CNN autoencoders and for the optimal linear solution. response curves of humans for moving sinusoids Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 23 Figure 11. Experiment 6: Spatiotemporal chromatic CSFs from biodistortion compensation. The first row shows the human CSFs in ( fx , ft ), and the following show the model CSFs. The RMSE numbers (average over channels) represent the distance between them. In this experiment, α f = 1 in all cases. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 24 Figure 12. Experiment 6: Representative examples of nonlinear responses for spatiotemporal achromatic and chromatic gratings in shallow (top) and deeper (bottom) networks for biodistortion compensation. Comput. goal CSFs As in the image case, the deviations in the chromatic εLMS RMSE CSFs in deep models do not come from not fulfilling the goal or having poor convergence in the training. First, Distortion 5.2 ± 0.1 all models (even the linear one) do reduce the error Linear Net 3.5 ± 0.2 38.2 of the original retinal degradation so they are solving the computational problem. And second, the learning CNNs Sigmoid Sigmoid curves in the Appendix C (Figure 22) show that all nonlinear nonlinear models achieved a plateau in the training thus indicating proper convergence. Moreover, the asymptotic values 2-Layers 2.07 ± 0.06 41.6 achieved in the learning are consistent with the εLMS in 4-Layers 1.96 ± 0.07 45.5 test shown in Table 4. 6-Layers 1.83 ± 0.06 47.4 8-Layers 1.89 ± 0.07 47.5 Table 4. Experiment 6: Emergence of spatiotemporal chromatic CSF in 3D CNNs for compensation of biodistortion. The Discussion measures of the achievement of the goal εLMS and the distance with human behavior (RMSE difference between model and Summary of results human CSF) have the same meaning as in the rest of experiments. The degradation included in the movies was the In these experiments, we trained a range of CNN same as the considered for images (same pupil diameter and autoencoders over natural scenes to solve different Fano factor). However, the numerical εLMS deviations turned low-level vision goals: the compensation of retinal out to be lower because the considered movies are darker and distortions, the compensation of changes in the hence smaller LMS values lead to substantially smaller Poisson illumination, the compensation of information loss noise. after simple bottlenecks (or pure reconstruction after bottlenecks), and combinations of these. (Simoncelli & Heeger, 1998; Morgan et al., 2006). Following the analysis of linearized networks Figure 12 shows illustrative examples of these response presented in Gomez-Villa et al. (2020), it makes sense curves: Although the two layer network (top row) to stimulate these nets with achromatic, red–green consistently displays saturating behavior, the deeper net and yellow–blue isoluminant sinusoids and moving (bottom row) shows greater variability on the shape of sinusoids. The attenuation suffered by these gratings the response. shows that: Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 25 • Human-like CSFs may emerge in systems that • The 3D autoencoders for retinal degradation compensate retinal distortion: specifically, 2D compensation display a wide diamond-shaped shallow autoencoders trained to compensate achromatic bandwidth and very narrow chromatic retinal distortion display narrow low-pass behavior bandwidths in the spatiotemporal Fourier domain, in the chromatic channels and wider band-pass in parallel with humans. And this similarity is larger behavior in the achromatic channel, so the shape in the linear and shallow autoencoders (RMSE of and relative bandwidth of these artificial CSFs approximately 17% of the maximum sensitivity) resemble those of humans (Figures 5, 7, and 8). Of while it decays for deeper networks (RMSE of course the match is not complete: the best CSFs approximately 22% of the maximum sensitivity), obtained from the explored CNNs still deviate again owing to poor scaling of the chromatic CSFs from human CSFs (RMSE of approximately 11% (Figure 11). of the maximum sensitivity). Deeper autoencoders • The gain in shallow autoencoders decays with for the same goal also show CSFs with these basic contrast and hence the contrast responses for shapes but the resemblance with human CSFs is gratings saturate with contrast. This happens consistently lower (RMSE of approximately 15% both in the spatial and the spatiotemporal cases of the maximum sensitivity), particularly due to (Figures 6 and 12). This resembles contrast masking poor scaling of the chromatic CSFs (Figures 5 in humans. However, in deeper autoencoders this and 7). consistent saturation (and hence similarity with • Artificial CSFs obtained from the compensation humans) is not found. of retinal distortion differ from human CSFs in two qualitative aspects: a) The decay of network The emergence of human-like features in the CSFs sensitivity found at low frequencies for achromatic (distinct bandwidth and shape of achromatic and gratings is not as big as in humans, and b) The chromatic channels) is related to the different properties relative amplitude of the red–green and the of achromatic and chromatic patterns in visual scenes. yellow–blue CSFs in the networks is inverted The statistical unbalance towards achromatic patterns is with regard to the humans. In our networks, known from long ago in terms of variance (Ruderman the yellow–blue CSF is always bigger than the et al., 1998) and, more recently, it has been quantified red–green CSF, and interestingly, this is pretty in accurate information theoretic units (Malo, 2020). consistent over different architectures and datasets The eventual problems in preserving the saturation (or with different image statistics. poor scaling of chromatic CSFs) in deeper models, • Similar sensitivities consistently appear in shallow do not come from training. Note that, according to autoencoders for a range of levels in retinal the learning curves, all the models achieved proper distortions (Figure 8). convergence. On the contrary, the problems may come • Human-like CSFs with distinct bandwidths in from the small (statistical) relevance of chromatic achromatic/chromatic channels do not appear in textures as opposed to the achromatic textures and the pure chromatic adaptation tasks, but they do as inability of deeper models to deal with this unbalance soon as the retinal distortion compensation goal is with a low-level εLMS goal: (too) flexible networks considered (with or without chromatic adaptation). optimized to compensate the distortions focus (too The compensation of chromatic shifts together with much) on the spatial achromatic information to the compensation of biodistortion leads to systems optimize the goal and are likely to distort chromatic in which the chromatic CSFs change their global information. The consequence is a negative impact gain similarly to a Von-Kries mechanism (Figure 9, on the chromatic CSFs. This does not seem to be a top). problem for more rigid shallower architectures and even • CSFs emerging from chromatic adaptation and the linear solution. degradation compensation goals are similar for At this low abstraction level, where the minimization natural images and cartoon images (Figure 9, of distortion in LMS is simply connected with bottom). information maximization, and in the set of • Pure reconstruction in architectures with a architectures considered, shallow networks seem more restrictive bottleneck induces changes in the relative appropriate to explain the CSFs. bandwidths of the achromatic and chromatic CSFs with regard to trivial all-pass filters found in systems without bottleneck. However (in the explored Relation to other accounts of the CSFs cases), these CSFs are remarkably non-human. Interestingly, the very same architectures lead Our results revisit classical work on the statistical to more human-like CSFs as soon as the retinal grounds of the CSFs (Atick et al., 1992; Atick distortion compensation goal is considered & Redlich, 1992; Atick, 2011) in light of the new (Figure 10). possibilities provided by automatic differentiation. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 26 From the technical point of view, a number of low-level infomax+energy goal (Karklin & Simoncelli, assumptions that had to be done in the 1990s, either 2011), or deeper nets with higher level classification have been confirmed with the use of large data sets, goals (Lindsey et al., 2019). These sensors could induce or are not necessary with the use of more flexible CSF-like bandwidths in the corresponding models but models. In particular, regarding the signal, Atick this aspect was not addressed in these works. et al. assumed translation invariance, independence between color and space-time, and second-order relations (autocorrelation with 1/| f |2 decay). Moreover, Individual non-Euclidean distances from the regarding the model, they restricted themselves to linear optimization of average Euclidean distance solutions similar to Wiener filters. More recent studies with colorimetrically calibrated scenes (Gutmann An interesting consequence of our low-level result is et al., 2014) have confirmed the correctness of the that the Euclidean measure, εLMS , averaged over the color/space independence assumption. However, set of natural images leads to systems that measure the focus on the power spectrum and the linear individual differences in non-Euclidean ways. Note that solutions has proven to be too limited for denoising in the systems that we trained given two input signals x (Gutiérrez et al., 2006). Adaptive (nonlinear) models and x + x, the difference between the corresponding that take into account additional features of the signal responses is y ≈ M · x. As a network should assess are required. Nevertheless, the nonlinear networks the difference between the two signals from y, the considered here turn out to be roughly translation perceived difference for the system will depend on M. invariant (Gomez-Villa et al., 2020), as expected from Specifically (Epifanio et al., 2003; Laparra et al., 2010), their convolutional architectures and the stationary the perceived distance for the system will depend on the nature of the problems they face. Another technical metric, M · M, and hence it will depend on λ2 (the difference is in the formulation of the statistical goal: eigenspectrum of M) or, as seen here, on the CSF2 . This Atick et al. maximized the mutual information between metric is non-Euclidean: for instance, high-frequency the clean signal and the response I (xc , y); while here distortions will be less relevant for the network than we minimized εLMS between the clean signal and the medium or low-frequency distortions. Even though response, |X c − Y |2 . These goals are exactly equivalent here we did not check the correlation between the image when the difference between clean signal and the distortions perceived by humans and networks, the response is Gaussian, which is not the case in general. observed CSFs in the networks are consistent with the However, note that these goals are always related fact that the Euclidean distance between images at the because the limit |X c − Y |2 → 0 implies I (xc , y) → ∞. retina is not a good representation of human distortion Beyond the spatiochromatic case, in our work we check metrics (Wang & Bovik, 2009; Laparra et al., 2010; the emergence of the CSFs with spatiotemporal signals, Hepburn et al., 2020). which was mentioned but not addressed by Atick et al. The emergence of a non-Euclidean distance from Finally, the consideration of nonlinear models allows the minimization of the average Euclidean distance us to show that the error-minimization goal may also over natural images is a counterintuitive consequence lead to saturation of the contrast responses that, of of the highly nonuniform distribution of natural course, was not possible in the linear framework of scenes: distortions in less populated regions of the Atick et al. image space (e.g., in high-frequency directions or in As stated in the Introduction, other group has been chromatic channels) have to be under-rated to favor working independently on the emergence of CSFs in the average match to the data in highly populated or artificial neural networks (Akbarinia et al., 2021). Their more informative regions (low-frequency, achromatic results are restricted to the spatial CSF (no chromatic patterns). nor spatiotemporal cases) and are based on networks Recent work on autoencoders with low-level trained for higher level goals, such as classification. rate-distortion constraints on natural images has shown Therefore, their results from higher level goals are the emergence of non-Euclidean distances correlated complementary to ours, obtained from lower-level with human opinion of distortion (Hepburn et al., goals intended for the analysis of early visual stages 2022). Human opinion of distortion is known to be such as the LGN. More generally, higher level goals strongly mediated by the CSFs, but the bandwidth of such as classification performance may be an indirect this autoencoder and its eventual similarity with the way to impose preservation of colorfulness or a proper CSF was not explored in that work. scaling of the chromatic CSFs. Although chromatic information may have small relevance to minimize Alternative low-level computational goals εLMS in reconstruction, it may be more crucial for recognition. Here we considered different low-level alternatives to Other works have obtained center surround sensors the retinal signal enhancement goal proposed by Atick by optimizing a linear+nonlinear network with a et al.: although our results are conclusive regarding the Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 27 role of chromatic adaptation, more research is definitely subjective opinion of distortion. The autoencoders required about the relative relevance of bottlenecks in we presented here do not include constraints on shaping the CSFs. information, but the emergence of a non-Euclidean On the one hand, given the small role of spatial metric depending on M (and hence on the CSFs) information in the changes of the LMS image purely suggests that the distance will be correlated with human owing to changes in illumination, it is not surprising opinion in line with Hepburn et al. (2022). that systems designed for pure chromatic adaptation Alternative low-level goals could include non-human have wide (all-pass) behavior in all channels (i.e., no retinal degradation. Other species have different optical spatial effect). As a consequence (as confirmed by quality and noise in their retinas may be substantially our results), pure chromatic adaptation does not lead different. This may affect the kind of computations to CSFs with a human-like shape. This shape and required to extract the appropriate information from relative bandwidths have to be related to other goals this degraded input, and hence their corresponding (e.g., the compensation of biodistortion). However, CSFs. training for chromatic adaptation does introduce All these issues, the specific impact of more an important human-like behavior (which may not sophisticated bottlenecks in the CSFs, which was not emerge from other tasks): it leads to adaptive global analyzed here or in Karklin and Simoncelli (2011), scaling of the red–green or yellow–blue CSFs. This Lindsey et al. (2019), or Hepburn et al. (2022), the effect is consistent with the observations done on emergence of human-like image distortion measures spatiochromatic adaptation under changes in spectral from the enhancement of retinal signals, and the illumination (Gutmann et al., 2014): the spatial consideration of retinal degradation for other species, is structure of the receptive fields remains almost constant a matter for future research. but their chromatic tuning basically changes according to Von-Kries adaptation. On the other hand, as opposed to chromatic shifts, Goal and architecture are not independent the spatial effect of bottlenecks is relevant. However, More important than the technical generalizations we only explored a small range of architectural over Atick et al. (1992), Atick and Redlich (1992), and bottlenecks: the toy examples of Figure 4-right. In this Atick (2011), is that the current freedom to explore restricted set our results suggest that the compensation different linear and nonlinear architectures stresses of the biodegradation at the retina plays a stronger the relevance of the architectural constraints. The role in the emergence of human-like CSFs than the conventional interpretation of the efficient coding consideration of the bottlenecks. However, bottlenecks hypothesis (Barlow et al., 1961) is the following: in architectures C, D, and U-Net favor the emergence obtaining human-like results from certain statistical of nontrivial frequency selectivity. This would be goal seems to suggest that the human visual system consistent with Lindsey et al. (2019), who reported has been shaped by this goal. However, it is important positive effects of bottlenecks in the emergence of to realize that the results have been obtained via the center surround receptive fields. Nevertheless, the optimization of certain model. In the case of Atick specific configuration of the bottleneck that maximizes et al., it was a single model (the linear filter), but in our the human nature of the CSFs and the relative role case here we tried a range of models (architectures). of bottlenecks in the compensation of the retinal Because the results for the different architectures is distortion are interesting matters for further research. markedly different, the conclusion can not be about More generally, other low-level goals could be the goal, but about specific combinations of goal and considered together with the distortion, as for architecture. Our results are a specific illustration of the instance the information or the energy of the signal. fact that the computational and the algorithmic levels of Architectural bottlenecks considered here or in Lindsey analysis of visual processing systems (Marr & Poggio, et al. (2019) indirectly constrain the energy and the 1976; Marr, 1982) are not independent (Poggio, 2021). entropy of the signal by reducing the dimensionality of This dependence prevents about premature conclusions the signal. However, one could consider more general about the organization principles at the computational factors beyond the dimensionality as for instance the level if sensible architectures are not adopted. neural noise, the PDFs of signal and noise and the redundancy of the visual signals in the representation. In fact, transmitted information may be modulated Beyond accuracy by changes of the representation and by the amount of noise even without changes in the dimensionality Human-like CSFs are obtained for shallow (Malo, 2020). autoencoders (two layers), or even linear networks, In a separate study (Hepburn et al., 2022) we have despite deeper architectures achieving similar shown that rate-distortion bottlenecks in autoencoders or better performance in the goal. The previous induce distance measures which are correlated with literature has warned about the limitations of a single Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 28 accuracy/performance measure to identify human-like strong dependence of the CSFs on the architecture with behavior. Achieving similar performance on a task better results for shallower autoencoders (although they does not guarantee that two models actually use the have similar or lower performance in the goal). same strategy (Firestone, 2020). For instance, different This is not in contradiction with the literature cited strategies may become evident if performance degrades elsewhere in this article showing that deep networks in different ways when changing the experimental with object recognition goals match very well higher setting (Wichmann et al., 2017; Geirhos et al., 2019). visual areas. Note that the scope of our low-level goal Therefore, additional checks different from the is restricted to early visual stages (e.g., the retina–LGN optimization goal have to be done in order to confirm path), and hence simpler architectures may be required the human-like behavior of a model. Examples include there. verifying additional psychophysics not included in Beyond this difference in abstraction level, our the goal (Martinez et al., 2019), or disaggregating the results do illustrate the relevance of using appropriate results checking the consistency between model and architectures when checking a statistical goal. Following humans in individual trials, not on averages over the the move from conventional CNNs in Cadena et al. data set (Geirhos et al., 2020). (2019) to more realistic divisive normalization models In this complexity/accuracy discussion, it is in Burg et al. (2021), we think that future goal-driven important to stress that our results (shallower networks derivations of low-level visual psychophysics (e.g., better reproduce the scale of human chromatic CSFs) pattern masking or perceptual distortion) should is in line with the results of Gomez-Villa et al. (2020) include more realistic architectures too, as opposed and Flachot et al. (2020), which also show that to conventional CNNs (although they may be flexible shallower networks obtain more human-like colour enough to fulfill the goal). Examples include divisive representation. In a similar vein, although using higher normalization with parametric interaction between level classification goals, Kubilius et al. (2019) show that features (Martinez et al., 2018, 2019) and generalizations lower performance networks may correlate better with of Wilson–Cowan interactions (Bertalmío et al., 2020). human brain activity or psychophysics. Learning frameworks with rate-distortion bottlenecks are already available (Ballé et al., 2017; Hepburn et al., 2022), and we advocate for the study of their Final remarks artificial psychophysics using realistic and interpretable architectures. In visual neuroscience, deep models are emerging as Keywords: spatiotemporal and chromatic contrast the new standard to reproduce the activity of visual sensitivity, convolutional autoencoders, modulation areas under natural scene stimulation. On the one hand, transfer function, noisy cones, deblurring and denoising, conventional deep models driven by object recognition chromatic adaptation, natural images, statistical goals, goals reproduce the response from V1 (Güçlü and van architectures Gerven, 2015; Kriegeskorte, 2015), dorsal and ventral streams (Cichy et al., 2016), and IT (Cadieu et al., 2014; Yamins et al., 2014). On the other hand, deep Acknowledgments networks are powerful enough to fit the mappings between stimuli and measured responses (Prenger et al., Partially funded by these grants from 2004; Antolík et al., 2016; Batty et al., 2017). These GVA/AEI/FEDER/EU: MICINN DPI2017- two approaches (goal-driven and measurement-driven 89867-C2-2-R, MICINN PID2020-118071GB-I00, deep models) have been thoroughly compared in V1 and GVA Grisolía-P/2019/035 (for JM and QL), and and were found to be superior to linear filter-banks and MICINN PGC2018-099651-B-I00 (for A.G.V. and simple linear–nonlinear models (Cadena et al., 2019). M.B.). However, more recently, the same team has shown that linear–nonlinear models with general divisive Commercial relationships: none. normalization make a significant step towards the Corresponding author: Jesus Malo. performance of state-of-the-art CNN with interpretable Email: jesus.malo@uv.es. parameters (Burg et al., 2021). Address: Image Processing Lab, Building E4, Parc In our low-level goal-driven case, the emergence Cientific de la Universitat de Valencia, Carrer of human-like CSFs for certain CNN autoencoders Catedràtic Escardino, 46980 Paterna, Valencia, Spain. generalizes in different ways previous statistical explanations of the CSF based on linear models (Atick & Redlich, 1992; Atick et al., 1992; Atick, 2011), and Footnotes is consistent with optimizations of nonlinear encoders using alternative low-level (Karklin & Simoncelli, 2011; 1 For example, neurons in conventional CNNs have fixed nonlinearities, as Hepburn et al., 2022) or higher level (Lindsey et al., opposed to the known adaptive nature of real neurons (Wilson & Cowan, 2019; Akbarinia et al., 2021) goals. However, we find a 1973; Carandini & Heeger, 2012). Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 29 2 For optical blur where the linear operator H can be obtained Berardino, A., Ballé, J., Laparra, V., & Simoncelli, from the MTF (Watson, 2013), and the retinal noise is Poisson, E. (2017). Eigen-distortions of hierarchical nr = F · D 1 · n, where Dv is a diagonal matrix with vector (|H·x| ) 2 representations. In Proceedings of the Neural v in the diagonal, F is the Fano factor, and n is drawn from a unit-variance Gaussian (Esteve et al., 2020); the Jacobian in Equation 9, is Information Processing Systems, 30, pp. 3533– ∇x Sθ (0) = ∇x Nθ (0) · (I − F2 · D 1 ) · H, where the Jacobian of the 3542, https://papers.nips.cc/book/advances-in- (n |H·0| ) 2 network, ∇x Nθ (0), can be obtained analytically (Martinez et al., 2018). neural-information-processing-systems-30-2017. 3 We prepared the samples that way before actually knowing how well the Bertalmío, M., Gomez-Villa, A., Martín, A., Vazquez, networks are able to cope with this distortion. 4 In our computer cluster typical training of the 2D models takes J., Kane, D., & Malo, J. (2020). Evidence for the approximately 10 to 20 hours. intrinsically nonlinear nature of receptive fields in vision. Scientific Reports, 10, 16277. Burg, M., Cadena, S., Denfield, G., Walker, E., Tolias, A., & Bethge, M. et al. (2021). Learning divisive References normalization in primary visual cortex. PLoS Computational Biology, 17(6), e1009028. Abadi, M., Agarwal, A., Barham, P., Brevdo, Cadena, S., Denfield, G., Walker, E., Gatys, L., Tolias, E., Chen, Z., Citro, C., . . . Zheng, X. (2016). A., & Bethge, M. et al. (2019). Deep convolutional TensorFlow: Large-scale machine learning on models improve predictions of macaque V1 heterogeneous distributed systems. arxiv Comp. Sci, responses to natural images. PLoS Computational https://arxiv.org/abs/1603.04467. Biology, 15(4), e1006897. Akbarinia, A., Morgenstern, Y., & Gegenfurtner, K. Cadieu, C., Hong, H., Yamins, D., Pinto, N., Ardila, D., R. (2021). Contrast sensitivity is formed by visual & Solomon, E. et al. (2014). Deep neural networks experience and task demands. Journal of Vision, rival the representation of primate it cortex for 21(9), 1996. core visual object recognition. PLoS Computational Antolík, J., Hofer, S. B., Bednar, J. A., & Mrsic-Flogel, Biology, 10(12), e1003963. T. D. (2016). Model constrained by visual hierarchy Cai, D., DeAngelis, G. C., & Freeman, R. D. (1997). improves prediction of neural responses to natural Spatiotemporal receptive field organization in scenes. PLoS Computational Biology, 12(6), the lateral geniculate nucleus of cats and kittens. e1004927, doi:10.1371/journal.pcbi.1004927. Journal of Neurophysiology, 78(2), 1045–1061. Atick, J., Li, Z., & Redlich, A. (1992). Understanding Campbell, F., & Robson, J. (1968). Application of retinal color coding from first principles. Neural Fourier analysis to the visibility of gratings. Journal Computation, 4(4), 559–572. of Physiology, 197, 551–566. Atick, J., & Redlich, A. (1992). What does the retina Carandini, M., & Heeger, D. (2012). Normalization as know about natural scenes? Neural Computation, a canonical neural computation. Nature Reviews 4(2), 196–210. Neuroscience, 13(1), 51–62. Atick, J. J. (2011). Could information theory provide an Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A., ecological theory of sensory processing? Network: & Oliva, A. (2016). Comparison of deep neural Computation in Neural Systems, 22, 4–44. networks to spatio-temporal cortical dynamics Ballé, J., Laparra, V., & Simoncelli, E. P. (2017). End- of human visual object recognition reveals to-end optimized image compression. International hierarchical correspondence. Scientific Reports, 6, Conference on Learning Representations (ICLR), 27755. https://openreview.net/forum?id=rJxdQ3jeg. Clarke, R. (1981). Relation between the Karhunen Barlow, H. B. et al. (1961). Possible principles Loève and cosine transforms. IEE Proceedings F underlying the transformation of sensory messages. Communications, Radar and Signal Processing, 128, Sensory Communication, 1, 217–234. 359–361. Batty, E., Merel, J., Brackbill, N., Heitman, A., Cottaris, N. P., Jiang, H., Ding, X., Wandell, B. A., & Sher, A., Litke, A. M., . . . Paninski, L. (2017). Brainard, D. H. (2019). A computational-observer Multilayer recurrent network models of primate model of spatial contrast sensitivity: Effects of retinal ganglion cell responses. 5th International wave-front-based optics, cone-mosaic structure, and Conference on Learning Representations (ICLR), inference engine. Journal of Vision, 19(4), 8. https://openreview.net/forum?id=9gmuVOlKfLa. Cottaris, N. P., Wandell, B. A., Rieke, F., & Brainard, Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & D. H. (2020). A computational observer model of Siskind, J. M. (2018). Automatic differentiation in spatial contrast sensitivity: Effects of photocurrent machine learning: A survey. Journal of Machine encoding, fixational eye movements, and inference Learning Research, 18(153), 1–43. engine. Journal of Vision, 20, 17. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 30 Donen, S. (1963). Charade. Universal Studios, CC Güçlü, U., & van Gerven, M. A. J. (2015). Deep neural Public Domain Mark 1.0, https://archive.org/ networks reveal a gradient in the complexity of details/Charade19631280x696. neural representations across the ventral stream. Díez-Ajenjo, M., Capilla, P., & Luque, M. J. (2011). Journal of Neuroscience, 35(27), 10005–10014. Red-green vs. blue-yellow spatio-temporal contrast Gutiérrez, J., Ferri, F. J., & Malo, J. (2006). sensitivity across the visual field. Journal of Modern Regularization operators for natural images based Optics, 58, 1–13. on nonlinear perception models. IEEE Transactions Enroth-Cugell, C., & Robson, J. (1966). The contrast on Image Processing, 15(1), 189–200. sensitivity of retinal ganglion cells on the cat. Gutmann, M. U., Laparra, V., Hyvärinen, A., & Journal of Physiology (London), 187, 516–552. Malo, J. (2014). Spatio-chromatic adaptation via Epifanio, I., Gutierrez, J., & Malo, J. (2003). Linear higher-order canonical correlation analysis of transform for simultaneous diagonalization of natural images. PloS One, 9(2), e86481. covariance and perceptual metric matrix in image Hepburn, A., Laparra, V., Malo, J., & Santos, R. (2020). coding. Pattern Recognition, 36(8), 1799–1811. Perceptnet: A human visual system inspired neural Esteve, J., Aguilar, G., Maertens, M., Wichmann, F., & network for estimating perceptual distance. In Malo, J. (2020). Psychophysical estimation of early Proceedings of the IEEE International Conference and late noise. Arxiv: Quantitative Biology, 1–15, on Image Processing (ICIP) (pp. 121–125), https://arxiv.org/abs/2012.06608. doi:10.1109/ICIP40778.2020.9190691. Firestone, C. (2020). Performance vs. competence in Hepburn, A., Laparra, V., Santos, R., Ballé, J., & human–machine comparisons. Proceedings of the Malo, J. (2022). On the relation between statistical National Academy of Sciences of the United States learning and perceptual distances. International of America, 117(43), 26562–26571. Conference on Learning Representations, ICLR, https://openreview.net/forum?id=zXM0b4hi5_B. Flachot, A., Akbarinia, A., Schütt, H. H., Fleming, R. W., Wichmann, F. A., & Gegenfurtner, K. R. Hunt, B. R. (1975). Digital image processing. (2020). Deep neural models for color discrimination Proceedings of the IEEE, 63, 693–708. and color constancy. CoRR, abs/2012.14402. Hurvich, L., & Jameson, D. (1957). An opponent- Freleng, I. (1963). The Pink Panther Show. Los Angeles, process theory of color vision. Psychological CA, USA: DePatie-Freleng Enterprises (DFE Review, 64(6), 384–404. Films), courtesy of Metro Goldwin Mayer. Hyvärinen, A., Hurri, J., & Hoyer, P., 2009. Natural Geirhos, R., Meding, K., & Wichmann, F. A. (2020). image statistics: A probabilistic approach to early Beyond accuracy: Quantifying trial-by-trial computational vision. Heidelberg, Germany: behaviour of CNNs and humans by measuring Springer-Verlag. error consistency. arXiv: 2006.16736. Ingling, C. R., & Martinez-Uriegas, E. (1983). The Geirhos, R., Rubisch, P., Michaelis, C., Bethge, relationship between spectral sensitivity and spatial M., Wichmann, F. A., & Brendel, W. (2019). sensitivity for the primate r-g x-channel. Vision Imagenet-trained CNNs are biased towards Research, 23, 1495–1500. texture; increasing shape bias improves Karklin, Y., & Simoncelli, E. (2011). Efficient coding accuracy and robustness. International of natural images with a population of noisy Conference on Learning Representations (ICLR), linear-nonlinear neurons. In Proceedings of the https://openreview.net/forum?id=Bygh9j09KX. Advances in Neural Information Processing Systems, Gomez-Villa, A., Martin, A., Vazquez, J., & Bertalmio, 24, https://proceedings.neurips.cc/paper/2011/file/ M. (2019). Convolutional neural networks can be 12e59a33dea1bf0630f46edfe13d6ea2-Paper.pdf. deceived by visual illusions. Proceedings of the Kelly, D. H. (1979). Motion and vision. ii. stabilized IEEE/CVF Conference on Computer Vision and spatio-temporal threshold surface. Journal of the Pattern Recognition (CVPR 19) (pp. 12309–12317), Optical Society of America, 69(10), 1340–1349. doi:10.1109/CVPR.2019.01259. Kingma, D. P., & Ba, J. (2017). Adam: A method for Gomez-Villa, A., Martin, A., Vazquez, J., Bertalmío, stochastic optimization. arXiv:1412.6980. M., & Malo, J. (2020). Color illusions also Kriegeskorte, N. (2015). Deep neural networks: A new deceive CNNs for low-level vision tasks: framework for modeling biological vision and brain Analysis and implications. Vision Research, 176, information processing. Annual Review of Vision 156–174. Science, 1(1), 417–446. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Kubilius, J., Schrimpf, M., Kar, K., Rajalingham, R., Deep Learning. Boston, MA, USA: MIT Press, Hong, H., Majaj, N., . . . DiCarlo, J. J. (2019). https://www.deeplearningbook.org. Brain-like object recognition with high-performing Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 31 shallow recurrent ANNs. In Proceedings Advances image databases in modeling vision. Frontiers in in Neural Information Processing Systems, 32, Neuroscience, doi:10.3389/fnins.2019.00008. https://proceedings.neurips.cc/paper/2019/file/ Martinez, M., Cyriac, P., Batard, T., Bertalmío, M., & 7813d1590d28a7dd372ad54b5d29d033-Paper.pdf. Malo, J. (2018). Derivatives and inverse of cascaded Laparra, V., Muñoz, J., & Malo, J. (2010). Divisive linear+nonlinear neural models. PLoS One, 13(10), normalization image quality metric revisited. doi:10.1371/journal.pone.0201326. Journal of the Optical Society of America A, 27(4), Martinez-Otero, L., Molano, M., Wang, X., Sommer, 852–864. F., & Hirsch, J. (2014). Statistical wiring of thalamic Legge, G. E. (1981). A power law for contrast receptive fields optimizes spatial sampling of the discrimination. Vision Research, 21(4), retinal image. Neuron, 81(4), 943–956. 457–467. Martinez-Uriegas, E. (1994). Chromatic-achromatic Legge, G. E., & Foley, J. M. (1980). Contrast masking multiplexing in human color vision. In D. H. Kelly in human vision. Journal of the Optical Society of (Ed.), (pp. 117–187). Boca Ratón, FL, USA: CRC America, 70(12), 1458–1471. Press. LeRoy, M. (1959). The FBI story. Los Angeles, CA, Martinez-Uriegas, E. (1997). Color detection and color USA: Warner Brothers/Seven Arts. contrast discrimination thresholds. In Proceedings Lillicrap, T., Santoro, A., Marris, L., Akerman, C., & of the OSA Annual Meeting ILS-XIII, p. 81. Hinton, G. (2020). Backpropagation and the brain. Morgan, M., Chubb, C., & Solomon, J. (2006). Nature Reviews Neuroscience, 21(6), 335–346. Predicting the motion after-effect from sensitivity Lindsey, J., Ocko, S. A., Ganguli, S., & Deny, S. loss. Vision Research, 46(15), 2412–2420. (2019). The effects of neural resource constraints Mullen, K. T. (1985). The CSF of human colour vision on early visual representations. International to red–green and yellow–blue chromatic gratings. Conference on Learning Representations, ICLR, Journal of Physiology, 359, 381–400. https://openreview.net/forum?id=S1xq3oR5tQ. van den Oord, A., & Schrauwen, B. (2014). The Malo, J. (2020). Spatio-chromatic information available student-t mixture as a natural image patch prior from different neural layers via gaussianization. with application to image compression. Journal of Journal of Mathematical Neuroscience, 10(18), Machine Learning Research, 15(60), 2061–2086. doi:10.1186/s13408-020-00095-8. Poggio, T. (2021). From Marr’s Vision to the Malo, J. (2022). Paraphrasing Magritte’s observation. problem of human intelligence (CBMM Memo ArXiV: Computer Vision and Pattern Recognition, No. 118). Boston, MA, USA: MIT Libraries, 1–4, https://arxiv.org/abs/2202.08103. https://dspace.mit.edu/handle/1721.1/131234. Malo, J., & Luque, M. (2002). ColorLab: A Matlab Prenger, R., Wu, M., David, S., & Gallant, J. (2004). Toolbox for color science and calibrated color Nonlinear v1 responses to natural scenes revealed image processing. Servei de Publicacions de by neural network analysis. Neural Networks, la Universitat de Valencia. Valencia, Spain, 17(5-6), 663–79. https://isp.uv.es/code/visioncolor/colorlab.html. Reid, R. C., & Shapley, R. (1992). Spatial structure of Malo, J., Pons, A., Felipe, A., & Artigas, J. (1997). cone inputs to receptive fields in primate lateral Characterization of the human visual system geniculate nucleus. Nature, 356, 716–718. threshold performance by a weighting function in Reid, R. C., & Shapley, R. (2002). Space and time maps the gabor domain. Journal of Modern Optics, 44(1), of cone photoreceptor signals in macaque lateral 127–148. geniculate nucleus. Journal of Neuroscience, 22, Mannos, J. L., & Sakrison, D. J. (1974). The effects 6158–6175. of a visual fidelity criterion of the encoding of Ruderman, D. L., Cronin, T. W., & Chiao, C.-C. (1998). images. IEEE Transactions on Information Theory, Statistics of cone responses to natural images: 20, 525–536. Implications for visual coding. Journal of the Marr, D. (1982). Vision: A computational approach. San Optical Society of America A, 15(8), 2036–2045. Francisco: Freeman & Co. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, Marr, D., & Poggio, T. (1976). From understanding S., & Ma, S. et al. (2015). Imagenet large scale computation to understanding neural circuitry (AI visual recognition challenge. International Journal Memo No. AIM-357). Boston, MA, USA: MIT of Computer Vision, 115, 211–252. Libraries, http://hdl.handle.net/1721.1/5782. Simoncelli, E., & Heeger, D. (1998). A model of Martinez, M., Bertalmío, M., & Malo, J. (2019). In neuronal reponses in visual area MT. Vision praise of artifice reloaded: Caution with natural Research, 38(5), 743–761. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 32 Simoncelli, E. P., & Olshausen, B. A. (2001). Natural Welles, O. (1946). The stranger. New York, NY, USA: image statistics and neural representation. Annual RKO Radio Picures. Review of Neuroscience, 24(1), 1193–1216. Wichmann, F. A., Janssen, D. H. J., Geirhos, R., Soh, J., & Cho, N. (2021). Deep universal blind image Aguilar, G., Schütt, H. H., Maertens, M., . . . denoising. In Proceedings International Conference Bethge, M. (2017). Methods and measurements to on Pattern Recognition (ICPR) (pp. 747–754), compare men against machines. Electronic Imaging, doi:10.1109/ICPR48806.2021.9412605. 10, 36–45. Stockman, A., & Sharpe, L. T. (2000). The spectral Wilson, H. R., & Cowan, J. D. (1973). A mathematical sensitivities of the middle- and long-wavelength- theory of the functional dynamics of cortical and sensitive cones derived from measurements in thalamic nervous tissue. Kybernetik, 13(2), 55–80. observers of known genotype. Vision Research, Wuerger, S. M., Ashraf, M., Kim, M., Martinovic, 40(13), 1711–1737. J., Pérez-Ortiz, M., & Mantiuk, R. K. (2020). Tao, X., Gao, H., Shen, X., Wang, J., & Jia, J. (2018). Spatio-chromatic contrast sensitivity under mesopic Scale-recurrent network for deep image deblurring. and photopic light levels. Journal of Vision, 20, 23, In Proceedings of the IEEE Conference on Computer https://doi.org/10.1167/jov.20.4.23. Vision and Pattern Recognition (pp. 8174–8182), Yamins, D., & DiCarlo, J. (2016). Using goal-driven doi:10.1109/CVPR.2018.00853. deep learning models to understand sensory cortex. Taubman, D. S., & Marcellin, M. W. (2001). Jpeg Nature Neuroscience, 19, 356–365. 2000: Image compression fundamentals, standards Yamins, D., Hong, H., Cadieu, C. F., Solomon, and practice. Norwell, MA: Kluwer Academic E. A., Seibert, D., & DiCarlo, J. J. (2014). Publishers. Performance-optimized hierarchical models predict Valois, R. L. de, & Pease, P. L. (1971). Contours and neural responses in higher visual cortex. Proceedings contrast: Responses of monkey lateral geniculate of the National Academy of Sciences of the United nucleus cells to luminance and color figures. State of America, 111, 8619–8624. Science, 171, 694–696. Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, Wallace, G. K. (1992). The JPEG still picture L. (2017). Beyond a gaussian denoiser: Residual compression standard. IEEE Transactions learning of deep CNN for image denoising. IEEE on Consumer Electronics, 38(1), xviii–xxxiv, Transactions on Image Processing, 26(7), 3142–3155. doi:10.1109/30.125072. Wang, Z., & Bovik, A. C. (2009). Mean squared error: Love it or leave it? A new look at signal fidelity measures. IEEE Signal Processing Magazine, 26(1), Appendix A: Implementation 98–117. details Watson, A. B. (2013). High frame rates and human vision: A view through the window of visibility. As stated in the main text, all the models follow the SMPTE Motion Imaging Journal, 122(2), the basic toy networks studied in Gomez-Villa et al. 18–32. (2019, 2020): autoencoders with convolutional layers Watson, A. B., & Ahumada, A. J. (2005). A standard made of eight feature maps with kernels of spatial size model for foveal detection of spatial contrast. 5 × 5 and sigmoids or ReLUs as activation functions. Journal of Vision, 5(9), 717–740. As illustrated in Figure 4, the last reconstruction Watson, A. B., & Ahumada, A. J. (2016). The layer, has three features in every case (the three color pyramid of visibility. In Proceedings Human channels) so that the input and output domains are Vision and Electronic Imaging HVEI16, 102, 1–6, the same. Following our purpose of using biologically doi:10.2352/ISSN.2470-1173.2016.16HVEI-102. plausible image representations the input to the networks and the output signals are expressed in the Watson, A. B., Ahumada, A. J., & Farrell, J. E. (1986). LMS color space (as opposed to generic RGB digital Window of visibility: A psychophysical theory of counts used in the cited references). fidelity in time-sampled visual motion displays. The spatiotemporal models follow the same spirit, Journal of The Optical Society of America A-optics in this case also including convolution in the temporal Image Science and Vision, 3, 300–307. dimension: autoencoders with 3D convolutional Watson, A. B., & Malo, J. (2002). Video quality layers made of eight feature maps with kernels of size measures based on the standard spatial observer. In 5 × 5 × 5 and sigmoid activation functions (we did not Proceedings of the IEEE International Conference explore ReLU in videos because in images we found no on Image Processing (Vol. III, pp. 41–44), qualitative difference between the ReLU and sigmoid doi:10.1109/ICIP.2002.1038898. results). As in the image case, the last (reconstruction) Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 33 layer for every architecture (two, four, six and Show (Freleng, 1963) reproduced with permission of eight layers) only has three feature maps (the LMS the MGM. In every case we take 128 × 128 images and channels). assume a sampling frequency f s = 70 cpd, that is, we Implementation and training is done in the same way assume that the images subtend 3.6°. as in Gomez-Villa et al. (2019, 2020): mean squared The spatiotemporal models are trained over 25 · 103 error is used as loss function in all cases and all the patches of size 32 × 32 × 25 from classical Hollywood models are implemented using Tensorflow (Abadi et al., films which are in public domain: the color movies 2016). We train our models using ADAM stochastic Charade (Donen, 1963) and The FBI story (LeRoy, gradient descent (Kingma & Ba, 2017) with a batch 1959), and the achromatic movie The Stranger (Welles, size of 32 examples, momentum of 0.9, and a weight 1946). In all the video cases we assume a spatial decay of 0.001. In principle, a standard early stopping sampling of 30 cpd and temporal sampling of 25 Hz, criterion for convergence was used based on the number that is, we assume the patches subtend 1.06° and last of iterations with no improvement in the validation for 1 second. These somewhat arbitrary selections of set. However, to ensure appropriate convergence, we the sampling frequencies (or extent of the stimuli) have visualized the learning curves and we let the iteration mild consequences on the quantitative evaluation of the continue until train and validation error reached a CSFs as discussed elsewhere in this appendix. common plateau. All the learning curves are explicitly The transform from digital counts to LMS shown in the Appendix C, these show that all the CSF tristimulus values was done assuming the primaries and considered in the main text come from models with the gamma curves of a standard CRT display (Malo & proper convergence. Luque, 2002). Regarding the stimuli for the estimation of the CSFs according to Equation 3, our b f are gratings in the Appendix B: Training stimuli and classical opponent space of Hurvich and Jameson stimuli for CSF estimation (1957). Figure 13 shows a representative subset of the gratings used to feed the networks for the estimation of the spatiochromatic CSFs. The justification of the The natural stimuli to train the networks are regular use of these waves to probe the autoencoders follows photographic images from the same dataset used in the eigenanalysis of the linearized networks introduced Gomez-Villa et al. (2019, 2020): the Large Scale Visual in Gomez-Villa et al. (2020): the eigenfunctions of the Recognition Challenge, 2014 CLS-LOC validation matrices in Equation 8 were shown to be oscillating dataset (which contains 50 ·103 images), leaving 10 ·103 functions in space with chromatic variations in images for validation purposes. This dataset is a subset luminance and opponent red–green and yellow–blue of the whole ImageNet dataset (Russakovsky et al., directions. Consistently with Gomez-Villa et al. (2020) 2015). The experiments with cartoon images were done the corresponding spatiotemporal oscillations of using 25 · 103 frames taken from The Pink Panther increasing frequency for decreasing eigenvalue are Figure 13. Representative spatiochromatic stimuli to feed the 2D networks. The 3D networks were probed with equivalent gratings mooving at different speeds (see text). Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 34 obtained when the considered Jacobian corresponds to achieving the functional goal it has been designed for. spatiotemporal autoencoders. The plateau achieved by the blue curves (not only in The full spatiochromatic set included gratings of 60 training but, more significantly in validation) implies spatial frequencies linearly spaced in the range [0.5, 35] that a steady convergence was achieved and the resulting cpd (the Nyquist region assuming fs = 70 cpd), and model is ready to be tested. Consistency between the nine contrasts linearly spaced in the range (0.07, 0.6). train and validation sets is apparent from the parallel The average color was the white of the color system behavior of the light and dark curves. Performance with 30 cd/m2 . The stimuli for the computation of tables in the main text (Table 1 for experiment 1) the spatiotemporal CSFs were moving sinusoids with and performance in the visual examples shown here 16 spatial frequencies in the range of 0 to 15 cpd, 10 (Figure 14 for experiment 1) refer to an independent temporal frequencies in the range or 0 to 10 Hz, and test set not used in the learning (training/validation) 9 contrasts in the range of 0.07 to 0.6. The average phase. In all CNNs used in experiment 1, the training color and luminance were the same as in the image has been done in a representative set because the errors case. in the independent test phase (Table 1 and Figure 14) The lower limit of the explored contrast range are consistent with the asymptotic behavior of the comes from the limitation owing to noise discussed learning curves shown in Figure 14. Figure 15 shows in Methods: Estimating contrast sensitivity in visual examples of the performance of the linearized autoencoders. The upper contrast limit and the average versions of the nonlinear models in experiment 1. It luminance were selected to ensure that corrupted signals is interesting to note that the optimal linear solution are reproducible in regular displays (which is the range (computed from the train set) has worse behavior than of the scenes used in the training). the linearized versions of the networks (as also seen in Table 1). Appendix C: Convergence and Experiment 2: Architecture trained on a range Performance of the models of distortion levels Figure 16 shows the learning curves of the model To guarantee that the presented CSF results trained in experiment 2 (two layer ReLU) for different do not come from eventual training artifacts, this levels of retinal degradation (noise/blur). Note that the appendix illustrates the proper convergence and proper specific cases where the iteration was stopped owing performance of all the considered CNNs in all the to the activation of the early stopping criterion (top goal/architecture scenarios. For each considered case left and bottom center), the convergence plateau was in the Experiments, we show the learning curves and already reached. Figure 17 shows examples of the explicit examples of the responses (reconstructed performance in test of the model considered in every signals in test). Finally, as an illustrative example, we training scenario considered in experiment 2. also show one extra case (for video, experiment 6) where convergence was not complete in one of the networks and the consequences in the reconstructions and in the CSFs. Experiment 3: Chromatic adaptation versus distortion compensation Experiment 1: Distortion compensation from a Figure 18 (top) demonstrates that the model range of CNN architectures trained for the five computational goals considered in experiment 3 actually achieves the goals and has proper Figure 14 shows the learning curves of all the CNN convergence. Figure 18 (bottom) shows visual examples models used in experiment 1. Throughout the Appendix of the performance. the gray/black curves refer to the εLMS distortion of the retinal signal in the LMS color space of Stockman and Sharpe (2000), which is constant along the learning. Experiment 4: Robustness under change of The cyan–blue curves show the evolution of the error signal statistics in the response of the networks. The light color curve describes the error over image batches in the training Figure 19 (top) demonstrates that the model phase while the dark color describes the same error in trained for the five computational goals considered in the validation set. The error of the response (solution) experiment 4 actually achieves the goals and has proper significantly drops below the error of the input signal convergence. Figure 19 (bottom) shows visual examples (problem), thus indicating that the network is actually of the performance. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 35 Figure 14. Experiment 1: Convergence and trained models. Learning curves (training/validation) and examples of visual performance (test) of all the models trained in experiment 1. The distortion εLMS -Problem refers to the original degradation of the images (previous to the application of the net). This distortion describes how difficult the compensation problem is. The distortion εLMS -solution refers to the degradation remaining in the signal after the application of the net. It describes how close the output is to the ideal result. Performance numbers have been truncated to the significance of the standard deviation in the test set. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 36 Experiment 5: CSFs from bottleneck compensation and biodistortion compensation Figure 20 demonstrates that the models trained for the goals considered in experiment 5 have proper convergence and actually achieve the goals within the constraints imposed by the bottlenecks of progressive severity. Figure 21 shows visual examples of the performance. The reconstruction error εLMS behaves quite intuitively (see Figure 20 here and Table 3 in the main text): On the one hand, in the cases that involve biodistortion all the architectures do reduce the original value of εLMS except the architecture G, which has a single feature in its bottleneck. On the other hand, the pure reconstructions cases introduce negligible distortion εLMS when the inner representation does not restrict spatial resolution nor number of features (the no-bottleneck cases A and B). And, as expected, more severe bottlenecks imply higher εLMS . Experiment 6: Spatiotemporal temporal CSFs The main text includes the CSF results from a range of models trained in Charade (1963). Figure 22 shows the regular evidences on convergence (top) and visual performance in test (bottom) shown for the other Experiments. In this Appendix, we also include a replication of experiment 6 trained on a movie with higher spatial resolution (The FBI Story, 1959). We give the corresponding learning curves (Figure 23) and CSFs (Figure 24). This is interesting for two reasons: 1) it confirms the superiority of shallower nets even for different resolution, and 2) it shows an example of failure in convergence (see that the eight layer model in Figure 23 got stuck in a local minimum (with poor performance) and this has consequences in the complete loss of chromatic information (frame not shown because of copyright issues). Also interesting is the fact that models with four or six layers (which converged as well as the two layer model), substantially over attenuate the red–green channel with the corresponding yellowish–bluish look of the reconstruction and the corresponding impact on the relative scaling of channels in the CSFs, which is not the case for the linear and the 2-layer solutions. This is consistent in the other image/video examples in the main text. Figure 15. Experiment 1: linearized models. Examples of visual performance (in test) for the linearized CNNs of experiment 1. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 37 Figure 16. Experiment 2: Convergence and performance. Learning curves (training/validation) and numerical performance (in test and in the reproduction of the CSFs) of the CNN model trained in all conditions considered in experiment 2. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 38 Figure 17. Experiment 2: Visual performance. Examples of reconstruction (in test) for the CNN in all the degradation scenarios considered in experiment 2. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 39 Figure 18. Experiment 3: Convergence and visual performance: Top, learning curves (train/validation) for the considered architecture in the different goals. Bottom: Visual example (test) for the CNNs in experiment 3 (natural images). Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 40 Figure 19. Experiment 4: Convergence and visual performance: Top, learning curves (train/validation) for the considered architecture in the different goals. Bottom: Visual example (test) for the CNNs in experiment 4 (cartoon images). Original image from The Pink Panther Show (Freleng, 1963) courtesy of MGM. Similar images (Malo, 2022) lead to equivalent performance in the networks. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 41 Figure 20. Experiment 5: Bottleneck compensation and biodistortion compensation. Learning curves (train/validation) for the considered architectures in reconstruction with compensation of biodistortion (top row and bottom row, left) and pure reconstruction (middle row and bottom row, right). The cases including biodistortion show the original εLMS of the problem. See Figure 4 for the structure of the architectures referred by the letters in blue. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 42 Figure 21. Experiment 5: Bottleneck compensation and biodistortion compensation. Examples of visual performance (in test) for the CNNs of experiment 5. See Figure 4 for the structure of the architectures referred by the letters in blue. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 43 Figure 22. Experiment 6 (spatiotemporal chromatic CSFs in main text): Convergence and visual performance. Top, learning curves for the considered architectures. Visual example (test) for the linear solution and the CNNs (Charade, low-resolution movie). The original frame comes from the film Charade (Donen, 1963), which is in the public domain. Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 44 ← The FBI story; LeRoy, 1959). All models converge except the 8-layer architecture. Visual results are not shown due to copyright issues. Interested readers can obtain these specific results from the authors. The local minimum in which the eight layer architecture was trapped has consequences in the complete loss of chromatic information (frame not shown because of copyright issues). Also interesting is the fact that models with four and six layers (which converged as well as the two layer model), substantially over attenuate the red–green channel with the corresponding yellowish–bluish look of the reconstruction and the corresponding impact on the relative scaling of channels in the CSFs, which is not the case for the linear and the two layer solutions. Figure 23. Extended experiment 6: Spatiotemporal chromatic CSFs with different training set (not shown in the main text): Convergence. Figure shows the learning curves for the considered architectures on a higher resolution movie (the film → Journal of Vision (2022) 22(6):8, 1–45 Li, Gomez-Villa, Bertalmío, & Malo 45 Figure 24. Extended experiment 6 (spatiotemporal chromatic CSFs of a high-resolution movie not shown in the main text). Note that in this case (see Figure 23) all models converged except the eight layer architecture, which totally removes the yellow–blue channel and almost removed the red–green channel.