This is a preprint of: “Periodic orbits and equilibria for a seventh-order generalized Hénon-Heiles Hamiltonian system”, Jaume Llibre, Tareq Saeed, Euaggelos E. Zotos, J. Geom. Phys., vol. 167, 104290:1–12, 2021. DOI: [10.1016/j.geomphys.2021.104290] Periodic orbits and equilibria for a seventh-order generalized Hénon-Heiles Hamiltonian system Jaume Llibrea , Tareq Saeedb , Euaggelos E. Zotosc,∗ a Departament de Matemàtiques, Universitat Autònoma de Barcelona, 08193 Bellaterra, Barcelona, Catalonia, Spain b Nonlinear Analysis and Applied Mathematics (NAAM)-Research Group, Department of Mathematics, Faculty of Science, King Abdulaziz University, P.O. Box 80203, Jeddah 21589, Saudi Arabia c Department of Physics, School of Science, Aristotle University of Thessaloniki, GR-541 24, Thessaloniki, Greece Abstract In this paper we study analytically the existence of two families of periodic orbits using the averaging theory of second order, and the finite and infinite equilibria of a generalized Hénon-Heiles Hamiltonian system which includes the classical Hénon-Heiles Hamiltonian. Moreover we show that this generalized Hénon- Heiles Hamiltonian system is not C 1 integrable in the sense of Liouville–Arnol’d, i.e. it has not a second C 1 first integral independent with the Hamiltonian. The techniques that we use for obtaining analytically the periodic orbits and the non C 1 Liouville-Arnol’d integrability, can be applied to Hamiltonian systems with an arbitrary number of degrees of freedom. Keywords: eneralized Hénon-Heiles potential – Finite equilibria – Infinite equilibria 1. Introduction and statement of results The classical Hénon-Heiles Hamiltonian consist of a two dimensional harmonic potential plus two cubic terms,i.e. 1 2 y3 H= (px + p2y + x2 + y 2 ) + x2 y − . 2 3 This Hamiltonian was introduced in 1964, it is a model for studying the existence of a third integral of motion of a star in an rotating meridian plane of a galaxy in the neighborhood of a circular orbit [1]. The generalized Hénon-Heiles Hamiltonian system here studied is  1 2 1 y3 Hε = (px + p2y ) + (x2 + y 2 ) + x2 y − + ε x6 y + x4 y 3 + x4 y 2 2 3 7 5  (1) 2 5 2 3 y y 1 2 2 2 1 2 2 3 +x y + x y − − + (x + y ) + (x + y ) , 7 5 4 6 where ε ≥ 0 is a small parameter. Of course, when ε = 0 the Hamiltonian H0 is the classical Hénon-Heiles Hamiltonian. The Hamiltonian (1) was introduced in [2]. ∗ Corresponding author Email addresses: jllibre@mat.uab.cat (Jaume Llibre), tsalmalki@kau.edu.sa (Tareq Saeed), evzotos@physics.auth.gr (Euaggelos E. Zotos) The Hamiltonian system associated to the Hamiltonian (1) is ẋ = px , ẏ = py ,  ṗx = −x − 2xy − ε 6x5 y + 4x3 y 3 + 4x3 y + 2xy 5 + 2xy 3  (2) +x(x2 + y 2 ) + x(x2 + y 2 )2 ,  ṗy = −y − x2 + y 2 − ε x6 + 3x4 y 2 + x4 + 5x2 y 4 + 3x2 y 2 − y 6 − y 4  +y(x2 + y 2 ) + y(x2 + y 2 )2 , here the dot denotes derivative with respect to the time t. The equilibrium points of the generalized Hénon-Heiles Hamiltonian system are analyzed in section 2. As we shall see are the four equilibrium points of the classical Hénon-Heiles Hamiltonian system slightly perturbed. In general to study the orbits which go or come from the infinity of a differential system defined in R4 is a difficult task, but when the differential system is polynomial as system (2), Poincaré in [3] and its gener- alization in [4], provided a tool for doing this study. This tool is now called the Poincaré compactification. Here we will describe the Poincaré compactification for a polynomial differential system in R4 , because the domain of definition of system (2) is R4 . Roughly speaking the Poincaré compactification consists in identifying R4 with the interior of the unit closed ball B6 of R4 centered at the origin of coordinates. Then the boundary of this ball, the 5–dimensional sphere S3 , is identified with the infinity of R4 , because in R4 we can go to infinity in as many as directions as points has the sphere S3 . There is a technique which extend a polynomial differential system from the interior of the ball B4 to its boundary in an analytic way, in such a way that the boundary S3 is invariant by the extended flow, i.e. if an orbit of the extended differential system has a point in the sphere S3 the whole orbit remains on S3 . Then studying the dynamics of the extended differential system on S3 we understand the dynamics in a neighborhood of the infinity of the initial polynomial differential system defined in R4 . For instance, if the extended differential system has an equilibrium point on S3 which is a local attractor, then we know that there is a set of dimension 4 of orbits of the polynomial differential system which escapes to infinity in the direction defined by this equilibrium point. In section 3 we provide the explicit equations of the Poincaré compactification in R4 . Using this com- pactification we shall prove that the classical and the generalized Hénon-Heiles Hamiltonian systems has no infinite equilibrium points, i.e. equilibrium point in the sphere S3 . After the equilibrium points the periodic orbits are the most simple interesting orbits of a differential system. This is due mainly to the following two facts. First the periodic orbits provide information on the motion in their neighborhoods studying their type of stability. Moreover, if there are isolated periodic orbits having some multiplier distinct from 1 in the energy levels of the Hamiltonian system (2) this orbit prevents the existence of a second C 1 first integral independent with the Hamiltonian, see details in section 6. In section 4 we present a brief introduction to the averaging theory of second order. Using this theory we shall compute two families of periodic orbits of the classical and generalized Hénon–Heiles Hamiltonian system (2), and we obtain the following result. Theorem 1. The generalized Hamiltonian system for ε sufficiently small in each Hamiltonian level H = ε2 h > 0 has two periodic solutions of the form (x(t, √ ε), y(t, ε), px (t, ε),√ py (t, ε)) = √ √ (ε h cos t + O(ε2 ), ±ε h sin t + O(ε2 ), −ε h sin t + O(ε2 ), ±ε h cos t + O(ε2 )). Theorem 1 is proved in section 5. 2 Theorem 2. The generalized Hamiltonian system for ε sufficiently small in each Hamiltonian level H = ε2 h > 0 satisfies (a) either it is Liouville-Arnol’d integrable and the gradients of the two constants of motion are linearly dependent on some points of the two periodic orbits found in Theorem 1, (b) or it is not Liouville–Arnol’d integrable with any second C 1 first integral. Theorem 2 is proved in section 6. At the beginning of this section we recall the notion of Liouville-Arnol’d integrability. 2. The finite equilibria The classical Hénon-Heiles Hamiltonian system has four finite equilibria (x, y, px , py ), namely √ ! √ ! − 3 1 3 1 p1 = (0, 0, 0, 0), p2 = (0, 1, 0, 0), p3 = , − , 0, 0 , p4 = , − , 0, 0 , 2 2 2 2 The eigenvalues of the linear part of the Hamiltonian system √ at p1 are ±i with multiplicity two, and the corresponding eigenvalues at pj for j = 2, 3, 4 are ±1 and ± 3 i. For ε > 0 sufficiently small the equilibria of the Hamiltonian system (2) are p1 (ε) = (0, 0, 0, 0), p2 (ε) = (0, 1, 0, 0), (3) p3 (ε) = (x3 (ε), y(ε), 0, 0), p4 (ε) = (x4 (ε), y(ε), 0, 0), where √ √ √ √ 3 √ 2585 3ε2 144335 3ε3 149836699 3ε4  x3 (ε) = − + 3ε − + − + O ε5 , 2 256 1024 65536 √ √ 2 √ 3 √ 3 √ 2585 3ε 144335 3ε 149836699 3ε4  x4 (ε) = − 3ε + − + + O ε5 , 2 256 1024 65536 1 15ε 1233ε2 138483ε3 18018531ε4  y(ε) = − + − + − + O ε5 . 2 16 128 1024 8192 The eigenvalues of the linear part of the Hamiltonian √ p at p1 (ε) are again ±i with multiplicity two. system The corresponding eigenvalues at p2 (ε) are ± 1 + 2ε and ± 3(1 + 2ε)i. While the eigenvalues of pj (ε) for j = 3, 4 are 65536 + 268288ε − 3097664ε2 + 48298085ε3  ± + O ε4 , 65536 and √ 3(65536 − 14336ε + 273280ε2 − 4085287ε3 )  ± i + O ε4 . 65536 In short we have proved the following proposition. Proposition 3. For ε ≥ 0 sufficiently small the generalized Hénon-Heiles Hamiltonian system has four finite singular points, namely pj (ε) for j = 1, 2, 3, 4 given in (3). 3. The infinite equilibria: 3.1. The Poincaré compactification in R4 In R4 we consider a polynomial differential system ẋk = Pk (x1 , x2 , x3 , x4 ), for k = 1, 2, 3, 4, 3 or equivalently its associated polynomial vector field X = (P1 , P2 , P3 , P4 ). The degree n of X is defined as n = max{deg(Pi ) : i = 1, 2, 3, 4}. We have that S4 = {y = (y1 , y2 , y3 , y4 , y5 ) ∈ R5 : kyk = 1}, and let S+ = {y ∈ S4 : y5 > 0} and S− = {y ∈ S4 : y5 < 0} be the northern and southern hemispheres of the sphere S4 , respectively. The tangent space to S4 at the point y is denoted by Ty S4 . Then the tangent hyperplane T(0,0,0,0,1) S4 = {(x1 , x2 , x3 , x4 , 1) ∈ R5 : (x1 , , x2 , x3 , x4 ) ∈ R4 } is identified with R4 . We consider the central projections f+ : R4 = T(0,0,0,0,1) S4 → S+ and f− : R4 = T(0,0,0,0,1) S4 → S− , defined by 1 1 f+ (x) = (x1 , x2 , x3 , x4 , 1) and f− (x) = − (x1 , x2 , x3 , x4 , 1) , ∆x ∆x  P4 1/2 where ∆x = 1 + i=1 x2i . Through these central projections, R4 can be identified with the northern and the southern hemispheres, respectively. The equator of S4 is S3 = {y ∈ S4 : y5 = 0}. Clearly S3 can be identified with the infinity of R4 . The maps f+ and f− define two copies of X, one Df+ ◦ X in the northern hemisphere and the other Df− ◦ X in the southern one. Denote by X the vector field on S4 \ S3 = S+ ∪ S− which restricted to S+ coincides with Df+ ◦ X and restricted to sss− coincides with Df− ◦ X. In what follows we shall work with the orthogonal projection of the closed northern hemisphere to y5 = 0. Note that this projection is the closed ball B4 of radius one centered at the origin of coordinates, whose interior is diffeomorphic to R4 and whose boundary S3 corresponds to the infinity of R4 . We shall extend analytically the polynomial vector field X defined on S+ ∪ S− to its boundary S3 , in such a way that the flow on the boundary be invariant. This new vector field on B4 will be called the Poincaré compactification of the vector field X , and B6 will be called the Poincaré ball. Poincaré [3] introduced this compactification for polynomial vector fields in R2 , and its extension to Rm can be found in [4]. Now we can extend analytically the vector field X(y) to the whole sphere S4 defining the new vector field p(X)(y) = y5n−1 X(y). This extended vector field p(X) is called the Poincaré compactification of X . As S4 is a differentiable manifold, to compute the expression for p(X) we can consider the ten local charts (Ui , Fi ), (Vi , Gi ) where Ui = {y ∈ S4 : yi > 0}, and Vi = {y ∈ S4 : yi < 0} for i = 1, . . . , 5; the diffeomorphisms Fi : Ui → R4 and Gi : Vi → R4 for i = 1, . . . , 5 the inverses of the central projections from the origin to the tangent planes at the points (±1, 0, 0, 0, 0), (0, ±1, 0, 0, 0), . . . , (0, 0, 0, 0, ±1), respectively. We now determine the expression of the extended vector field p(X) on the local chart U1 . Suppose that the origin (0, 0, 0, 0, 0), the point (y1 , . . . , y5 ) ∈ S4 and the point (1, z1 , . . . , z4 ) in the tangent plane to S4 at (1, 0, 0, 0, 0) are collinear, then we have 1 z1 z2 z3 z4 = = = = , y1 y2 y3 y4 y5 and consequently   y2 y3 y4 y5 F1 (y) = , , = (z1 , z2 , z3 , z4 ) y1 y1 y1 y1 defines the coordinates on U1 . 4 As   −y2 /y12 1/y1 0 0 0 −y3 /y12 0 1/y1 0 0  DF1 (y) =  −y4 /y12  0 0 1/y1 0  −y5 /y12 0 0 0 1/y1  z n−1 4 and y5n−1 = . Then the analytical vector field p(X) in the local chart U1 becomes ∆z z4n n−1 (−z1 P1 + P2 , −z2 P1 + P3 , −z3 P1 + P4 , −z4 P1 ) , (4) (∆z) where Pi = Pi (1/z4 , z1 /z4 , z2 /z4 , z3 /z4 ). In a similar way we can deduce the expressions of p(X) in the local charts Uk for k = 2, 3, 4, 5. These are z4n n−1 (−z1 P2 + P1 , −z2 P2 + P3 , −z3 P2 + P4 , −z4 P2 ) , (5) (∆z) where Pi = Pi (z1 /z4 , 1/z4 , z2 /z4 , z3 /z4 ) in U2 , z4n n−1 (−z1 P3 + P1 , −z2 P3 + P2 , −z3 P3 + P4 , −z4 P3 ) , (6) (∆z) where P i = P i (z1 /z4 , z2 /z4 , 1/z4 , z3 /z4 ) in U3 , z4n n−1 (−z1 P4 + P1 , −z2 P4 + P2 , −z3 P4 + P3 , −z4 P4 ) , (7) (∆z) where P i = P i (z1 /z4 , z2 /z4 , z3 /z4 , 1/z4 ) in U4 . The expression of p(X) in U5 is z4n+1 (P1 , P2 , P3 , P4 ) where the component Pi = Pi (z1 , z2 , z3 , z4 ). The expression of p(X) in the local chart Vi is the same as in Ui multiplied by (−1)n−1 . When we shall work with the expression of the compactified vector field p(X) in the local charts we shall omit the factor 1/(∆z)n−1 . We can do that through a rescaling of the time. We remark that all the points on the sphere at infinity in the coordinates of any local chart have z4 = 0. 3.2. The Poincaré compactification of system (2) in the local chart U1 In this chart from (4) system (2) writes ż1 = −(z2 − z3 )z45 , ż2 = −z44 (2z1 + z4 + z22 z4 ) + ε(−6z1 − 4z13 − 2z15 − z4 − 2z12 z4 −z14 z4 − 4z1 z42 − 2z13 z42 − z43 − z12 z43 ), ż3 = −z44 (1 − z12 + z1 z4 + z2 z3 z4 ) + ε(−1 − 3z12 − 5z14 + z16 −z1 z4 − 2z13 z4 − z15 z4 − z42 − 3z12 z42 + z14 z42 − z1 z43 − z13 z43 ), ż4 = −z2 z46 . This system has no infinite equilibria (z1 , z2 , z3 , 0). 5 3.3. The Poincaré compactification of system (2) in the local chart U2 In this chart from (5) system (2) writes ż1 = (z2 − z1 z3 )z45 , ż2 = −z44 (2z1 + z1 z4 + z2 z3 z4 ) − εz1 (2 + 4z12 + 6z14 + z4 + 2z12 z4 +z14 z4 + 2z42 + 4z12 z42 + z43 + z12 z43 ), (8) ż3 = −z44 (−1 + z12 + z4 + z32 z4 ) + ε(1 − 5z12 − 3z14 − z16 − z4 −2z12 z4 − z14 z4 + z42 − 3z12 z42 − z14 z42 − z43 − z12 z43 ), ż4 = −z3 z46 . In system (8) we must look for the infinite equilibria of the form (0, z2 , z3 , 0), because if there were infinite equilibria with z1 6= 0 these would be appear in the local chart U1 . But system (8) has no equilibria of the form (0, z2 , z3 , 0). 3.4. The Poincaré compactification of system (2) in the local chart U3 In this chart from (6) system (2) writes ż1 = z44 (2z12 z2 + z4 + z12 z4 ) + εz12 (6z14 z2 + 4z12 z23 + 2z25 + z14 z4 +2z12 z22 z4 + z24 z4 + 4z12 z2 z42 + 2z23 z42 + z12 z43 + z22 z43 ), ż2 = z44 (2z1 z22 + z1 z2 z4 + z3 z4 ) + εz1 z2 (6z14 z2 + 4z12 z23 + 2z25 +z14 z4 + 2z12 z22 z4 + z24 z4 + 4z12 z2 z42 + 2z23 z42 + z12 z43 + z22 z43 ), ż3 = −z44 (z12 − z22 − 2z1 z2 z3 + z2 z4 − z1 z3 z4 ) + ε(−z16 − 3z14 z22 −5z12 z24 + z26 + 6z15 z2 z3 + 4z13 z23 z3 + 2z1 z25 z3 − z14 z2 z4 (9) −2z12 z23 z4 − z25 z4 + z15 z3 z4 + 2z13 z22 z3 z4 + z1 z24 z3 z4 − z14 z42 −3z12 z22 z42 + z24 z42 + 4z13 z2 z3 z42 + 2z1 z23 z3 z42 − z12 z2 z43 − z23 z43 +z13 z3 z43 + z1 z22 z3 z43 ), ż4 = z1 z45 (2z2 + z4 ) + εz1 z4 (6z14 z2 + 4z12 z23 + 2z25 + z14 z4 + 2z12 z22 z4 +z24 z4 + 4z12 z2 z42 + 2z23 z42 + z12 z43 + z22 z43 ). In system (9) we must look for the infinite equilibria of the form (0, 0, z3 , 0), because if there were infinite equilibria with z12 + z22 6= 0 these would be appear in the local charts U1 ∪ U2 . But system (9) has no equilibria of the form (0, 0, z3 , 0). 3.5. The Poincaré compactification of system (2) in the local chart U4 In this chart from (7) system (2) writes ż1 = z44 (z13 − z1 z22 + z1 z2 z4 + z3 z4 ) + εz1 (z16 + 3z14 z22 + 5z12 z24 − z26 +z14 z2 z4 + 2z12 z23 z4 + z25 z4 + z14 z42 + 3z12 z22 z42 − z24 z42 + z12 z2 z43 +z23 z43 ), ż2 = z44 (z12 z2 − z23 + z4 + z22 z4 ) − εz2 (−z16 − 3z14 z22 − 5z12 z24 + z26 −z14 z2 z4 − 2z12 z23 z4 − z25 z4 − z14 z42 − 3z12 z22 z42 + z24 z42 − z12 z2 z43 −z23 z43 ), ż3 = z44 (−2z1 z2 + z12 z3 − z22 z3 − z1 z4 + z2 z3 z4 ) + ε(−6z15 z2 − 4z13 z23 (10) −2z1 z25 + z16 z3 + 3z14 z22 z3 + 5z12 z24 z3 − z26 z3 − z15 z4 − 2z13 z22 z4 −z1 z24 z4 + z14 z2 z3 z4 + 2z12 z23 z3 z4 + z25 z3 z4 − 4z13 z2 z42 − 2z1 z23 z42 +z14 z3 z42 + 3z12 z22 z3 z42 − z24 z3 z42 − z13 z43 − z1 z22 z43 + z12 z2 z3 z43 +z23 z3 z43 ), ż4 = z45 (z12 − z22 + z2 z4 ) + εz4 (z16 + 3z14 z22 + 5z12 z24 − z26 + z14 z2 z4 +2z12 z23 z4 + z25 z4 + z14 z42 + 3z12 z22 z42 − z24 z42 + z12 z2 z43 + z23 z43 ). 6 In system (10) we must look for the infinite equilibria of the form (0, 0, 0, 0), because if there were infinite equilibria with z12 + z22 + z32 6= 0 these would be appear in the local charts U1 ∪ U2 ∪ U3 . But the point (0, 0, 0, 0) is not an equilibrium point for system (10). In short we have proved the following proposition. Proposition 4. The generalized Hénon-Heiles Hamiltonian system has no infinite singular points. 4. The averaging theory of first and second order Here we summarize the averaging theory of order two for finding periodic orbits. See the paper [5] the proofs of the results presented in his section. Theorem 5. We assume that the non-autonomous differential system ẋ(t) = εF1 (t, x) + ε2 F2 (t, x) + ε3 R(t, x, ε), (11) being F1 , F2 : R × D → Rn , R : R × D × (−εf , εf ) → Rn continuous functions, T-periodic in the t variable, and D is an open subset of Rn , satisfies the following assumptions. (i) The functions F1 (t, ·) ∈ C 1 (D) for all t ∈ R, F1 , F2 , R and Dx F1 are locally Lipschitz with respect to x, and R is differentiable with respect to ε. We define f1 , f2 : D → Rn as Z T f1 (z) = F1 (s, z)ds, 0 Z T Z s f2 (z) = [Dz F1 (s, z) F1 (t, z)dt + F2 (s, z)]ds. 0 0 (ii) There exists an open and bounded set V ⊂ D such that for each ε ∈ (−εf , εf )\{0}, there is a ∈ V verifying f1 (a) + εf2 (a) = 0 and the Brouwer degree dB (f1 + εf2 )(a) of the function f1 + εf2 at its fixed point a is not zero. Then for |ε| > 0 small enough, there exists a T −periodic solution x(·, ε) of system (11) verifying that x(0, ε) → a when ε → 0. We recall that if the Jacobian of a function f at its fixed point a is non–zero, then the Brouwer degree dB (f1 + εf2 )(a) in non-zero, see [6]. When the function f1 6≡ 0, then for ε small enough the zeros of f1 + εf2 are essentially the zeros of f1 . In this case Theorem 5 provides the averaging theory of first order. When the function f1 ≡ 0 and f2 6≡ 0, then the zeros of f2 are the zeros of f1 + εf2 , and Theorem 5 provides the averaging theory of second order. In the following section we will show that after adequate changes of variables, we can apply Theorem 5 to the classical and generalized Hénon–Heiles Hamiltonian systems (2). And consequently we can prove the existence of two families of periodic orbits for these systems. 5. Proof of Theorem 1 It is well known that for Hamiltonian system with more than one degree of freedom their periodic orbits generically live on cylinders filled of periodic orbits. Hence it is not possible to apply directly Theorem 5 to a Hamiltonian system, because then the Jacobian of the function f1 + εf2 at its fixed point a will be always zero. Therefore Theorem 5 must be applied to every fixed Hamiltonian level where generically the periodic orbits appear isolated. Remember that in the sense of Liouville–Arnol’d Theorem the integrable Hamiltonian systems are non–generic, see [7]. From the statement of Theorem 5 it follows that in order to apply it the differential system needs to be periodic in the independent variable. Therefore in the Hamiltonian system (2) we change the variables 7 (x, y, px , py ) to (r, θ, ρ, α) given by (r cos θ, ρ cos(θ + α), r sin θ, ρ sin(θ + α)), and later on we will take as the new independent variable the θ. In these new variables system (2) becomes  ṙ = −ρr sin(2θ) cos(α + θ) − εr sin θ cos θ ρ2 cos2 (α + θ)  2ρ3 cos3 (α + θ) + ρ2 cos2 (α + θ) + 2ρ cos(α + θ) + 1 +r4 cos4 θ(6ρ cos(α + θ) + 1) + r2 cos2θ 4ρ3 cos3 (α + θ)  +2ρ2 cos2 (α + θ) + 4ρ cos(α + θ) + 1 ,  θ̇ = −1 − 2ρ cos2 θ cos(α + θ) − ε cos2 θ ρ2 cos2 (α + θ)  1 + 2ρ3 cos3 (α + θ) + ρ2 cos2 (α + θ) + 2ρ cos(α + θ) +r4 cos4 θ(6ρ cos(α + θ) + 1) + r2 cos2 θ 1 + 4ρ3 cos3 (α + θ)  +2ρ2 cos2 (α + θ) + 4ρ cos(α + θ) ,  ρ̇ =  sin(α + θ) ρ2 cos2 (α + θ) − r2 cos2 θ − ε sin(α + θ) ρ3 cos3 (α + θ) 1 − ρ3 cos3 (α + θ) + ρ2 cos2 (α + θ)  (12) −ρ cos(α + θ) + r6 cos6 θ + r4 cos4 θ 3ρ2 cos2 (α + θ) +ρ cos(α + θ) + 1 + ρr2 cos2 θ cos(α + θ) 5ρ3 cos3 (α + θ) 2 2  +2ρ cos (α + θ) + 3ρ cos(α + θ) + 1 ,  cos(α + θ) ρ2 cos(2(α + θ)) + 3ρ2 − cos(2θ)(r2 − 2ρ2 ) − r2 α̇ = 2ρ 1 3 +ε ρ cos (α + θ) ρ cos (α + θ) − ρ2 cos2 (α + θ) 4 3 3 ρ  +ρ cos(α + θ) − 1 + r2 cos4 θ ρ + ρ(2ρ2 − r2 ) cos2 (α  + θ) −(r2 − 4ρ2 ) cos(α + θ) + (4ρ4 − 3ρ2 r2 ) cos3 (α + θ) +ρ cos2 θ cos2 (α + θ) ρ2 + (2ρ3 − 3ρr2 ) cos(α + θ)  +(2ρ5 − 5ρ3 r2 ) cos3 (α + θ) + (ρ4 − 2ρ  2 2 r ) cos2 (α + θ) − r2  −r4 cos6 θ (r2 − 6ρ2 ) cos(α + θ) − ρ . This system is not Hamiltonian but it has the first integral 1 2  ρ (3 − 2ρ cos3 (α + θ)) + 3r2 cos2 θ(2ρ cos(α + θ) + 1) + 3r2 sin2 θ 6  1 4 +ε ρ cos4 (α + θ) 105 − 2ρ cos(α + θ)(5ρ cos(α + θ)(6ρ cos(α + θ) 420  1 1 −7) + 42) + r6 cos6 θ(6ρ cos(α + θ) + 1) + r4 cos4 θ 2ρ cos(α + θ) (13) 6 4  1 2 2 (ρ cos(α + θ)(2ρ cos(α + θ) + 1) + 2) + 1 + ρ r cos2 θ cos2 (α + θ)  2 (2ρ cos(α + θ) + 1)(ρ2 cos2 (α + θ) + 1) . In order to apply Theorem 5 we also need a small parameter in front of the vector field associated to the differential system, so we will do the rescaling (r, ρ) = ε(R, σ) using the parameter ε, and we take θ as the 8 new independent variable. Hence the differential system (12) becomes R0 = εRσ sin(2θ) cos(α + θ) − ε2 2Rσ 2 sin(2θ) cos2 θ cos2 (α + θ) +O(ε3 ),   σ 0 = ε R2 cos2 θ sin(α + θ) − σ 2 sin(α + θ) cos2 (α + θ)  +ε2 2σ 3 cos2 θ sin(α + θ) cos3 (α + θ) − 2R2 σ cos4 θ sin(α + θ)  cos(α + θ) + O(ε3 ), (14) −ε    α0 = cos(α + θ) σ 2 cos(2(α + θ)) − cos(2θ) R2 − 2σ 2 − R2 + 3σ 2 2σ   +ε2 cos2 θ cos2 (α + θ) σ 2 cos(2(α + θ)) − cos(2θ) R2 − 2σ 2 − R2  +3σ 2 + O(ε3 ), here the prime denotes derivative with respect to the variable θ. Of course, system (14) is 2π-periodic in the variable θ. System (14) has the first integral (13) which in the variables (R, ρ, α) writes   21 1 3 H = ε (R + σ ) + ε R σ cos θ cos(α + θ) − σ cos (α + θ) + O(ε4 ). 2 2 3 2 2 3 (15) 2 3 We fix the value of the first integral H at ε2 h > 0 in order that the averaging theory can provide information about the periodic orbits of system (14). Computing σ from equation (15) we obtain p ε  σ = 2h − R2 + 2h cos3 (α + θ) − R2 cos3 (α + θ) − 3R2 cos2 θ cos(α + θ) + O(ε2 ). 3 Now substituting σ in system (14), this differential system reduces to √ ε2 R0 = εR 2h − R2 sin(2θ) cos(α + θ) + R sin(2θ) cos2 (α + θ)  6   (2h − R2 ) cos(2(α + θ)) + 3 R2 − 4h cos(2θ) + 2(R2 − 5h) +O(ε3 ), ε cos(α + θ)  2 α0 = √ (R − 2h) cos(2(α + θ)) + (3R2 − 4h) cos(2θ) (16) 2 2h − R2  ε2 cos2 (α + θ)  −6h + 4R2 + 3 cos2 θ (8h2 − 6hR2 + R4 ) 12h − 6R2 2 cos(2(α + θ)) + (16h − 16hR2 + 5R4 ) cos(2θ) + 24h2 − 22hR2 +6R + (2h − R ) cos2 (α + θ) (R 4 2 2 − 2h) cos(2(α + θ)) 2 2  +(R − 4h) cos(2θ) − 6h + 2R + O(ε3 ). Now system (16) satisfies all the assumptions for applying Theorem 5, i.e. it has the form (11) with x = (R, σ), t = θ, T = 2π, F1 = (F11 , F12 ) and F2 = (F21 , F22 ), where √ F11 = R 2h − R2 sin(2θ) cos(α + θ), cos(α + θ)  2  F12 = √ (R − 2h) cos(2(α + θ)) + (3R2 − 4h) cos(2θ) − 6h + 4R2 , 2 2h − R2 9 and  1  F21 = R sin(2θ) cos2 (α + θ) (2h − R2 ) cos(2(α + θ)) + 3 R2 − 4h cos(2θ) 6  +2(R2 − 5h) , ε2 cos2 (α + θ)  F22 = 3 cos2 θ (8h2 − 6hR2 + R4 ) cos(2(α + θ)) + (16h2 12h − 6R2  −16hR2 + 5R4 ) cos(2θ) + 24h2 − 22hR2 + 6R4 + (2h − R2 )  cos2 (α + θ) (R2 − 2h) cos(2(α + θ)) + (R2 − 4h) cos(2θ) − 6h + 2R2 . The averaging theory of first order does not provide any information about the periodic solutions of system (16), because the average functions of F11 and F12 in the period becomes zero, i.e. Z 2π f1 (R, α) = (F11 , F12 ) dθ = (0, 0). 0 As the averaged function f1 of Theorem 5 is identically zero, we compute the function f2 by applying the second order averaging theory. We have that Z 2π f2 (R, α) = (DRα F1 (θ, R, α).y1 (θ, R, α) + F2 (θ, R, α)) dθ, 0 where Z θ y1 (θ, R, α) = F1 (t, R, α) dt. 0 and the Jacobian matrix is   ∂F11 ∂F11  ∂r ∂α  Drα F1 (θ, r, α) =   ∂F12 . ∂F12  ∂r ∂α The two components of the vector y1 are Z θ y11 = F11 (t, R, α) dt 0 1 √ = R 2h − R2 (−3 cos(α − θ) − cos(α + 3θ) + 4 cos(α)), 6 Z θ y12 = F12 (t, R, α) dt 0 1  = √ 12h sin(α − θ) − 42h sin(α + θ) − 2h sin(3(α + θ)) 12 2h − R2 +(3R − 4h) sin(α + 3θ) + sin(α)(34h − 21R2 ) + sin(3α)(2h 2  − R2 ) −9R2 sin(α − θ) + 27R2 sin(α + θ) + R2 sin(3(α + θ)) . Now we calculate the averaged function of second order f2 = (f21 , f22 ) defined in Theorem 5 and we have 7  f21 = − πR sin(2α) R2 − 2h , 6 14  f22 = − π sin2 α h − R2 . 3 We must comput the zeros (R∗ , α∗ ) of f2 (R, α), and to verify that the Jacobian determinant |DR,α f2 (R∗ , α∗ )| = 6 0. (17) 10 Solving the system f2 (R, α) = 0 of two equations and two unknowns R and α we get six solutions (R∗ , α∗ ) with R∗ ≥ 0, namely √ √ √ √ (0, 0), (0, π), ( 2h, 0), ( 2h, π), ( h, π/2), ( h, −π/2). (18) The first four solutions are not good, because for them the Jacobian (18) vanishes, but for the last two solution the Jacobian (18) is 196h2 π 2 /9 6= 0. So, by Theorem 5 these two solutions provide two periodic solutions (R± (θ,√ ε), α± (θ, ε)) of the differential system (16) with ε sufficiently small such that (R± (0, ε), α± (0, ε)) → ( h, ±π/2) when ε → 0. Going back to the differential system (14) we get for this system with ε sufficiently √ √ small two periodic solutions (R± (θ, ε), σ± (θ, ε), α± (θ, ε)) such that (R± (0, ε), σ± (0, ε), α± (0, ε)) → ( h, h, ±π/2) when ε → 0. Again going back to the differential system (12) we obtain √ for this system with √ ε sufficiently small two periodic solutions (r± (t, ε), θ(t, ε), ρ± (t, ε), α± (t, ε)) = (ε h + O(ε2 ), −t + O(ε), ε h + O(ε2 ), ±π/2 + O(ε)). Finally going back to the initial Hamiltonian system (2) we have for this system with ε sufficiently small two periodic solutions (x(t, √ ε), y(t, ε), px (t, ε),√ py (t, ε)) = √ √ (ε h cos t + O(ε2 ), ±ε h sin t + O(ε2 ), −ε h sin t + O(ε2 ), ±ε h cos t + O(ε2 )), in each positive Hamiltonian level H = ε2 h. This completes the proof of Theorem 1. In the next section we will use the existence of these two periodic orbits with multipliers different from 1 to study the non–integrability of the Hamiltonian system (2). 6. Periodic orbits and the Liouville–Arnol’d integrability First we present some results on the Liouville–Arnol’d integrability of the Hamiltonian systems, and also on the periodic orbits of the differential equations, see more details in [8, 9] and the subsection 7.1.2 of [9], respectively. We restrict our attention to the Hamiltonian systems with two degrees of freedom like our generalized Hénon-Heiles Hamiltonian system, but we remark that these results work in Hamiltonian systems with an arbitrary number of degrees of freedom. It is well known that a Hamiltonian system with Hamiltonian H of two degrees of freedom is integrable in the sense of Liouville–Arnol’d if it has a second first integral C independent with H (i.e. the gradient vectors of H and C are independent in all the points of the phase space except perhaps in a set of zero Lebesgue measure), and in involution with H (i.e. the parenthesis of Poisson of H and C is zero). For Hamiltonian systems with two degrees of freedom the involution condition is redundant, because the fact that C is a first integral of the Hamiltonian system, implies that the mentioned Poisson parenthesis is always zero. A flow defined on a subspace of the phase space is complete if its solutions are defined for all time t ∈ R. The Liouville–Arnol’d Theorem restricted to Hamiltonian systems of two degrees of freedom is: Theorem 6. Consider a Hamiltonian system with two degrees of freedom defined on the phase space M with Hamiltonian H and having a second first integral C independent with H. Let Ihc = {p ∈ M : H(p) = 6 ∅ be. If (h, c) is a regular value of the map (H, C), then the following statements hold. h and C(p) = c} = (a) Ihc is a two dimensional submanifold of M invariant under the flow of the Hamiltonian system. ∗ ∗ (b) If the flow on a connected component Ihc of Ihc is complete, then Ihc is diffeomorphic either to the torus S × S , or to the cylinder S × R, or to the plane R . If Ihc is compact, then the flow on it is 1 1 1 2 ∗ ∗ always complete and Ihc ≈ S1 × S1 . ∗ (c) Under the assumptions of statement (b) the flow on Ihc is conjugated to a linear flow on either S1 × S1 , or on S × R, or on R . 1 2 11 The main result of this theorem states that the connected components of the invariant sets associated with the two independent first integrals in involution are generically submanifolds of the phase space, and if the flow on them is complete then they are diffeomorphic to a torus, a cylinder or a plane, where the flow is conjugated to a linear one. ∗ Using the notation of Theorem 6 when a connected component Ihc is diffeomorphic to a torus, either all orbits on this torus are periodic if the rotation number associated to this torus is rational, or they are quasi-periodic (i.e. every orbit is dense in the torus) if the rotation number associated to this torus is not rational. Consider the autonomous differential system ẋ = f (x), where f : U → Rn is C 2 , and U is an open subset of Rn . We write its general solution as φ(t, x0 ) with φ(0, x0 ) = x0 ∈ U and t belonging to its maximal interval of definition. We say that φ(t, x0 ) is T -periodic with T > 0 if and only if φ(T, x0 ) = x0 and φ(t, x0 ) 6= x0 for t ∈ (0, T ). The periodic orbit associated to the periodic solution φ(t, x0 ) is γ = {φ(t, x0 ), t ∈ [0, T ]}. The variational equation associated to the T -periodic solution φ(t, x0 ) is   ∂f (x) Ṁ = M, (19) ∂x x=φ(t,x0 ) where M is an n × n matrix. The monodromy matrix associated to the T -periodic solution φ(t, x0 ) is the solution M (T, x0 ) of (19) satisfying that M (0, x0 ) is the identity matrix. The eigenvalues λ of the monodromy matrix associated to the periodic solution φ(t, x0 ) are called the multipliers of the periodic orbit. For an autonomous differential system, one of the multipliers is always 1, and its corresponding eigen- vector is tangent to the periodic orbit. A periodic solution of an autonomous Hamiltonian system always has two multipliers equal to one. One multiplier is 1 because the Hamiltonian system is autonomous, and the other 1 is due to the existence of the first integral given by the Hamiltonian. Theorem 7. If a Hamiltonian system with two degrees of freedom and Hamiltonian H is Liouville–Arnol’d integrable, and C is a second first integral such that the gradients of H and C are linearly independent at each point of a periodic orbit of the system, then all the multipliers of this periodic orbit are equal to 1. Theorem 7 is due to Poincaré [3], section 36, see also [10]. It provides a tool for studying the non Liouville–Arnol’d integrability, independently of the class of differentiability of the second first integral. The main problem for applying this theorem is to find periodic orbits having multipliers different from 1. Proof of Theorem 2. Consider the two periodic solutions stated in Theorem 1. Their corresponding Jacobian 196h2 π 2 /9 6= 1 playing with the energy level h. Since this Jacobian is the product of the four multipliers of these periodic solutions with two of them always equal to 1, the remainder two multipliers cannot be equal to 1. Hence, by Theorem 7, either the Hénon–Heiles systems cannot be Liouville–Arnol’d integrable with any second first integral C, or the system is Liouville-Arnol’d integrable and the differentials of H and C are linearly dependent on some points of these periodic orbits. Therefore the theorem is proved. Acknowledgments The first author is partially supported by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación grants MTM2016-77278-P (FEDER), the Agència de Gestió d’Ajuts Universitaris i de Recerca grant 2017SGR1617, and the H2020 European Research Council grant MSCA-RISE-2017-777911. 12 References [1] Hénon M. and Heiles C., The applicability of the third integral of motion: some numerical experiments, Astron. J. 69 (1964), 73–84. [2] Dubeibe F.L., Zotos E.E. and Chen W., On the dynamics of a seventh-order generalized Hénon-Heiles potential, preprint, 2020. [3] Poincaré H., Les Mèthodes Nouvelles de la Méchanique Céleste, 3 Vols., Gauthier–Villar, Paris, 1892–1899, (reprinted by Dover, New York, 1957). [4] Cima A. and Llibre J., Bounded polynomial vector fields, Trans. Amer. Math. Soc. 318 (1990), 557–579. [5] Buică A. and Llibre J., Averaging methods for finding periodic orbits via Brouwer degree, Bull. Sci. Math. 128 (2004), 7–22. [6] Lloyd N.G., Degree Theory, Cambridge University Press, 1978. [7] Markus L. and Meyer, K.R., Generic Hamiltonian Dynamical Systems are neither integrable nor ergodic, Memoirs of the Amer. Math. Soc. 144, 1974. [8] Abraham R. and Marsden J.E., Foundations of Mechanics, Benjamin, Reading, Masachusets, 1978. [9] Arnol’d V.I., Kozlov V. and Neishtadt A., Dynamical Systems III. Mathematical Aspects of Classical and Celestial Me- chanics, Third Edition, Encyclopaedia of Mathematical Science, Springer, Berlin, 2006. [10] Llibre J. and Valls C., On the C 1 integrability of differential systems via periodic orbits, Euro. Jnl of Applied Mathematics 22 (2011), 381–391. 13