A Counterexample to a Result on Lotka–Volterra Systems

In the article of Dancsó et al. (Acta Appl. Math. 23:103–127, 1991) the authors claim the existence of a Hopf bifurcation which in general does not exist.

Theorem 1 Assume that the functionsp(μ), p(μ),q(μ), q(μ) and C(μ) are continuously differentiable and for admissible values of μ these functions are positive and satisfŷ If, for some μ 0 ,p and Unfortunately there is a problem in the proof of Theorem 1 due to the following counterexample.
Proof Comparing system (1) with system (5) we havê Then we havep So, condition (2) holds. Take μ 0 = 0. Then, condition (3) is immediately satisfied, and for condition (4) we obtain In short, all the conditions of Theorem 1 are satisfied by system (5). The unique equilibria of system (5) are the (0, 0) and (1, 1). Around the equilibrium point (0, 0) cannot exist periodic orbits because the straight lines x = 0 and y = 0 are invariant by the flow of system (5), i.e. they are formed by orbits of system (5). Therefore, if there are periodic orbits these must surround the equilibrium point (1, 1). We recall that in the bounded region limited by a periodic orbit of a differential system in the plane it must be an equilibrium point, see for instance Theorem 1.31 of [2].
We claim that the unique periodic orbits of systems (5) for all μ exist when μ = 0, and they are the periodic orbits surrounding the center (1, 1) of system (5) with μ = 0. Now we shall prove the claim. Clearly once the claim is proved it follows that system (5) cannot exhibit an Andronov-Hopf bifurcation.
System (5) with μ = 0 has the first integral H = x + y + 1/(xy), because the derivative of H on the orbits of system (5) with μ = 0 satisfies that Since the eigenvalues of the linear differential system (5) with μ = 0 at the equilibrium (1, 1) are ± √ 3i, this equilibrium either is a focus or a center, but it cannot be a focus  (1, 1). Hence, we have proved that the equilibrium (1, 1) for system (5) with μ = 0 is a center. Now we shall see that the periodic orbits of this center filled all the positive quadrant Q = {(x, y) : x > 0 and y > 0}. Assume that they do not filled all that quadrant. Since in that quadrant the unique equilibrium is the (1, 1), the external boundary of the continuum set of periodic orbit surrounding the center (1, 1) must be a periodic orbit γ , and after that orbit the nearby orbits must spiral. Consider the Poincaré map defined in an analytic transversal arc Σ to this periodic orbit γ . Since the flow of the polynomial differential system (5) with μ = 0 is analytic, such a Poincaré map is analytic, but it is not possible that an analytic map of one variable be the identity on the piece of the arc Σ contained in the bounded region limited by γ , and different to the identity on the piece of the arc Σ outside the bounded region limited by γ . So such a last periodic orbit γ does not exist and the periodic orbits surrounding the center (1, 1) filled all the positive quadrant Q. See a picture of the phase portrait of system (5) with μ = 0 on the Poincaré disc in Fig. 1, for more details on the Poincaré disc see Chap. 5 of [2]. For completing the proof of the claim we must show that system (5) with μ = 0 has no periodic solutions surrounding the equilibrium (1, 1). We shall use the Dulac criterium: Let P and Q be the polynomials defined in (5). If there exists a C 1 function B(x, y) in a simply connected region R such that ∂(BP )/∂x + ∂(BQ)/∂y has constant sign and is not identically zero in any subregion of R, then system (5) does not have a periodic orbit lying entirely in R. For a proof of this criterium see for instance Theorem 7.12 of [2].
Consider the function B = 1/(x 2 y 2 ) defined in the positive quadrant Q. Then ∂(BP ) ∂x + ∂(BQ) ∂x = μ x 2 y 2 = 0 in Q if and only if μ = 0. Therefore, by the Dulac criterium, system (5) with μ = 0 has no periodic solutions surrounding the equilibrium (1, 1), and this prevents the existence of a Hopf bifurcation. The proof of the claim and of Theorem 2 is completed.