English

not defined

no text concepts found

The Annals of Probability 2009, Vol. 37, No. 4, 1273–1328 DOI: 10.1214/08-AOP432 © Institute of Mathematical Statistics, 2009 THE LARGEST SAMPLE EIGENVALUE DISTRIBUTION IN THE RANK 1 QUATERNIONIC SPIKED MODEL OF WISHART ENSEMBLE B Y D ONG WANG Brandeis University We solve the largest sample eigenvalue distribution problem in the rank 1 spiked model of the quaternionic Wishart ensemble, which is the first case of a statistical generalization of the Laguerre symplectic ensemble (LSE) on the soft edge. We observe a phase change phenomenon similar to that in the complex case, and prove that the new distribution at the phase change point is the GOE Tracy–Widom distribution. 1. Introduction. The Wishart ensemble is defined as follows [24]: Consider M independent observation x1 , . . . , xM of an N -variate normal distribution with mean 0 and covariance matrix . Here the values of the normal distribution can be real, complex or even quaternion. If the variables are complex or quaternionic, then the definition of the mean is as usual, and the (co)variance is defined as cov(x, y) = E (x − x̄)(y − ȳ)∗ , where x̄ (resp. ȳ) is the mean of x (resp. y), and ∗ is the complex or quaternionic conjugation operator. Then is a real symmetric/Hermitian/quaternionic Hermitian matrix. Without loss of generality, we assume to be a diagonal matrix, with population eigenvalues l = (l1 , . . . , lN ). If we put the above data into an N × M double array X = (x1 : · · · : xM ), then the positively defined real sym1 XX∗ is the sample matrix metric/Hermitian/quaternionic hermitian matrix S = M and its eigenvalues λ = (λ1 , . . . , λN ) are sample eigenvalues. (X∗ is the transpose, Hermitian transpose or quaternionic Hermitian transpose of X depending on type of X’s entries.) The probability space of λi ’s is called the Wishart ensemble. It is a classical result [2] that (in the real category) if M N , λi ’s are good approximations of li ’s. But if M and N are of the same order of magnitude, that is, M/N = γ 2 ≥ 1 and M and N are very large, the problem is subtler. The simplest case with = I , the white Wishart ensemble, is the Laguerre ensemble, well studied in random matrix theory (RMT) under the name LOE, LUE and LSE—they are abbreviations of Laguerre Orthogonal/Unitary/Symplectic Ensemble, and GOE, GUE and GSE appearing later are abbreviations of Gaussian Received April 2008. AMS 2000 subject classifications. Primary 15A52, 41A60; secondary 60F99. Key words and phrases. Wishart distribution, quaternionic spiked model. 1273 1274 D. WANG Orthogonal/Unitary/Symplectic Ensemble—over all the three base fields, respectively. Naturally, the next question is: If is slightly deviate from I , such that li = 1 + ai , i = 1, . . . , k, and lk+1 = · · · = lN = 1, what is the distribution of the λi ’s? This is called the spiked model [19] and k is defined as its rank. If M and N are very large and k and ai ’s are small constants, the density of λi ’s is the same as that in the white Wishart model, proved in [22] in real and complex categories. The distribution of the largest sample eigenvalue, however, may change. For the complex ensemble, Baik, Ben Arous and Péché [4] solved the problem completely. They show that if max(ai ) is smaller than a threshold, then the distribution of the largest sample eigenvalue is the same as that in the white ensemble, which is the GUE Tracy–Widom distribution, but if max(ai ) exceeds the threshold, that distribution is changed into a Gaussian whose mean and variance depend on max(ai ). Furthermore, in the case that max(ai ) equals the critical value, they find a series of new distributions, indexed by the multiplicity of max(ai ). In the real category, which is practically the most important and mathematically the most difficult, much less is known. In this paper I solve the distribution of the largest sample eigenvalue for the rank 1 spiked model in the quaternionic category. I believe the similarity of LOE and LSE [13] suggests that the solution to the quaternionic spiked model is an intermediate step toward the solution to the real one. 1.1. Some known results for the largest sample eigenvalue in white and rank 1 spiked models. In latter part of the paper, we concentrate on the distribution of the largest sample eigenvalue in the rank 1 spiked model, so denote a to be the only perturbation parameter. The result in the complex category is complete. First we recall the result for the complex white Wishart ensemble. P ROPOSITION 1. The distribution of the largest sample eigenvalue in the complex white Wishart ensemble satisfies that, max(λ) almost surely approaches [15] (1 + γ −1 )2 with fluctuation scale M −2/3 , and [11, 18] γ M 2/3 ≤T M→∞ (1 + γ )4/3 where FGUE is the GUE Tracy–Widom distribution. lim P max(λ) − (1 + γ −1 )2 · = FGUE (T ), The GUE Tracy–Widom distribution is defined by Fredholm determinant [11, 29]: FGUE (T ) = det 1 − KAiry (ξ, η)χ(T ,∞) (η) , where χ(T ,∞) is the step function: χ(T ,∞) (η) = 1, 0, if η ∈ (T , ∞), otherwise, 1275 QUATERNIONIC WISHART and KAiry (ξ, η) is the well-known Airy kernel defined by the Airy function Ai(x): KAiry (ξ, η) = (1) ∞ 0 Ai(ξ + t) Ai(η + t) dt. The Airy function can be defined in different ways, and here we take an integral representation suitable for our asymptotic analysis [4]: −1 Ai(ξ ) = 2πi (2) e−ξ z+1/3z dz, 3 ∞ where ∞ = 1∞ ∪ 2∞ ∪ 3∞ , which are defined as (see Figure 1) 1∞ = {−teπ i/3 |−∞ < t ≤ −1}, 2∞ = e−tπ i |− 13 ≤ t ≤ 1 3 , 3∞ = {te5π i/3 |1 ≤ t < ∞}. The breakthrough in the complex category is by [4], which is for any finite rank spiked model. In the rank 1 case, it is: P ROPOSITION 2. In the rank 1 complex spiked model: 1. If −1 < a < γ −1 , then the distribution of the largest sample eigenvalue is the same as that of the complex white Wishart ensemble in Proposition 1. 2. If a = γ −1 , then the limit and the fluctuation scale are the same as those of the complex white Wishart ensemble, but the distribution function is (3) lim P max(λ) − (1 + γ M→∞ −1 2 γ M 2/3 ) · ≤T (1 + γ )4/3 F IG . 1. ∞ . = FGUE1 (T ). 1276 D. WANG 3. If a > γ −1 , then the limit and the fluctuation scale are changed as well as the distribution function, which is a Gaussian: √ 1 M lim P max(λ) − (a + 1) 1 + 2 · ≤T M→∞ γ a (a + 1) 1 − 1/(γ 2 a 2 ) (4) T 1 2 √ e−t /2 dt. = −∞ 2π The function FGUE1 occurring in (3) is defined similarly to FGUE [4]: (5) FGUE1 (T ) = det 1 − KAiry (ξ, η) + s (1) (ξ ) Ai(η) χ(T ,∞) (η) , where s (1) is one of a series of functions defined in [4], and has the integral representation ∞ 1 (1) −ηz+1/3z3 1 (1) s (η) = dz and s (η) = 1 − e Ai(t) dt, 2πi ¯¯ ∞ z η where ¯¯ ∞ = ¯¯ ∞ ∪ ¯¯ ∞ ∪ ¯¯ ∞ , which are defined as (see Figure 2 ε is a positive 1 constant, used later) 2 3 5 ε tπ i 1 e ≤t ≤ , ¯¯ ∞ = 2 2 3 3 ε π i/3 ¯¯ ∞ −∞ < t ≤ − , 1 = −te 2 ε 5π i/3 ¯¯ ∞ ≤t <∞ . 3 = te 2 R EMARK 1. The kernel in (5) is not in trace class, but the Fredholm determinant is well defined and we can easily conjugate it into a trace class kernel. Several kernels below are in similar situations. F IG . 2. ¯¯ ∞ . 1277 QUATERNIONIC WISHART In the real category, we have the result for the real white Whishart ensemble: P ROPOSITION 3. The distribution of the largest sample eigenvalue in the real white Wishart ensemble satisfies that, max(λ) almost surely approaches [15] (1 + γ −1 )2 with fluctuation scale M −2/3 , and [19] lim P max(λ) − (1 + γ −1 2 γ M 2/3 ) · ≤T (1 + γ )4/3 M→∞ = FGOE (T ), where FGOE is the GOE Tracy–Widom distribution. Here the function FGOE is defined by the Fredholm determinant of a matrix integral operator [30]: FGOE (T ) = det I − PGOE (ξ, η) and PGOE (ξ, η) = χ(T ,∞) (ξ ) S1 (ξ, η) SD1 (ξ, η) χ(T ,∞) (η), 1 IS1 (ξ, η) − 2 sgn(x − y) S1 (η, ξ, ) where S1 (ξ, η) = KAiry (ξ, η) − SD1 (ξ, η) = − IS1 (ξ, η) = − (6) ∞ Ai(t) dt + η ∂ 1 KAiry (ξ, η) − Ai(ξ ) Ai(η), ∂η 2 ∞ − R EMARK 2. 1 Ai(ξ ) 2 ξ 1 2 KAiry (t, η) dt + ∞ ξ Ai(t) dt + 1 2 1 2 ∞ Ai(t) dt ξ ∞ 1 Ai(ξ ), 2 ∞ Ai(t) dt η Ai(t) dt. η We have a more convenient form of FGOE [12]: FGOE = det 1 − KAiry (ξ, η) + s (1) (ξ ) Ai(η) χ(T ,∞) (η) , so [4] FGUE1 (T ) = (FGOE (T ))2 . In the real spiked model, Baik and Silverstein [8] compute the almost sure limit of the largest population eigenvalue, which is the same as that in the complex category, and Paul [25] proves the Gaussian distribution property in the case a > γ −1 , which is similar to (4). Neither of their methods can find the distribution function when a ≤ γ −1 . For the quaternionic white Wishart ensemble, we have: 1278 D. WANG P ROPOSITION 4. The distribution of the largest sample eigenvalue in the quaternionic white Wishart ensemble satisfies that, max(λ) almost surely approaches (1 + γ −1 )2 with fluctuation scale M −2/3 , and [14] −1 2 γ (2M)2/3 ≤T lim P max(λ) − (1 + γ ) · M→∞ (1 + γ )4/3 where FGSE is the GSE Tracy–Widom distribution. = FGSE (T ), Here the function FGSE is defined by the Fredholm determinant of a matrix integral operator [30]: FGSE (T ) = det I − P(ξ, η) and 4 (ξ, η) 4 (ξ, η) SD S P̂ (ξ, η) = χ(T ,∞) (ξ ) 4 (η, ξ, ) χ(T ,∞) (η), IS4 (ξ, η) S where 4 (ξ, η) = 1 KAiry (ξ, η) − 1 Ai(ξ ) S 2 4 (ξ, η) = − SD 4 ∞ Ai(t) dt, η 1 ∂ 1 KAiry (ξ, η) − Ai(ξ ) Ai(η), 2 ∂η 4 4 (ξ, η) = − 1 IS 2 ∞ ξ 1 KAiry (t, η) dt + 4 ∞ Ai(t) dt ∞ ξ Ai(t) dt. η 1.2. Statement of main results. The main theorem in this paper is: T HEOREM 1. In the rank 1 quaternionic spiked model: 1. If −1 < a < γ −1 , then the distribution of the largest sample eigenvalue is the same as that of the quaternionic white Wishart ensemble in Proposition 4. 2. If a = γ −1 , then the limit and the fluctuation scale are the same as those of the quaternionic white Wishart ensemble, but the distribution function is lim P M→∞ max(λ) − γ +1 γ 2 · γ (2M)2/3 ≤T (1 + γ )4/3 = FGSE1 (T ). 3. If a > γ −1 , then the limit and the fluctuation scale are changed as well as the distribution function, which is a Gaussian: √ 2M 1 ≤T · lim P max(λ) − (a + 1) 1 + 2 M→∞ γ a (a + 1) 1 − 1/(γ 2 a 2 ) = T 1 2 √ e−t /2 dt. −∞ 2π 1279 QUATERNIONIC WISHART Here the function FGSE1 is defined by the Fredholm determinant of a matrix integral operator: FGSE1 (T ) = det I − P (ξ, η) and S 4 (ξ, η) SD4 (ξ, η) P (ξ, η) = χ(T ,∞) (ξ ) χ(T ,∞) (η), IS4 (ξ, η) S 4 (η, ξ, ) where 1 4 (ξ, η), Ai(ξ ), SD4 (ξ, η) = SD 2 ∞ 1 ∞ 4 (ξ, η) − 1 IS4 (ξ, η) = IS Ai(t) dt + Ai(t) dt. 2 ξ 2 η 4 (ξ, η) + S 4 (ξ, η) = S Although the distribution FGSE1 seems to be new, we have that T HEOREM 2. FGSE1 (T ) = FGOE (T ). 1.3. Relation with other models and conjecture on the rank 1 real spiked model. The results of Theorems 1 and 2 give a phase transition pattern FGSE – FGOE –Gaussian as the parameter a increases from −1 to +∞. This pattern appears as limiting distributions indexed by a parameter in several other combinatorial and statistical physical models, for example, the lengths of the longest monotone subsequences of random involutions with condition on the number of fixed points [6] and the symmetrized last passage percolation [7] studied by Baik and Rains. In semi-infinite totally asymmetric simple exclusion process [26] studied by Prähofer and Spohn, and the symmetric polynuclear growth process [5] studied by Baik et al., 2-dimensional phase transition diagrams are obtained, and the 1-dimensional FGSE –FGOE –Gaussian pattern is contained in both of them. Although there is no model which can give hints to the rank 1 real spiked model, it is plausible that it has a phase transition from FGOE to Gaussian for the limiting distributions of the largest sample eigenvalue as a goes across γ −1 . Based on the duality of orthogonal and symplectic models from the Virasoro structure’s point of view, we have: C ONJECTURE 1. In the rank 1 real spiked model: 1. If −1 < a < γ −1 , then the distribution of the largest sample eigenvalue is the same as that of the real white Wishart ensemble in Proposition 3. 1280 D. WANG 2. If a = γ −1 , then the limit and the fluctuation scale are the same as those of the quaternionic white Wishart ensemble, but the distribution function is lim P M→∞ γ +1 max(λ) − γ 2 γ (M/2)2/3 · ≤T (1 + γ )4/3 = FGSE (T ). 3. If a > γ −1 , then the limit and the fluctuation scale are changed as well as the distribution function, which is a Gaussian (proved by Paul in [25]): √ M/2 1 ≤T · lim P max(λ) − (a + 1) 1 + 2 M→∞ γ a (a + 1) 1 − 1/(γ 2 a 2 ) = T 1 2 √ e−t /2 dt. −∞ 2π 1.4. Structure of the paper. In Section 2 we use combinatorial techniques to express the joint distribution function of {λj }, and then by skew orthogonal polynomial techniques express the distribution function of max(λj ) in the square root of a Fredholm determinant of a matrix integral operator. In Section 3 we do asymptotic analysis on the kernel of the matrix integral operator, and prove the three cases of Theorem 1 in the three subsections, respectively. Section 4 contains the proof of Theorem 2. In the proof of Theorem 1, we use some trace norm convergence results which generalize the old result on the LUE [11], and we give a method of proof to them in the Appendix. 2. The Fredholm determinantal formula. 2.1. The joint distribution function. In this subsection, we prove the following: T HEOREM 3. The joint probability distribution function of λ in the quaternionic spiked model is (7) P (λ) = N 2(M−N)+1 −2Mλ 1 4 j . λj e Ṽ (λ) C j =1 In this paper, C stands for any constants, and here 1 λ 1 λ21 .. Ṽ 4 (λ) = . 2N −2 λ 1 ea/(1+a)2Mλ1 0 1 2λ1 .. . −3 (2N − 2)λ2N 1 ··· ··· ··· ··· ··· λ2N−2 N a a/(1+a)2Mλ1 1+a 2Me ··· ea/(1+a)2MλN 1 λN λ2N .. . , a 2Mea/(1+a)2MλN 1+a 0 1 2λN .. . (2N − 2)λ2N−3 N 1281 QUATERNIONIC WISHART the determinant of a 2N × 2N matrix whose (2N, 2k − 1) entry is ea/(1+a)2Mλk , j −1 (j, 2k − 1) entry is λk for j = 1, . . . , 2N − 1, and 2ith column is the derivative of the (2i − 1)st column. Ṽ 4 (λ) is a variation of the V (λ)4 appearing in the LSE (see [23] and (9)). For the Wishart ensemble defined in the introduction section, we first have the distribution function for the sample matrix in the N × N positive definite quaternionic Hermitian matrix space [3]: P (S) = 1 −2M Tr( −1 S) (det S)2(M−N)+1 . e C R EMARK 3. Due to the noncommutativity of the quaternions, det S is not well defined in the usual way. Since S is quaternionic Hermitian, we can diagonalize it into a real-valued diagonal matrix by the conjugation of a quaternionic unitary matrix U , and define det S = eigenvalues of U SU ∗ . N R EMARK 4. In the distribution function in real and complex categories of sample matrices, we do not need to take the real part of the trace, since the trace is already real. Unfortunately, this does not hold in the quaternionic category due to its noncommutativity, and luckily Tr behaves better. [For example, Tr(AB) = Tr(BA), but Tr(AB) = Tr(BA) in general.] The distribution function for sample eigenvalues λ, the eigenvalues of S, is (8) N 1 λj2(M−N)+1 P (λ) = (V (λ))4 C j =1 e−2M Tr( −1 QQ−1 ) dQ, Q∈Sp(N) where we integrate on the compact symplectic group with the Haar measure, V (λ) = i<j (λi − λj ) is the Vandermonde, and = diag(λ1 , . . . , λN ). (See [23] for a derivation of the similar GSE case.) If the perturbation parameter a = 0, then l1 = l2 = · · · = lN = 1, e−2M Tr( Q∈Sp(N) −1 QQ−1 ) dQ = N e−2Mλj j =1 and (9) P (λ) = is the standard LSE [23]. N 2(M−N)+1 −2Mλ 1 j , λj e (V (λ))4 C j =1 1282 D. WANG Generally, e−2M Tr( −1 QQ−1 ) dQ Q∈Sp(N) = (10) = −1 ) e−2M Tr(I QQ e−2M Tr(( −1 −I )QQ−1 ) dQ Q∈Sp(N) N e−2Mλj e2M Tr((I − −1 )QQ−1 ) dQ. Q∈Sp(N) j =1 Then by the integral formula of the quaternionic Zonal polynomials [17], we get e2M Tr((I − −1 )QQ−1 ) dQ Q∈Sp(N) (11) = ∞ (2M)j j! j =0 Cκ(1/2) (I − −1 )Cκ(1/2) () (1/2) Cκ l(κ)≤N κj , (IN ) (1/2) where Cκ (x1 , . . . , xN ) is the N variable quaternionic Zonal polynomial, that is, the Jack polynomial with the parameter α = 1/2 (see [21] and [27]) and the Cnormalization [10], so that [κ = (k1 , . . . , kl ), k1 ≥ k2 ≥ · · · ≥ kl > 0, then l(κ) = l] Cκ(1/2) (x1 , . . . , xm ) = (x1 + · · · + xm )k . l(κ)≤m κk In the formula, a symmetric polynomial of a matrix is equivalent to the symmetric polynomial of its eigenvalues, so Cκ(1/2) (I − −1 ) = Cκ(1/2) a , 0, . . . , 0 . 1+a (1/2) Since all variables except for one vanish in Cκ (I − −1 ), we simply find Cκ(1/2) (I − −1 )l(κ)>1 = 0. (12) We have [27] (1/2) C(j ) a a , 0, . . . , 0 = 1+a 1+a j and since the number of variables is N [27] (1/2) C(j ) (1, . . . , 1) = j −1 1 (2N + i), (j + 1)! i=0 1283 QUATERNIONIC WISHART so with (11) and (12), we get e2M Tr((I − −1 )QQ−1 ) dQ Q∈Sp(N) −1 ∞ (2M)j C(j ) (I − )C(j ) () (1/2) = j! j =0 ∞ (1/2) (1/2) C(j ) (IN ) j +1 a 2M = j −1 1+a j =0 i=0 (2N + i) j (1/2) C(j ) (). In [27] there is an identity ∞ (1/2) (j + 1)C(j ) ()t j = j =0 N 1 . (1 − λj t)2 j =1 Comparing it with the well-known identity for Schur polynomials ∞ s(j ) ()t j = j =0 N 1 , 1 − λj t j =1 we get the identity (1/2) (j + 1)C(j ) () = s(j ) (λ1 , λ1 , λ2 , λ2 , . . . , λN , λN ), (13) with each λi appearing twice as variables of the s(j ) . For notational simplicity, we denote the right-hand side of (13) as s̃(j ) (), which is a plethysm [21] s̃(j ) () = s(j ) ◦ 2p1 (). Now we get e2M Tr((I − −1 )QQ−1 ) dQ Q∈Sp(N) (14) ∞ 1 a 2M = j −1 1+a j =0 i=0 (2N + i) j s̃(j ) (). Then we need a lemma to simplify (14) further. L EMMA 1. (15) 1 0 λ 1 1 2λ1 λ21 .. s̃(j ) () = .. . . 2N−2 2N−3 λ1 (2N − 2)λ 1 λ2N+j −1 (2N + j − 1)λ2N+j −2 1 1 × V (λ) −4 , ··· ··· ··· ··· ··· ··· 1 λN λ2N .. . λ2N−2 N 2N+j −1 λN 2N+j −2 (2N + j − 1)λ 0 1 2λN .. . (2N − 2)λ2N−3 N N 1284 D. WANG with the (k, 2j − 1) entry of the matrix being a power of λj with the exponent k − 1 if k = 2N and 2N + j − 1 if k = 2N , and the (k, 2j ) entry being the derivative of the (k, 2j − 1) entry with respect to λj . To prove this lemma, we need the well-known fact (see [23]), proven by L’Hôpital’s rule 1 λ1 4 V (λ) = .. . 2N−1 λ (16) 1 ··· ··· 0 1 .. . (2N 1 λN .. . 0 1 .. . ··· 2N−1 · · · λN − 1)λ12N−2 , 2N−2 (2N − 1)λN with the (k, 2j − 1) entry being λk−1 and the (k, 2j ) entry (k − 1)λk−2 j j . P ROOF OF L EMMA 1. Applying the L’Hôpital’s rule repeatedly with respect to x2i , i = 1, . . . , N , we get the identity 1 λ1 0 1 ··· ··· 1 λN λ21 . . . 2λ1 . . . ··· ··· λ2N . . . −2 λ2N 1 −3 (2N − 2)λ2N 1 ··· −2 λ2N N 2N +j −1 λ1 (2N + j 2N +j −2 − 1)λ1 ··· 2N +j −1 0 1 . . . ··· ··· ··· 1 λN . . . −1 λ2N 1 (2N − 1)λ2N −2 ··· −1 λ2N N N ⎜ ∂ ⎜ =⎜ ⎝ ∂x2 ∂x4 · · · ∂x2N (2N + j − 1)λN λN 1 λ1 . . . ⎛ 2λN . . . −3 (2N − 2)λ2N N 2N +j −2 0 1 2N −2 0 1 . . . (2N − 1)λN 1 x1 . . . 1 x2 . . . ··· ··· ··· 1 x2N −1 . . . x12N −2 x22N −2 ··· 2N +j −1 x1 2N +j −1 x2 2N −2 x2N −1 N ∂ ∂x2 ∂x4 · · · ∂x2N 2N +j −1 · · · x2N −1 1 x1 . . . 1 x2 . . . ··· ··· ··· 1 x2N −1 . . . x12N −2 x22N −2 ··· 2N +j −1 x1 2N +j −1 x2 2N −2 x2N −1 ··· x2N 2N +j −1 x2N −1 2N −2 x2N 2N +j −1 1 x2N . . . ⎞ 1 ⎟ ⎟ x2N ⎟ ⎟ . . ⎟ . ⎟ 2N −2 ⎟ x2N ⎟ 2N +j −1 ⎠ x2N x2i−1 = x2i = λi i = 1, . . . , N = s(j ) (λ1 , λ1 , λ2 , λ2 , . . . , λN , λN ) = s̃(j ) (), from the matrix representation of Schur polynomials, and now use (16) to get the compact formula (15). 1285 QUATERNIONIC WISHART Substituting (15) into (14), we get 4 e2M Tr((I − V (λ) −1 )QQ−1 ) Q∈Sp(N) 0 ··· 1 1 λ1 1 ··· λN λ2 2λ1 ··· λ2N 1 . . = .. . . . . ··· . λ2N −2 (2N − 2)λ2N −3 · · · λ2N −2 1 N 1 p(λ ) p (λ1 ) · · · p(λN ) 1 1 0 λ1 1 2 λ 2λ 1 1 1 . . = . . . . C λ2N −2 −3 (2N − 2)λ2N 1 1 ea/(1+a)2Mλ1 a 2Mea/(1+a)2Mλ1 (17) 1+a = dQ 2λN . . . 2N −3 (2N − 2)λN p (λ ) 0 1 N ··· ··· 1 λN 0 1 ··· λ2N . . . 2λN . . . −2 λ2N N a/(1+a)2Mλ N e −3 (2N − 2)λ2N N a a/(1+a)2Mλ N 1+a 2Me ··· ··· ··· 1 4 Ṽ (λ), C where ∞ 1 a 2M p(x) = j −1 1+a j =0 i=0 (2N + i) j x 2N+j −1 2N−2 1 (2N − 1)! a a/(1+a)2Mx = e − 2Mx 2N−1 (a/(1 + a)2M) j! 1 + a j =0 j , and if k = 2N , the (k, 2j − 1) entries in both matrices are λk−1 j , and the (k, 2j ) k−2 entries are (k − 1)λj , and the 2N, 2i − 1 entry in the former (latter) matrix is p(λi ) (resp. ea/(1+a)2Mλi ) and the 2N, 2i entry p (λi ) (resp. P ROOF OF T HEOREM 3. sult (7). a a/(1+a)2Mλi ). 1+a 2Me Formulas (8), (10) and (17) together give the re- 2.2. The Pfaffian and determinantal formulas. With the formula (7) ready to use, we apply the standard RMT technique to get the distribution formula for the largest sample eigenvalue, in the same spirit as the solution of the LSE. Our process below is closely parallel to that in [31] to the LSE. First, we find a skew orthogonal basis {ϕ0 (x), ϕ1 (x), . . . , ϕ2N−1 (x)} of the linear space spanned by {1, x, x 2 , . . . , x 2N−2 , ea/(1+a)2Mx }. We require that the ϕj (x) is a linear combination of {1, x, x 2 , . . . , x j } if j < 2N − 1, while ϕ2N−1 (x) can be 1286 D. WANG arbitrary, with the skew inner products among them ϕj (x), ϕk (x)4 = = ∞ 0 ϕj (x)ϕk (x) − ϕj (x)ϕk (x) x 2(M−N)+1 e−2Mx dx ⎧ ⎨ rj/2 , ⎩ if j is even and k = j + 1, if k is even and j = k + 1, otherwise. −rk/2 , 0, Then we can reformulate the distribution function of λ as 1 P (λ) = C × (18) ϕ0 (λ1 ) ϕ1 (λ1 ) .. . ϕ2N−1 (λ1 ) ϕ0 (λ1 ) ϕ1 (λ1 ) .. . ϕ2N−1 (λ1 ) ··· ··· ϕ0 (λN ) ϕ1 (λN ) .. . ϕ0 (λN ) ϕ1 (λN ) .. . ··· · · · ϕ2N−1 (λN ) ϕ2N−1 (λN ) N 2(M−N)+1 −2Mλ j λ e j =1 1 = C j ψ0 (λ1 ) ψ1 (λ1 ) .. . ψ2N−1 (λ1 ) ψ0 (λ1 ) ψ1 (λ1 ) .. . ψ2N−1 (λ1 ) ··· ··· ψ0 (λN ) ψ1 (λN ) .. . ··· · · · ψ2N−1 (λN ) ψ0 (λN ) ψ1 (λN ) .. . ψ2N−1 (λN ) , where ψi (x) = ϕi (x)x M−N+1/2 e−Mx . (19) For an arbitrary function f (x) on [0, ∞), by the formula of de Bruijn [9], ∞ ∞ ··· 0 0 (20) ψ0 (λ1 ) ψ1 (λ1 ) .. . ψ2N−1 (λ1 ) × N ψ0 (λ1 ) ψ1 (λ1 ) .. . ψ2N−1 (λ1 ) ··· ··· ψ0 (λN ) ψ1 (λN ) .. . ··· · · · ψ2N−1 (λN ) ψ0 (λN ) ψ1 (λN ) .. . ψ2N−1 (λN ) 1 + f (λi ) dλi = C Pf P (1 + f ) , i=1 where P (1 + f ) is a 2N × 2N matrix, whose entries depend on 1 + f in the following way: P (1 + f ) j,k = ∞ 0 ψj −1 (x)ψk−1 (x) − ψj −1 (x)ψk−1 (x) 1 + f (x) dx. 1287 QUATERNIONIC WISHART Now we define a matrix Z as ⎛ 0 ⎜ −r0 ⎜ ⎞ r0 0 ⎜ ⎜ ⎜ Z=⎜ ⎜ ⎜ ⎜ ⎝ 0 −r1 r1 0 .. . 0 −rN−1 with Zj,k = ⎧ ⎨ rk/2−1 , , −r ⎩ j/2−1 0, rN−1 0 ⎟ ⎟ ⎟ ⎟ ⎟ ⎟, ⎟ ⎟ ⎟ ⎠ if k is even and j = k − 1, if j is even and k = j − 1, otherwise, and define for j = 0, . . . , N − 1, η = Z −1 ψ, that is, η2j (x) = − So we have P (1 + f ) j,k = ψ2j +1 (x) rj and η2j +1 (x) = ψ2j (x) . rj ∞ 0 + ψj −1 (x)ψk−1 (x) − ψj −1 (x)ψk−1 (x) dx ∞ 0 = Zj,k + ψj −1 (x)ψk−1 (x) − ψj −1 (x)ψk−1 (x) f (x) dx ∞ 0 ψj −1 (x)ψk−1 (x) − ψj −1 (x)ψk−1 (x) f (x) dx. And if we denote Q(1 + f ) = Z −1 P (1 + f ), then Q(1 + f )j,k = δj,k + ∞ 0 ηj −1 (x)ψk−1 (x) − ηj −1 (x)ψk−1 (x) f (x) dx. If we choose f to be −χ(T ,∞) , then the integral on the left-hand side of (20), after multiplying a constant, is the probability of all λi ’s smaller than T . In latter part of the paper, we abbreviate χ(T ,∞) to χ . So we get for a T -independent constant P max(λi ) ≤ T = C Pf P (1 − χ ) , and P max(λi ) ≤ T 2 In linear algebra, we have the determinant identity (21) = C 2 det P (1 − χ ) = C 2 det Q(1 − χ ) . det(I − AB) = det(I − BA), 1288 D. WANG for A an m × n matrix and B an n × m matrix, but the identity still holds in infinite dimensional settings [16]. Letting det mean a Fredholm determinant for matrix integral operators, we describe a setting due to Tracy–Widom [31]. If A is an operator from L2 ([0, ∞)) × L2 ([0, ∞)) to the vector space R2N with A g(x) h(x) j = ∞ 0 χ (x)ηj −1 (x)g(x) dx − ∞ 0 χ (x)ηj −1 (x)h(x) dx, and B is an operator from R2N to L2 ([0, ∞)) × L2 ([0, ∞)) with ⎛ ⎛ 2N ⎞ ck ψk−1 (x)χ(x) ⎟ ⎜ ⎟ ⎟ ⎜ k=1 ⎜ ⎟, ⎠ = ⎜ 2N ⎟ ⎝ ⎠ ⎞ c1 ⎜ . B ⎝ .. c2N ck ψk−1 (x)χ(x) k=1 then I − AB = Q(1 − χ ) and S4 (x, y) SD4 (x, y) I − BA = I − χ (x) χ (y), IS4 (x, y) S4 (y, x) where S4 (x, y), IS4 (x, y) and SD4 (x, y) are integral operators whose kernels are S4 (x, y) = 2N−1 j =0 = N−1 j =0 SD4 (x, y) = 1 −ψ2j (x)ψ2j +1 (y) + ψ2j +1 (x)ψ2j (y) , rj 2N−1 j =0 (22) = N−1 j =0 IS4 (x, y) = ψj (x)ηj (y) −ψj (x)ηj (y) 1 ψ2j (x)ψ2j +1 (y) − ψ2j +1 (x)ψ2j (y) , rj 2N−1 ψj (x)ηj (y) j =0 = N−1 j =0 1 −ψ2j (x)ψ2j +1 (y) + ψ2j +1 (x)ψ2j (y) , rj 1289 QUATERNIONIC WISHART S4 (y, x) = 2N−1 j =0 (23) = N−1 j =0 −ψj (x)ηj (y) 1 ψ2j (x)ψ2j +1 (y) − ψ2j +1 (x)ψ2j (y) . rj R EMARK 5. It is clear that the nomenclature of SD4 (x, y) is due to the fact that SD4 (x, y) is the negative of the derivative of S4 (x, y). But IS4 (x, y), which gets its name in the same way in earlier literature in GSE (e.g., [30]), in our problem may not satisfy the equation IS4 (x, y) = − ∞ x S4 (t, y) dt, since the integral on the right-hand side may diverge. In conclusion, 2 P max(λi ) ≤ T S4 (x, y) SD4 (x, y) = C det I − χ (x) χ (y) , IS4 (x, y) S4 (y, x) 2 and we can find that C 2 = 1 by taking the limit T → ∞. We define a 2 × 2 matrix kernel as PT (x, y) = χ (x) = S4 (x, y) SD4 (x, y) χ (y) IS4 (x, y) S4 (y, x) χ (x)S4 (x, y)χ(y) χ(x)DS4 (x, y)χ(y) , χ (x)IS4 (x, y)χ(y) χ(x)S4 (y, x)χ(y) then we have P max(λi ) ≤ T 2 = det I − PT (x, y) . 2.3. S4 (x, y) in terms of Laguerre polynomials. In manipulation of skew orthogonal polynomials, we take the approach of [1], and all classical orthogonal polynomial properties are from [28]. Since Laguerre polynomials by definition satisfy the orthogonal property ∞ 0 (α) α −x L(α) dx = j Lk x e (j + α)! δj,k , j! (α) and they have the differential identity [we assume Ln (x) = 0 if n < 0] (24) x d (α) (α) L (x) = nL(α) n (x) − (n + α)Ln−1 (x), dx n 1290 D. WANG it is easy to get that (2(M−N)) Lj = (2Mx), L(2(M−N)) (2Mx) 4 k ∞ 0 d (2(M−N)) L (2Mx) dx k d (2(M−N)) (2(M−N)) − Lk (2Mx) Lj (2Mx) dx (2(M−N)) Lj (2Mx) × x 2(M−N)+1 e−2Mx dx ⎧ ⎪ 1 2(M−N)+1 (j + 2(M − N))! ⎪ ⎪ , ⎪ ⎪ ⎨ 2M (j − 1)! = 1 2(M−N)+1 (k + 2(M − N))! ⎪ ⎪ − , ⎪ ⎪ ⎪ 2M (k − 1)! ⎩ 0, if j = k + 1, if k = j + 1, otherwise. So we can choose for j = 0, . . . , N − 2, ϕ2j (x) = (25) j k k=0 (2(M−N)) ϕ2j +1 (x) = −L2j +1 (26) 2i − 1 (2(M−N)) L2k (2Mx), 2i + 2(M − N) i=1 (2Mx) and (27) rj = 1 2M 2(M−N)+1 We can also choose ϕ2N−2 (x) = j (2j + 2(M − N) + 1)! 2k − 1 . (2j )! 2k + 2(M − N) k=1 N−1 k k=0 2i − 1 L(2(M−N)) (2Mx), 2k 2i + 2(M − N) i=1 but ϕ2N−1 (x) is not a polynomial and needs to be treated separately. By the Rodrigues’ representation 1 d n −x n+α (e x ), n! dx n and repeated integration by parts, we get for n > 0 x α e−x L(α) n (x) = a/(1+a)2Mx (2(M−N)) e , Ln (2Mx) 4 2(M−N)+1 = 1+a 2M (n + 2(M − N) + 1)! n! (n + 2(M − N))! − (−a)n−1 (n − 1)! (−a)n+1 1291 QUATERNIONIC WISHART and a/(1+a)2Mx (2(M−N)) 1 + a 2(M−N)+1 e , L0 (2Mx) 4 = − a 2(M − N) + 1 !, 2M so that a/(1+a)2Mx e , ϕ2j (x) 4 =− and 1+a 2M 2(M−N)+1 a 2j +1 j (2j + 2(M − N) + 1)! 2k − 1 (2j )! 2k + 2(M − N) k=1 a/(1+a)2Mx e , ϕ2j +1 (x) 4 2(M−N)+1 =− 1+a 2M a 2j +2 (2j + 2(M − N) + 2)! (2j + 1)! (2j + 2(M − N) + 1)! −a . (2j )! Now by the skew orthogonality, we can choose 2j ϕ2N−1 (x) = ea/(1+a)2Mx − N−2 j =0 1 a/(1+a)2Mx e , ϕ2j +1 (x) 4 ϕ2j (x) rj − ea/(1+a)2Mx , ϕ2j (x) 4 ϕ2j +1 (x) − (1 + a)2(M−N)+1 a 2N−2 N−1 j =1 =e a/(1+a)2Mx − (1 + a) 2j + 2(M − N) ϕ2N−2 (x) 2j − 1 2(M−N)+1 2N−2 j =0 and rN−1 = 1+a 2M 2(M−N)+1 a 2N−1 (2(M−N)) (−a)j Lj (2Mx) (2M − 1)! N−1 2k − 1 . (2N − 2)! k=1 2k + 2(M − N) Now, we write S4 (x, y) as S4a (x, y) + S4b (x, y), where (28) S4a (x, y) = N−2 j =0 1 −ψ2j (x)ψ2j +1 (y) + ψ2j +1 (x)ψ2j (y) rj and (29) S4b (x, y) = 1 −ψ2N−2 (x)ψ2N−1 (y) + ψ2N−1 (x)ψ2N−2 (y) , rN−1 and simplify them separately. 1292 D. WANG The formula (28) of our S4a (x, y) is also the formula for S4 (x, y) in the LSE problem, with parameters M and N − 2, and has been well studied. For completeness we derive its Laguerre polynomial expression here, following [1]. By the differential identity (24) and the identity (α) (α) nL(α) n (x) = (−x + 2n + α − 1)Ln−1 (x) − (n + α − 1)Ln−2 (x), we get, remembering the definition (19), the telescoping sequence ψ2j (x) = j k k=0 2i − 1 2i + 2(M − N) i=1 d (2(M−N)) × M − N + 1/2 − Mx + x L2k (2Mx) dx × x M−N−1/2 e−Mx j = (30) k 2i − 1 1 (2(M−N)) (2k + 1)L2k+1 (2Mx) 2 k=0 i=1 2i + 2(M − N) (2(M−N)) − 2k + 2(M − N) L2k−1 × x M−N−1/2 e−Mx j 1 2k − 1 = 2 k=1 2k + 2(M − N) × (2j + 1)L(2(M−N)) (2Mx)x M−N−1/2 e−Mx 2j +1 and ψ2j +1 (x) = − M − N + 1/2 − Mx + x d (2(M−N)) L dx 2j +1 × (2Mx)x M−N−1/2 e−Mx 1 (2(M−N)) = − (2j + 2)L2j +2 (2Mx) 2 (31) (2(M−N)) − 2j + 2(M − N) + 1 L2j (2Mx) × x M−N−1/2 e−Mx . Therefore, if we substitute (27), (30) and (31) into (28), we get after some trick, 1 S4a (x, y) = (2M)2(M−N)+1 x M−N−1/2 e−Mx y M−N+1/2 e−My 2 × !2N−2 j =0 (2Mx) j! (2(M−N)) (2(M−N)) Lj (2Mx)Lj (2My) (j + 2(M − N))! 1293 QUATERNIONIC WISHART 2j + 2(M − N) (2N − 2)! N−1 − (2M − 2)! j =1 2j − 1 " × L(2(M−N)) (2Mx)ϕ2N−2 (y) 2N−2 . Furthermore, we can simplify ψ2N−2 (x). Since for j = 2N − 1 [if we define ϕj (x) and then ψj (x) for j > 2N − 1 by the formula (25) and (26)] ∞ 0 ψ2N−2 (x)ψj (x) − ψ2N−2 (x)ψj (x) dx = 0, we get for j = 2N − 1, using integration by parts, ∞ 0 ψ2N−2 (x)Lj (2(M−N)) (2Mx)x M−N+1/2 e−Mx dx = 0. So by the orthogonal property of Laguerre polynomials, we get ψ2N−2 (x) = CL(2(M−N)) (2Mx)x M−N−1/2 e−Mx , 2N−1 and we can determine that 2j − 1 2N − 1 N−1 2 2j + 2(M − N) j =1 C= without much difficulty. Together with the fact limx→∞ ψ2N−2 (x) = 0, we get ψ2N−2 (x) = − 2j − 1 2N − 1 N−1 2 2j + 2(M − N) j =1 × ∞ x t M−N−1/2 e−Mt L(2(M−N)) (2Mt) dt. 2N−1 Now, we can write S4a (x, y) as S4a1 (x, y) + S4a2 (x, y), where 1 S4a1 (x, y) = (2M)2(M−N)+1 2 (32) × 2N−2 j =0 j! (2(M−N)) Lj (2Mx)x M−N−1/2 e−Mx (j + 2(M − N))! (2My)y M−N+1/2 e−My × L(2(M−N)) j and 1 (2N − 1)! S4a2 (x, y) = (2M)2(M−N)+1 4 (2M − 2)! (33) (2(M−N)) × L2N−2 × ∞ y (2Mx)x M−N−1/2 e−Mx t M−N−1/2 e−Mt L(2(M−N)) (2Mt) dt. 2N−1 1294 D. WANG Finally, 1 2M S4b (x, y) = − 2 1+a (2(M−N)) × L2N−1 (34) 2(M−N)+1 a −(2N−1) (2N − 1)! (2M − 1)! (2Mx)x M−N−1/2 e−Mx ψ2N−1 (y) + ψ2N−1 (x) ∞ y (2(M−N)) L2N−1 (2Mt)t M−N−1/2 e−Mt dt , and we can take the asymptotic analyses of S4a1 (x, y), S4a2 (x, y) and S4b (x, y) separately. 3. Asymptotic analysis. In order to consider the rescaled distribution problem, we wish to find the probability of the largest sample eigenvalue being in the domain (0, p + qT ]. We can put the kernel in the new coordinate system [after a conjugation by q 1/2 0 0 ], q −1/2 P max(λi ) ≤ p + qT 2 and get $ 4 (ξ, η) # (ξ, η) SD S = det I − χ (x) #4 #4 (η, ξ ) χ (η) IS4 (ξ, η) S = det I − P#T (ξ, η) , where as L2 functions, $ 4 (ξ, η) = q 2 SD4 (x, y)|x=p+qξ , SD (35) y=p+qη #4 (ξ, η) = qS4 (x, y)|x=p+qξ , S (36) y=p+qη # 4 (ξ η) = IS4 (x, y)|x=p+qξ IS (37) y=p+qη and $ 4 (ξ, η) # (ξ, η) SD S P#T (ξ, η) = χ (ξ ) #4 #4 (η, ξ ) χ (η). IS4 (ξ, η) S In this section, we want to prove that for fixed γ ≥ 1 and a > −1, we can choose suitable pM and qM depending on M, so that for any T , lim P max(λi ) ≤ pM + qM T M→∞ 2 = lim det I − P#T (ξ, η) = fa (T ), M→∞ where fa is a function to be determined. To prove the convergence of Fredholm determinants, we may use that P#T (ξ, η) is in trace class for any M and converges to a certain 2 × 2 matrix kernel in trace norm. Equivalently, we may use that each entry of P#T (ξ, η) is in trace class and converges to a scalar kernel in trace norm. It turns out later that the P#T (ξ, η)’s may not satisfy these requirements, but certain conjugates do. 1295 QUATERNIONIC WISHART Since the IS4 (x, y) and DS4 (x, y) are of the same form as S4 (x, y), we only show the asymptotic analysis of S4 (x, y), and state the result for the other two, for which the arguments are the same. 3.1. Proof of the −1 < a < γ −1 part of Theorem 1. In case −1 < a ≤ γ −1 , )4/3 , and denote [here ∗ stands for 4, we choose pM = (1 + γ −1 )2 and qM = γ(1+γ (2M)2/3 #∗ (ξ, η) in (38) is only used in Sections 3.1 4a, 4a1, 4a2 and 4b; the definition of S and 3.2] (38) #∗ (ξ, η) = S (1 + γ )4/3 S∗ (x, y)|x=(1+γ −1 )2 +(1+γ )4/3 /(γ (2M)2/3 )ξ . γ (2M)2/3 y=(1+γ −1 )2 +(1+γ )4/3 /(γ (2M)2/3 )η S4a (x, y) is the formula for the upper-left entry of the 2 × 2 matrix kernel of the LSE problem with parameters M and N − 1, and its asymptotic behavior is well studied [14]. We want to prove that as M → ∞, S4a (x, y) dominates S4 (x, y) in the domain that we are interested in, and so naturally the distribution of the largest sample eigenvalue in the perturbed problem is the same as that in the LSE problem. (The difference between N and N − 1 is negligible.) S4a1 (x, y) is almost the kernel for √ the LUE problem with parameters 2M − 2 and 2N − 2, besides a factor y/x/2. From a standard result for LUE #4a1 (ξ, η)χT (η) is in trace class and converges in trace norm to half of [11], χT (ξ )S the Airy kernel #4a1 (ξ, η)χ(η) = 1 χ (ξ )KAiry (ξ, η)χ(η). lim χ (ξ )S 2 More discussion see the Appendix. For the S4a2 (x, y) part, we also have in trace norm [14], (39) (40) M→∞ #4a2 (ξ, η)χ(η) = − 1 χ (ξ ) Ai(ξ ) lim χ (ξ )S 4 M→∞ ∞ Ai(t) dt χ(η). η #4a2 (ξ, η) is a rank 1 operator, for the trace norm We just sketch the proof. Since S convergence, we only need to prove that in L2 norm as functions in ξ and respectively η, lim γ −2N (1 + γ )4/3 (2M)1/3 eM−N M→∞ (2(M−N)) × L2N−2 (41) (2Mx)x M−N−1/2 e−Mx χ (ξ ) = Ai(ξ )χ(ξ ), lim γ −2N M→∞ (42) =− 2Me ∞ η M−N ∞ y L(2(M−N)) (2Mt)t M−N−1/2 e−Mt dt χ(η) 2N−1 Ai(t) dt χ(η) 1296 D. WANG and by the Stirling’s formula, lim (2M)2(M−N)−1 M→∞ (2N − 1)! 2(N−M) 4N−1 e γ = 1. (2M − 2)! By (33), (38), (41) and (42), we get #4a2 (ξ, η)χ(η) = 1 (2M)2(M−N)−1 (2N − 1)! e2(N−M) γ 4N−1 γ −2N χ (ξ )S 4 (2M − 2)! × (1 + γ )4/3 (2M)1/3 eM−N L(2(M−N)) (2Mx) 2N−2 × x M−N−1/2 e−Mx χ (ξ )γ −2N 2MeM−N × ∞ y (2(M−N)) L2N−1 (2Mt)t M−N−1/2 e−Mt dt χ(η). Therefore we get the trace norm convergence from the L2 convergence by the fact that if fn (x) → f (x) and gn (y) → g(y) in L2 norm, then we have the convergence of integral operators in trace norm: fn (x)gn (y) → f (x)g(y). Finally, we need to analyze the term S4b (ξ, η), new to the perturbed problem. We need the following results: P ROPOSITION 5. For fixed γ ≥ 1 and −1 < a < γ −1 and any T , we have the convergences in L2 norm with respect to ξ or η: lim γ −2N−1 (1 + γ )4/3 (2M)1/3 eM−N M→∞ (2(M−N)) × L2N−1 (2Mx)x M−N−1/2 e−Mx χ (ξ ) = − Ai(ξ )χ(ξ ), lim γ −2N 2MeM−N M→∞ =− ∞ ∞ y (2(M−N)) L2N−1 Ai(t) dt χ(η), η lim (1 + a)2(N−M)−1 a −2N+1 (43) (2Mt)t M−N−1/2 e−Mt dt χ(η) M→∞ (1 − aγ )(2M)1/3 M−N e ψ2N−1 (y)χ(η) (γ + 1)2/3 γ 2N−1 = Ai(η)χ(η), lim (1 + a)2(N−M)−1 a −2N+1 M→∞ = Ai (ξ )χ(ξ ). (1 − aγ )(γ + 1)2/3 M−N e ψ2N−1 (x)χ(ξ ) γ 2N (2M)1/3 1297 QUATERNIONIC WISHART P ROOF. We just prove the identity (43), and others can be done in the same way. By the integral representation of Laguerre polynomials, % e−2Mxz (z + 1)n+2(M−N) 1 dz, 2πi C zn+1 where C is a contour around the pole 0, therefore we get L(2(M−N)) (2Mx) = n (44) ϕ2N−1 (y) = ea/(1+a)2My − (1 + a)2(M−N)+1 2πi × % e−2Myz C (45) = ea/(1+a)2My − − (1 + a)2(M−N) 2πi % e−2Myz C (z + 1)2(M−N) dz z + a/(a + 1) (1 + a)2(M−N)+1 a 2N−1 2πi × If the pole 1 + (a(z + 1)/z)2N−1 (z + 1)2(M−N) dz 1 + a(z + 1)/z z a z = − a+1 % e−2Myz C (z + 1)2M z dz. 2N z ((a + 1)z + a)(z + 1) is inside of C, then % (1 + a)2(M−N) 2πi e−2Myz C (z + 1)2(M−N) dz = ea/(a+1)2My z + a/(a + 1) and ϕ2N−1 (y) = − (46) (1 + a)2(M−N)+1 a 2N−1 2πi × % C e−2Myz (z + 1)2M z dz. 2N z ((a + 1)z + a)(z + 1) In later part of the proof, we make this condition hold, and will not mention the canceled terms, and we are then free to deform C in (46) as we wish, provided it includes 0. We then proceed to a stationary phase analysis. Since 4/3 (z + 1)2M 2M(−(1+γ −1 )2 z+log(z+1)−γ −2 log z)− (1+γγ) (2M)1/3 ηz = e z2N (we do not need to concern ourselves about the ambiguity of the value of logarithmic functions), if we denote e−2Myz (47) f (z) = −(1 + γ −1 )2 z + log(z + 1) − γ −2 log z, 1298 D. WANG then we get: −1 )z+γ • f (z) = − ((1+γz(z+1) −1 ) 1 ) = 0; • f (− γ +1 1 • f (− γ +1 )= 2(γ +1)4 γ3 1 , with the zero point z = − γ +1 ; > 0. 1 , So locally around z = − γ +1 f − (48) 1 γ +1 +w = + log γ − (1 − γ −2 ) log(γ + 1) + γ −2 πi γ +1 γ2 + (γ + 1)4 3 w + R1 (w), 3γ 3 where R1 (w) = O(w4 ), (49) After the substitution w = z + % e2M(−yz+log(z−1)−γ C = % exp 2M M −2 log z) 1 γ +1 , as w → 0. we get z dz ((a + 1)z + a)(z + 1) γ +1 + log γ − (1 − γ −2 ) log(γ + 1) + γ −2 πi γ2 + w − 1/(γ + 1) dw ((a + 1)w + (aγ − 1)/(γ + 1))(w + γ /(γ + 1)) × =− (γ + 1)4 3 (1 + γ )4/3 1 w + R (w) − η w− 1 3 2/3 3γ γ (2M) γ +1 γ 2M−1 1 e2M/(1+γ )y a + 1 (γ + 1)2(M−N) × % M exp × −(1 + γ )4/3 (1 + γ )4 (2M)1/3 ηw + 2Mw3 + 2MR1 (w) γ 3γ 3 1 −(γ + 1)w + 1 dw, (γ + 1)/γ w + 1 w + (aγ − 1)/((γ + 1)(a + 1)) where M is a contour around are defined as (see Figure 3) 1 γ +1 , composed of 1M , 2M , 3M and 4M , which 1M γ 1 = (4 − t) (2M)−1/3 , eπ i/3 0 ≤ t ≤ 4 − γ +1 (1 + γ )1/3 2M 1 γ −1/3 −tπ i 1 = (2M) e , − ≤ t ≤ 4/3 (1 + γ ) 3 3 1299 QUATERNIONIC WISHART F IG . 3. M . 1 γ (2M)−1/3 ≤ t ≤ 4 , e5π i/3 γ +1 (1 + γ )1/3 √ √ γ γ γ + it −2 3 ≤t ≤2 3 = 2 . γ +1 γ +1 γ +1 3M = t 4M For the asymptotic analysis, we define (50) (51) M local = {z ∈ M |(z) ≤ (2M)−10/39 }, ∞ <c = {w ∈ ∞ |(w) < c}, M M remote = (1M ∪ 3M ) \ local , ∞ ∞ ≥c = ∞ \ <c . Now, we denote FaM (η, w) = (1 + γ )4/3 γ (2M)1/3 (1 + γ )4/3 (1 + γ )4 (2M)1/3 ηw + × exp − 2Mw3 + 2MR1 (w) γ 3γ 3 × 1 −(γ + 1)w + 1 , (γ + 1)/γ w + 1 w + (aγ − 1)/((γ + 1)(a + 1)) and establish several lemmas for the proof. L EMMA 2. If T is fixed and M is large enough, then for any η > T , 1 2πi 4M FaM (η, w) dw < 1 e−η/2 . 3 M 1/40 1300 D. WANG P ROOF. By (48) and (47), (1 + γ )4 2Mw3 + 2MR1 (w) 3γ 3 = 2M f − 1 γ +1 +w − γ +1 γ2 − log γ + (1 − γ −2 ) log(γ + 1) − γ −2 πi (52) γ +1 (γ + 1)2 = 2M − w + log w+1 2 γ γ −γ −2 log (γ + 1)w − 1 − γ −2 πi . If w ∈ 4M , (w) = 2 γ γ+1 , and denote θ = arg(w) ∈ [− π3 , π3 ], we have (1 + γ )4 2Mw3 + 2MR1 (w) 3γ 3 = 2M −2 γ +1 + log 32 + (2 tan θ )2 γ − γ −2 log (1 + 2γ )2 + (2γ tan θ )2 √ √ γ +1 + log 21 − γ −2 log(1 + 2γ ) < log 21 − 2 2M < 0. γ √ So on 4M , if η ≥ T , 0 < ε < 2 − log 21 and M large enough, ≤ 2M −2 |FaM (η, w)| < (1 + γ )4/3 (2M)1/3 γ × exp −2(η − T )(1 + γ )1/3 (2M)1/3 √ + (log 21 − 2) − 2T (1 + γ )−2/3 2M −(γ + 1)w + 1 × < e−2(η−T )(1+γ ) 1/3 (2M)1/3 √ e(log 21−2+ε )2M , √ where ε is a positive number and ε < 2 − log 21. If M is large enough, e(log (53) 1 (γ + 1)/γ w + 1 w + (aγ − 1)/((γ + 1)(a + 1)) √ 21−2+ε )2M e−2(η−T )(1+γ ) 1/3 (2M)1/3 2π 1 e−T /2 < √ , 2 3γ /(γ + 1) 3 M 1/40 < eT /2 e−η/2 , 1301 QUATERNIONIC WISHART and we get the result, since √ 2 3γ /(γ + 1) max |FaM (η, w)|. FaM (η, w) dw ≤ 2π 4M w∈4M 1 2πi (54) L EMMA 3. If T is fixed and M is large enough, then for any η > T , 1 2πi 1 e−η/2 FaM (η, w) dw < . M 3 M 1/40 remote M P ROOF. For w ∈ remote , we denote l = (w) = get by (52) |w| 2 . Since arg(w) = ± π3 , we (1 + γ )4 2Mw3 + 2MR1 (w) 3γ 3 γ +1 (γ + 1)2 1 γ +1 l+4 l = 2M − l + log 1 + 2 γ2 2 γ γ 2 − γ −2 log 1 − 2(γ + 1)l + 4(γ + 1)2 l 2 . 2 Then we take derivative d γ +1 (γ + 1)2 1 γ +1 l+4 l − l + log 1 + 2 dl γ2 2 γ γ 2 − γ −2 log 1 − 2(γ + 1)l + 4(γ + 1)2 l 2 2 γ +1 (γ + 1)4 2 γ +1 l+4 l = −8 l 1 − (γ − 1) γ3 γ γ × 1+2 γ +1 γ +1 l+4 l γ γ 2 2 −1 2 2 × 1 − 2(γ + 1)l + 4(γ + 1) l , and are able to find a positive number ε > 0, such that for 0 < l ≤ 2 γ γ+1 , −8 (γ + 1)4 2 l γ3 × 1 − (γ − 1)(γ + 1)/γ l + 4((γ + 1)/γ l)2 (1 + 2(γ + 1)/γ l + 4((γ + 1)/γ l)2 )(1 − 2(γ + 1)l + 4(γ + 1)2 l 2 ) < 3ε l 2 , 1302 D. WANG M and two left-most points of remote , (1 + √ on the−10/39 3i)(2M) , √ 3i)(2M)−10/39 and (1 − (1 + γ )4 3 2Mw + 2MR(w) √ 3γ 3 w=(1± 3i)M −10/39 = 2M − =− 8 (γ + 1)4 (2M)−10/13 + O(M −40/39 ) 3 γ3 8 (γ + 1)4 (2M)3/13 1 + O(M −1/39 ) 3 3 γ (2M)−10/39 < 2M 0 −3ε t 2 dt. M , Therefore we know that for w ∈ remote l (1 + γ )4 2Mw3 + 2MR1 (w) < 2M −3ε t 2 dt = −2Mε l 3 , 3 3γ 0 and have the estimation that if η ≥ T , 0 < ε < ε and M large enough [l ≥ (2M)−10/39 ], |FaM (ξ, w)| < (1 + γ )4/3 (2M)1/3 γ × exp −(η − T ) (1 + γ )4/3 (2M)1/3 l γ (1 + γ )4/3 (2M)−2/3 l 2M − ε l +T γ 3 −(γ + 1)w + 1 1 × (γ + 1)/γ w + 1 w + (aγ − 1)/((γ + 1)(a + 1)) < e−(η−T )(1+γ ) 4/3 /γ (2M)1/13 −ε (2M)3/13 . Now we get the result by inequalities similar to (53)–(54). L EMMA 4. P ROOF. If T is fixed and c is large enough, 1 1 −T u+u3 /3 du < . 2πi ∞ e c ≥c Obvious. L EMMA 5. 1 2πi If T is fixed and M is large enough, then for any η > T , holds: M local FaM (η, w) dw − 1 e−η/2 (γ + 1)(a + 1) Ai(η) < . 1 − aγ 3 M 1/40 1303 QUATERNIONIC WISHART M , |w| ≤ 2(2M)10/39 , so by (49) On local P ROOF. FaM (η, w) = (γ + 1)(a + 1) (1 + γ )4/3 (2M)1/3 aγ − 1 γ × e−(1+γ ) After the substitution u = M local 4/3 /γ (2M)1/3 ηw+(1+γ )4 /(3γ 3 )2Mw 3 (1+γ )4/3 (2M)1/3 w, γ FaM (η, w) dw = (55) 1 + O(M −1/39 ) . we get (γ + 1)(a + 1) (aγ − 1) e−ηu+u ∞ 3 /3 du <(1+γ )4/3 /γ (2M)1/13 × 1 + O(M −1/39 ) , and the O(M −1/39 ) term is independent to w. On ∞ , if η > T , e−(η−T )u < eT /2 e−η/2 . By (2) and (55), we have 1 2πi (γ + 1)(a + 1) FaM (η, w) dw − Ai(η) M 1 − aγ local (γ + 1)(a + 1) < eT /2 e−η/2 2πi(1 − aγ ) ∞ T /2 −η/2 (γ + 1)(a + 1) +e e 2πi(1 − aγ ) × |e−T u+u 3 /3 | du ≥(1+γ )4/3 /γ (2M)1/13 |e ∞ −T u+u3 /3 | du O(M −1/39 ), <(1+γ )4/3 /γ (2M)1/13 and we can get the result by direct calculation. C ONCLUSION OF THE PROOF OF (43). the convergence in L2 norm: lim (1 + a)2(N−M)−1 a −2N+1 M→∞ Putting Lemmas 2–5 together, we get (γ + 1)2(M−N)+1/3 (1 − aγ )(2M)1/3 γ 2M × e−2M/(1+γ )y ϕ2N−1 (y)χT (η) = Ai(η)χT (η). On the other hand, for η ∈ [T , ∞), (56) lim (1 + γ −1 )2(N−M)−1 eM−N y M−N+1/2 e(1−γ )/(1+γ )My = 1 M→∞ and (57) (1 + γ −1 2(N−M)−1 M−N M−N+1/2 −(1−γ )/(1+γ )My ) e y e η ≤1+O √ . M 1304 D. WANG Therefore, in L2 norm, lim (1 + a)2(N−M)−1 a −2N+1 M→∞ (1 − aγ )(2M)1/3 M−N e ψ2N−1 (y)χ(η) (γ + 1)2/3 γ 2N−1 = Ai(η)χ(η). Now we conclude the proof of the −1 < a < γ −1 part of Theorem 1. By Stirling’s formula, we get lim (2M)2(M−N) (58) M→∞ (2N − 1)! 2(N−M) 4N−1 γ = 1, e (2M − 1)! and then by (34), (38) and Proposition 5, we have the convergence in trace norm (1 − aγ )(2M)1/3 #4b (ξ, η)χ(η) χ (ξ )S M→∞ (1 + γ )2/3 lim (59) 1 = χ (ξ ) Ai(ξ ) Ai(η) + Ai (ξ ) 2 ∞ Ai(t) dt χ (η), η which implies that in trace norm, #4b (ξ, η)χ(η) = 0. lim χ (ξ )S M→∞ Now we get the desired result #4 (ξ, η)χ(η) = lim χ (ξ )S #4a (ξ, η)χ(η) = χ (ξ )S 4 (ξ, η)χ(η), lim χ (ξ )S M→∞ M→∞ and in the same way $ 4 (ξ, η)χ(η) = χ (ξ )SD 4 (ξ, η)χ(η), lim χ (ξ )SD M→∞ # 4 (ξ, η)χ(η) = χ (ξ )IS 4 (ξ, η)χ(η). lim χ (ξ )IS M→∞ Therefore, in trace norm lim P#T (ξ, η)χ(η) = χ (ξ ) M→∞ S4 (ξ, η) 4 (ξ, η) IS 4 (ξ, η) SD 4 (η, ξ ) χ (η), S and the convergence of Fredholm determinant follows. 3.2. Proof of the a = γ −1 part of Theorem 1. When a = γ −1 , the 1 − aγ −1 in (η). (59) vanishes, so we need other asymptotic formulas for ψ2N−1 (η) and ψ2N−1 −1 The approach is similar to that in the a < γ case, so we just sketch the proof. 1305 QUATERNIONIC WISHART P ROPOSITION 6. For fixed γ ≥ 1, a = γ −1 , ε > 0 and any T , we have the convergences in L2 norm with respect to ξ or η: lim γ −2N−1 (1 + γ )4/3 (2M)1/3 eM−N eεξ L2N−1 (2(M−N)) M→∞ × (2Mx)x M−N−1/2 e−Mx χ (ξ ) = −eεξ Ai(ξ )χ(ξ ), lim γ −2N M→∞ (60) 2Me = −e−εη M−N −εη e ∞ ∞ y L(2(M−N)) (2Mt)t M−N−1/2 e−Mt dt χ(η) 2N−1 Ai(t) dt χ(η), η lim (1 + a)2(N−M)−1 a −2N+1 eM−N γ −2N+1 e−εη ψ2N−1 (y)χ(η) M→∞ = e−εη s (1) (η)χ(η), lim (1 + a)2(N−M)−1 a −2N+1 eM−N M→∞ (γ + 1)4/3 εξ e ψ2N−1 (x)χ(ξ ) γ 2N (2M)−2/3 = eεξ Ai(ξ )χ(ξ ). S KETCH OF PROOF OF (60). We perform the same algebraic procedure and ¯¯ M ¯¯ M ¯¯ M use the contour ¯¯ M = ¯¯ M 1 ∪ 2 ∪ 3 ∪ 4 which is slightly different from the M in the a < γ −1 case (see Figure 4): γ ε/2 M π i/3 −1/3 ¯ ¯ e 1 = (4 − t) (2M) , 0 ≤ t ≤ 4 − γ +1 (1 + γ )1/3 ¯¯ M 2 = 5 γ ε/2 −1/3 tπ i 1 (2M) e , ≤t ≤ 4/3 (1 + γ ) 3 3 γ ε/2 (2M)−1/3 ≤ t ≤ 4 , e5π i/3 γ +1 (1 + γ )1/3 √ √ γ γ γ M ¯ ¯ + it −2 3 ≤t ≤2 3 4 = 2 γ +1 γ +1 γ +1 ¯¯ M ¯¯ ∞ ¯¯ ∞ and for asymptotic analysis, we define ¯¯ M remote , local , <c and ≥c in the same way as (50)–(51). Then we get ¯¯ M 3 = t ϕ2N−1 (y) = −(1 + a)2(M−N)+1 a 2N−1 × γ 2M e(2M)/(1+γ )y (γ + 1)2(M−N)+1 1 × 2πi % ¯¯ M exp − (1 + γ )4/3 (2M)1/3 ηw γ 1306 D. WANG F IG . 4. ¯¯ M . (1 + γ )4 + 2Mw3 + 2MR1 (w) 3 3γ × If we denote FM (η, w) = exp − × −(γ + 1)w + 1 dw . (γ + 1)/γ w + 1 w (1 + γ )4/3 (1 + γ )4 (2M)1/3 ηw + 2Mw3 + 2MR1 (w) γ 3γ 3 −(γ + 1)w + 1 1 , (γ + 1)/γ w + 1 w then parallel to Lemmas 2–5, we have: L EMMA 6. For any T fixed, and M large enough, if η > T , then −εη 1 1 e−εη/2 e < F (η, w) dw M 3 M 1/40 . 2πi ¯¯ M 4 L EMMA 7. For any T fixed, and M large enough, if η > T , then −εη 1 e 2πi ¯¯ M remote L EMMA 8. FM (η, w) dw< If T is fixed and c is large enough, 1 e−εη/2 . 3 M 1/40 1 −T u+u3 /3 du 1 e < . 2πi ¯¯ ∞ u c ≥c 1307 QUATERNIONIC WISHART L EMMA 9. For any T fixed, and M large enough, if η > T , then −εη 1 1 e−εη/2 −η/2 (1) e < F (η, w) dw − e s (η) M 3 M 1/40 . 2πi ¯¯ M local Using Lemmas 6–9, we get the convergence in L2 norm: lim (1 + a)2(N−M)−1 a −2N+1 M→∞ (γ + 1)2(M−N)+1 −2M/(1+γ )y e γ 2M × e−εη ϕ2N−1 (y)χ(η) = e−εη s (1) (η)χ(η). Furthermore, because of the limit result (56) and (57), we get the L2 convergence lim (1 + a)2(N−M)−1 a −2N+1 γ −2N+1 eM−N e−εη ψ2N−1 (y)χ(η) M→∞ = e−εη s (1) (η)χ(η). Now we conclude the proof of the a > γ −1 part of Theorem 1. Using (34), (58) and Proposition 6 we have the convergence in trace norm #4b (ξ, η)e−εη χ (η) lim χ (ξ )eεξ S M→∞ 1 = χ (ξ )eεξ Ai(ξ )s (1) (η) + Ai(ξ ) 2 ∞ Ai(t) dt e−εη χ (η) η 1 = χ (ξ )eεξ Ai(ξ )e−εη χ (η), 2 and this together with the conjugated convergence result (discussed in the Appen#4a (ξ, η) in formulas (39) and (40) of Section 3.1 conclude dix) of S (61) #4 (ξ, η)e−εη χ (η) = χ (ξ )eεξ S 4 (ξ, η)e−εη χ (η). lim χ (ξ )eεξ S M→∞ In the same way we get $ 4 (ξ, η)eεη χ (η) = χ (ξ )eεξ SD4 (ξ, η)eεη χ (η), lim χ (ξ )eεξ SD M→∞ # 4 (ξ, η)e−εη χ (η) = χ (ξ )e−εξ IS4 (ξ, η)e−εη χ (η). lim χ (ξ )e−εξ IS M→∞ Then we get the convergence in trace norm of a conjugate of P#T (ξ, η) lim χ (ξ ) M→∞ #4 (ξ, η)e−εη eεξ S −εξ # 4 (ξ, η)e−εη e IS eεξ S (ξ, η)e−εη = χ (ξ ) −εξ 4 e IS4 (ξ, η)e−εη $ 4 (ξ, η)eεη eεξ SD −εξ #4 (η, ξ )eεη χ (η) e S eεξ SD4 (ξ, η)eεη χ (η), e−εξ S 4 (η, ξ )eεη and the convergence of Fredholm determinant follows. 1308 D. WANG 3.3. Proof of the a > γ −1 part of Theorem 1. If a > γ −1 , the location as well as the fluctuation scale of the largest sample eigenvalue is changed. We change 1 variables as pM = (a + 1)(1 + γ 2 a ) and qM = (a + 1) 1 − γ 21a 2 √ 1 , and then by 2M (36) the kernel S∗ (x, y) after substitution is [here ∗ stands for 4, 4a or 4b, and the #∗ (ξ, η) in this subsection is not identical to that in Sections 3.1 and 3.2] S & #∗ (ξ, η) = (a + 1) 1 − S (62) 1 γ 2a2 1 √ 2M × S∗ (x, y)|x=(a+1)(1+1/(γ 2 a))+(a+1)√1−1/(γ 2 a 2 )1/(√2M)ξ . √ √ y=(a+1)(1+1/(γ 2 a))+(a+1) 1−1/(γ 2 a 2 )1/( 2M)η #4b (ξ, η) first. We analyze S P ROPOSITION 7. For fixed γ ≥ 1, a > γ −1 , ε > 0 and any T , we have convergences in L2 norm with respect to ξ or η: (γ 2 a + 1)M−N+1/2 (γ 2 a 2 − 1)2MeM−N M→∞ (γ 2 a)M+N+1/2 (a + 1)M−N−1/2 lim × e(γ (63) 2 a 2 −1)/((γ 2 a+1)(a+1))Mx × eεξ L(2(M−N)) (2Mx)x M−N−1/2 e−Mx χ (ξ ) 2N−1 1 1 γ 4 a 2 + γ 2 a 2 + 4γ 2 a + γ 2 + 1 2 = − √ exp − ξ + εξ χ (ξ ), 4 (γ 2 a + 1)2 2π 1 (γ 2 a + 1)M−N−1/2 (γ 2 a 2 − 1) 2 2 γ a − 1(2M)3/2 eM−N M→∞ 2 (γ 2 a)M+N+1/2 (a + 1)M−N+1/2 lim × e(γ (64) × 2 a 2 −1)/((γ 2 a+1)(a+1))My ∞ y eεη L(2(M−N)) (2Mt)t M−N−1/2 e−Mt dt χ(η) 2N−1 1 4 2 2 2 2 2 2 2 2 = − √ e−1/4(γ a +γ a +4γ a+γ +1)/(γ a+1) η +εη χ (η), 2π lim M→∞ (65) γ 2a (γ 2 a + 1)(a + 1) × e−(γ M−N+1/2 2 a 2 −1)/((γ 2 a+1)(a+1))My = e−1/4(γ eM−N e−εη ψ2N−1 (y)χ(η) 2 a 2 −1)(γ 2 −1)/(γ 2 a+1)2 η2 −εη χ (η), 1309 QUATERNIONIC WISHART lim M→∞ γ 2a (γ 2 a + 1)(a + 1) × e−(γ (66) M−N−1/2 2 a 2 −1)/((γ 2 a+1)(a+1))Mx = e−1/4(γ eM−N γ 2a (γ 2 a 2 − 1)M e−εξ ψ2N−1 (x)χ(ξ ) 2 a 2 −1)(γ 2 −1)/(γ 2 a+1)2 ξ 2 −εξ χ (ξ ). We only prove (65). The identity (45) still holds, but we need to use another contour and a new procedure of steepest-descent analysis. Since e−2Myz (z + 1)2M z2N = e2M(−(a+1)(1+1/(γ 2 a))z+log(z+1)−γ −2 log z)−(a+1) √ √ 1−1/(γ 2 a 2 ) 2Mηz , if we denote (ignoring the ambiguity of values of logarithm) g(z) = −(a + 1) 1 + 1 γ 2a z + log(z + 1) − γ −2 log z, then we get: • g (z) = −(a + 1)(1 + a z = − 1+a ; 1 γ 2a )+ 1 z+1 − γ −2 z , γ −2 , g (− 1+γ1 2 a ) z2 a g (− 1+a ) = (1 + a)2 ( γ 21a 2 − 1) < 0. 1 • g (z) = − (z+1) 2 + with zero points z = − 1+γ1 2 a and = (γ −1 + γ a)2 (1 − 1 ) γ 2 a2 > 0 and So we take z = − 1+γ1 2 a as the saddle point, and locally around that point, after the substitution w = z + g − 1 , 1+γ 2 a we get 1 a+1 + w = 2 + log(γ 2 a) − (1 − γ −2 ) log(γ 2 a + 1) + γ −2 πi 2 1+γ a γ a 1 1 + (γ −1 + γ a)2 1 − 2 2 w2 + R2 (w), 2 γ a where R2 (w) = O(w3 ) as w → 0, so that % e2M(−yz+log(z+1)−γ C (67) = % M exp 2M −2 log z) z dz ((a + 1)z + a)(z + 1) a+1 + log(γ 2 a) − (1 − γ −2 ) log(γ 2 a + 1) γ 2a 1310 D. WANG 1 1 + γ −2 πi + (γ −1 + γ a)2 1 − 2 2 w2 + R2 (w) 2 γ a & η 1 1 w− 2 − (a + 1) 1 − 2 2 √ γ a γ a+1 2M × =− w − 1/(γ 2 a + 1) dw ((a + 1)w + (γ 2 a 2 − 1)/(γ 2 a + 1))(w + (γ 2 a)/(γ 2 a + 1))) 1 (γ 2 a)2M−1 2 e2M/(γ a+1)x a + 1 (γ 2 a + 1)2(M−N) × % M & exp −(a + 1) 1 − 1 √ 2Mηw γ 2a2 1 1 + (γ −1 + γ a)2 1 − 2 2 2Mw2 + 2MR2 (w) 2 γ a × × −(γ 2 a + 1)w + 1 (γ 2 a + 1)/(γ 2 a)w + 1 w + (γ 2 a 2 where M is a contour around are defined as (see Figure 5) 1 dw, − 1)/((γ 2 a + 1)(a + 1)) 1 , composed of 1M , 2M , 3M γ 2 a+1 1M = {−it|−2 ≤ t ≤ 2}, 3M = {4 + it|−2 ≤ t ≤ 2}, and 4M , which 2M = {4 − t + 2i|0 ≤ t ≤ 4}, 4M = {t − 2i|0 ≤ t ≤ 4}. And for the asymptotic analysis, we define (see Figure 6) M = {w ∈ M ||w| ≤ M −2/5 }, local F IG . 5. M M remote = 1M \ local , M . 1311 QUATERNIONIC WISHART ∞. F IG . 6. ∞ = {−it|−∞ < t < ∞}, ∞ <c = {w ∈ ∞ ||w| < c}, ∞ ∞ = ∞ \ <c . ≥c Then if we denote GaM (η, w) = (γ −1 & + γ a) 1 1 − 2 2 2M γ a & × exp −(a + 1) 1− 1 2Mηw 2 γ a2 1 1 + (γ −1 + γ a)2 1 − 2 2 2Mw2 + 2MR2 (w) 2 γ a × 1 −(γ 2 a + 1)w + 1 , 2 2 2 2 (γ a + 1)/(γ a)w + 1 w + (γ a − 1)/((γ 2 a + 1)(a + 1)) we have four lemmas similar to Lemmas 2–5: L EMMA 10. For any T fixed, and M large enough, if η > T , then −εη 1 e 2πi L EMMA 11. 1 e−εη GaM (η, w) dw< . 3 M 1/10 2M ∪3M ∪4M For any T fixed, and M large enough, if η > T , then −εη 1 e 2πi M remote L EMMA 12. GaM (η, w) dw< 1 e−εη . 3 M 1/10 If T is fixed and c is large enough, 1 2πi ∞ ≥c e−(a+1)/(γ −1 +γ a)T u+u2 /2 1 du< . c 1312 D. WANG L EMMA 13. For any T fixed, and M large enough, if η > T , then −εη 1 e 2πi M local GaM (η, w) dw (γ 2 a + 1)(a + 1) −1/2((γ (a+1))/(γ 2 a+1)η)2 √ e − εη − 2 2 (γ a − 1) 2π < 1 e−εη . 3 M 1/10 Their proofs are the same as those of Lemmas 2–5, and we need the identity 1 2πi ∞ S KETCH e−(a+1)/(γ −1 +γ a)ηu+u2 /2 OF PROOF OF (65). 1 2 2 du = − √ e−1/2(γ (a+1)/(γ a+1)η) . 2π a Because the pole z = − a+1 , which is w = a −1 in the w plane, is not in side of M , so − (γ 2γa+1)(a+1) 2 2 % (z + 1)2(M−N) dz = 0. z + a/(a + 1) C Similar to but subtler than (56) and (57), if we denote (here we have a notation conflict with the ri defined in Section 2.3, but there should be no confusion) (68) rM (η) = e−2Mxz γ 2a (γ 2 a + 1)(a + 1) × e−(γ M−N+1/2 eM−N y M−N+1/2 2 −1)a/((γ 2 a+1)(a+1))My−εη , we have for η ∈ [T , ∞), lim rM (η) = e−1/4(γ 2 a 2 −1)(γ 2 −1)/(γ 2 a+1)2 η2 −εη M→∞ , and for a large enough positive C, η ∈ [C, ∞) and M1 < M2 , pointwisely rM1 (η) > rM2 (η) > 0, so that we can use the dominant convergence theorem to prove that in L2 norm, lim rM (η)χ(η) = e−1/4(γ 2 a 2 −1)(γ 2 −1)/((γ 2 a+1)2 )η2 −εη M→∞ Finally, since from (45), (67) and (68), ψ2N−1 (y) = y M−N+1/2 e(a−1)/(a+1)My + (1 + a)2(M−N) a 2N−1 × 1 (γ 2 a)2M 2 2(M−N)+1 (γ a + 1) (γ 2 a 2 − 1)2M χ (η). 1313 QUATERNIONIC WISHART × y M−N+1/2 e(1−γ = y M−N+1/2 e−((γ ' × e(γ 2 a)/(1+γ 2 a)My 1 2πi M GaM (η, w) dw 2 −1)a)/((γ 2 a+1)(a+1))My 2 a 2 −1)/(γ 2 a+1)(a+1)My + (1 + a)2(M−N) a 2N−1 × e−(γ (γ 2 a)2M 1 (γ 2 a + 1)2(M−N)+1 (γ 2 a 2 − 1)2M 2 a 2 −1)/((γ 2 a+1)(a+1))My 1 2πi M ( GaM (η, w) dw , we get γ 2a (γ 2 a + 1)(a + 1) × eM−N e−(γ M−N+1/2 2 a 2 −1)/((γ 2 a+1)(a+1))My ' e−εη ψ2N−1 (y)χ (η) = rM (η) 1 + (1 + a)2(M−N) a 2N−1 (γ 2 a)2M (γ 2 a + 1)2(M−N)+1 (γ 2 a 2 − 1) 2My exp − 2 × (γ a + 1)(a + 1) (γ 2 a 2 − 1)2M 1 1 × 2πi M ( GaM (η, w) dw χ (η), and get the L2 convergence lim M→∞ γ 2a (γ 2 a + 1)(a + 1) × eM−N e−(γ M−N+1/2 2 a 2 −1)/((γ 2 a+1)(a+1))My = lim rM (η)χ (η) = e−1/4(γ e−εη ψ2N−1 (y)χ (η) 2 a 2 −1)(γ 2 −1)/(γ 2 a+1)2 η2 −εη M→∞ χ (η), because for a > γ −1 and η ∈ [T , ∞), we can verify by by elementary but tricky estimation that lim (1 + a)2(M−N) a 2N−1 M→∞ e−(γ a −1)/((γ a+1)(a+1))2My (γ 2 a)2M =0 (γ 2 a + 1)2(M−N)+1 (γ 2 a 2 − 1)2M 2 2 × 2 1314 D. WANG uniformly, and by Lemmas 10–13, in L2 norm lim e−εη M→∞ = 1 2πi M GaM (η, w) dw χ(η) (γ 2 a + 1)(a + 1) −1/2(γ (a+1)/(γ 2 a+1)η)2 −εη √ e χ (η). (γ 2 a 2 − 1) 2π For notational simplicity, we denote functions on the left-hand sides of (63)– (66) by F1 (ξ )χ(ξ ), F2 (η)χ(η), F3 (η)χ(η) and F4 (ξ )χ(ξ ), and denote cM = (2M)2(M−N) (2N − 1)! 2(N−M) 4N−1 e γ . (2M − 1)! By (58), we have lim cM = 1. M→∞ Then we get from (34), (62) and (63)–(66) #4b (ξ, η) = − S (69) cM −(γ 2 a 2 −1)/((γ 2 a+1)(a+1))M(x−y)−ε(ξ −η) e F1 (ξ )F3 (η) 2 + e(γ 2 a 2 −1)/((γ 2 a+1)(a+1))M(x−y)+ε(ξ −η) If we define SD4a (x, y) = N−2 j =0 IS4a (x, y) = N−2 j =0 1 ψ2j (x)ψ2j +1 (y) − ψ2j +1 (x)ψ2j (y) , rj 1 −ψ2j (x)ψ2j +1 (y) + ψ2j +1 (x)ψ2j (y) , rj and SD4b (x, y) = IS4b (x, y) = 1 rN−1 1 rN−1 ψ2N−2 (x)ψ2N−1 (y) − ψ2N−1 (x)ψ2N−2 (y) , −ψ2N−2 (x)ψ2N−1 (y) + ψ2N−1 (x)ψ2N−2 (y) , like S4a (x, y) = N−2 j =0 S4b (x, y) = 1 −ψ2j (x)ψ2j +1 (y) + ψ2j +1 (x)ψ2j (y) , rj 1 rN−1 F4 (ξ )F2 (η) . −ψ2N−2 (x)ψ2N−1 (y) + ψ2N−1 (x)ψ2N−2 (y) , 1315 QUATERNIONIC WISHART in (28) and (29), and by (35) and (37) define [∗ stands for 4, 4a or 4b] $ ∗ (ξ, η) = (a + 1)2 1 − SD 1 1 γ 2 a 2 2M × SD∗ (x, y)|x=(a+1)(1+1/(γ 2 a))+(a+1)√1−1/(γ 2 a 2 )1/(√2M)ξ , √ √ y=(a+1)(1+1/(γ 2 a))+(a+1) 1−1/(γ 2 a 2 )1/( 2M)η √ # ∗ (ξ, η) = IS∗ (x, y)| √ IS , x=(a+1)(1+1/(γ 2 a))+(a+1) 1−1/(γ 2 a 2 )1/( 2M)ξ √ √ y=(a+1)(1+1/(γ 2 a))+(a+1) 1−1/(γ 2 a 2 )1/( 2M)η like #∗ (ξ, η) = (a + 1) 1 − 1/(γ 2 a 2 ) √ 1 S 2M × S∗ (x, y)|x=(a+1)(1+1/(γ 2 a))+(a+1)√1−1/(γ 2 a 2 )1/(√2M)ξ √ √ y=(a+1)(1+1/(γ 2 a))+(a+1) 1−1/(γ 2 a 2 )1/( 2M)η in (62), then in the same way of (69), we have $ 4b (ξ, η) = SD cM 2 2 2 CM e−(γ a −1)/((γ a+1)(a+1))M(x−y)−ε(ξ −η) F1 (ξ )F4 (η) 4 − e(γ # 4b (ξ, η) = IS 2 a 2 −1)/((γ 2 a+1)(a+1))M(x−y)+ε(ξ −η) F4 (ξ )F1 (η) , cM −(γ 2 a 2 −1)/((γ 2 a+1)(a+1))M(x−y)−ε(ξ −η) e F2 (ξ )F3 (η) CM − e(γ 2 a 2 −1)/((γ 2 a+1)(a+1))M(x−y)+ε(ξ −η) with CM √ (γ 2 a 2 − 1)3/2 2M . = aγ (γ 2 a + 1) Now we write P#T (ξ, η) as the sum P#T (ξ, η) = P#T a (ξ, η) + P#T b (ξ, η), with P#T a (ξ, η) = χ (ξ ) # S4a (ξ, η) # 4a (ξ, η) IS # $ 4a (ξ, η) SD #4a (η, ξ ) χ (η), S F3 (ξ )F2 (η) , $ 4b (ξ, η) S (ξ, η) SD P#T b (ξ, η) = χ (ξ ) #4b #4b (η, ξ ) χ (η). IS4b (ξ, η) S 1316 D. WANG If we denote ⎛ γ 2a2 − 1 M(x − x0 ) + εξ ⎜ exp U (ξ ) = ⎝ (γ 2 a + 1)(a + 1) 0 CM F4 (ξ ) (γ 2 a 2 −1)/((γ 2 a+1)(a+1))M(x−x0 )+εξ ⎞ e − ⎟ 2 F3 (ξ ) ⎠, ⎛ e(γ 2 a 2 −1)/((γ 2 a+1)(a+1))M(x−x )+εξ 0 2 2 2 ⎜ e−(γ a −1)/((γ a+1)(a+1))M(y−y0 )−εη U −1 (η) = ⎝ 0 CM F4 (ξ ) −(γ 2 a 2 −1)/((γ 2 a+1)(a+1))M(y−y0 )−εξ ⎞ e ⎟ 2 F3 (ξ ) ⎠, e(γ 2 a 2 −1)/((γ 2 a+1)(a+1))M(y−y )+εξ 0 with x0 = y0 = (a + 1) 1 + 1 & γ 2a + (a + 1) 1 − 1 γ 2a2 T √ , 2M then we have the result of kernel conjugation ⎛ c F2 (ξ )F4 (ξ ) M − F (ξ ) + F3 (η) 1 F3 (ξ ) U (ξ )P#T b (ξ, η)U −1 (η) = χ (ξ ) ⎝ 2 U (ξ )P#T b (ξ, η)U −1 (η)21 ⎞ 0 CM F2 (η)F4 (η) ⎠ χ (η), − F3 (ξ ) F1 (η) + 2 F3 (η) with the entry U (ξ )P#T b (ξ, η)U −1 (η)21 cM −2(γ 2 a 2 −1)/((γ 2 a+1)(a+1))M(x−x0 )−2εξ = e F2 (ξ )F3 (η) CM − F3 (ξ )F2 (η)e−2(γ 2 a 2 −1)/((γ 2 a+1)(a+1))M(y−y )−2εη 0 . We want U (ξ )P#T b (ξ, η)U −1 (η) to converge in trace norm as M → ∞, and need the results: L EMMA 14. In trace norm, lim U (ξ )P#T b (ξ, η)U −1 (η)21 = 0. M→∞ 1317 QUATERNIONIC WISHART L EMMA 15. In L2 norm, F2 (ξ )F4 (ξ ) χ (ξ ) M→∞ F3 (ξ ) lim 1 1 γ 4 a 2 + γ 2 a 2 + 4γ 2 a + γ 2 + 1 2 = − √ exp − η + εξ χ (ξ ). 4 (γ 2 a + 1)2 2π The proof of Lemma 14 is obvious. The main ingredient in the proof of Lemma 15 is (64) and the fact that F4 (ξ )/F3 (ξ ) approaches to 1 uniformly on [T , ∞). We need another convergence result on U (ξ )P#T a (ξ, η)U −1 (η): P ROPOSITION 8. (70) In trace norm, lim U (ξ )P#T a (ξ, η)U −1 (η) = 0. M→∞ The proof is left to the reader. Since all the four entries in P#T a (ξ, η) can be expressed by Laguerre polynomials like (32) and (33), the asymptotic results like (63) and (64) give the convergence (70). By Lemmas 14 and 15 and Proposition 8, we get in trace norm lim det I − P#T (ξ, η) M→∞ = lim det I − U (ξ )P#T (ξ, η)U −1 (η) M→∞ = lim det I − U (ξ )P#T b (ξ, η)U −1 (η) M→∞ = T ∞ 1 2 √ e−t /2 dt 2π 2 , and we get the proof of the a > γ −1 part of Theorem 1. 4. Proof of FGSE1 = FGOE . In manipulation of kernels, we follow the method of [30]. The procedure seems informal and cursory, but is carefully justified in [30]. For notational simplicity, we denote [χ (ξ ) = χ(T ,∞) (ξ )] B(ξ ) = 1 − s (1) (ξ ) = ∞ Ai(t) dt. ξ First, we express the integral operator χ (ξ )S 4 (ξ, η)χ(η) χ(ξ )SD4 (ξ, η)χ(η) χ (ξ )P (ξ, η)χ (η) = χ (ξ )IS4 (ξ, η)χ(η) χ(ξ )S 4 (η, ξ, )χ (η) 1318 D. WANG by χ (ξ ) ∂ ∂ξ 0 0 χ (ξ ) IS4 (ξ, η)χ(η) S 4 (η, ξ )χ(η) , IS4 (ξ, η)χ(η) S 4 (η, ξ, )χ (η) since by (22)–(23) and taking limit, ∂ IS4 (ξ, η) = S 4 (ξ, η), ∂ξ ∂ S 4 (η, ξ ) = SD4 (ξ, η). ∂ξ Then using (21) for A bounded and B trace class, upon suitably defining the Hilbert spaces our operators A and B are acting on, we find ∂ 0 χ (ξ ) IS4 (ξ, η)χ(η) S 4 (η, ξ )χ(η) det I − ∂ξ IS4 (ξ, η)χ(η) S 4 (η, ξ, )χ (η) 0 χ (ξ ) ⎛ ⎞⎞ ⎛ ∂ 0 χ (η) IS (ξ, η)χ(η) S (η, ξ )χ(η) 4 4 ⎠⎠ ⎝ = det ⎝I − ∂η IS4 (ξ, η)χ(η) S 4 (η, ξ, )χ (η) 0 χ (η) ⎛ ⎛ ⎞ ⎞ ∂ IS (ξ, η)χ(η) S 4 (η, ξ )χ(η) ⎜ ⎜ 4 ⎟⎟ ∂η ⎜ ⎟⎟ , = det ⎜ I − ⎝ ⎝ ⎠⎠ ∂ IS4 (ξ, η)χ(η) S 4 (η, ξ, )χ (η) ∂η and by conjugation with ⎛ ⎛ 1 0 −1 1 , we get ⎞⎞ ∂ IS4 (ξ, η)χ(η) + S 4 (η, ξ )χ(η) S 4 (η, ξ )χ(η) ⎠⎠ = det ⎝I − ⎝ ∂η 0 0 ∂ = det I − IS4 (ξ, η)χ(η) + S 4 (η, ξ )χ(η) ∂η Since ∞ ∂ η=∞ IS4 (ξ, η) f (η) dη = IS4 (ξ, η)f (η)|η=T − ∂η T as an operator IS4 (ξ, η)χ(η) . ∞ ∂ T ∂η IS4 (ξ, η)f (η) dη, ∂ ∂ = IS4 (ξ, ∞)δ∞ (η) − IS4 (ξ, T )δT (η) − IS4 (ξ, η)χ(η), ∂η ∂η where δ∞ and δT are (generalized) Dirac functions. Then with the help of identity ∞ ξ KAiry (t, η) dt + ∞ η KAiry (ξ, t) dt = ∞ ξ Ai(t) dt ∞ ξ Ai(t) dt, 1319 QUATERNIONIC WISHART which can be proved directly from (1), we get I − IS4 (ξ, η)χ(η) ∂ + S 4 (η, ξ )χ(η) ∂η 1 = I − KAiry (ξ, η) − B(ξ ) Ai(η) + Ai(η) χ (η) 2 + ∞ 1 2 T 1 1 1 KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) δT (η) 4 2 2 1 + B(ξ )δ∞ (η). 2 Now we denote R(ξ, η) as the resolvent of KAiry (ξ, η)χ(η), such that as integral operators −1 I + R(ξ, η) = I − KAiry (ξ, η)χ(η) (71) , then ∂ I − IS4 (ξ, η)χ(η) + S 4 (η, ξ )χ(η) ∂η = I − KAiry (ξ, η)χ(η) 1 × I − (I + R) 1 − B(ξ ) Ai(η)χ(η) 2 ∞ 1 + (I + R) 2 T KAiry (ξ, t) dt 1 1 1 − B(T )B(ξ ) − B(ξ ) + B(T ) δT (η) 4 2 2 1 + (I + R)B(ξ )δ∞ (η) . 2 Again by the formula (21), in the form of (formula (17) in [30]) (72) det I − n αk ⊗ βk = det δj,k − (αj , βk ) j,k=1,...,n k=1 we get 1 det I − (I + R) 1 − B(ξ ) Ai(η)χ(η) 2 ∞ 1 1 1 1 + (I + R) KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) δT (η) 2 T 4 2 2 1320 D. WANG ⎛ 1 + α11 = det ⎝ α21 α31 ⎞ α12 1 + α22 α32 α13 α23 ⎠ , 1 + α33 where upon the definition f (ξ ), g(ξ )T = we define ) ∞ 1 1 + (I + R)B(ξ )δ∞ (η) 2 ∞ f (ξ )g(ξ ) dξ, T α11 = (I + R) 1 − 12 B(ξ ) , − Ai(ξ ) T , α12 = (I + R) 2 1 1 1 KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) , 4 2 2 T * − Ai(ξ ) , α13 = 1 2 (I T + R)B(ξ ), − Ai(ξ ) T , α21 = (I + R) 1 − 12 B(ξ ) ξ =T , ∞ 1 α22 = (I + R) 2 T 1 1 1 KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) , 4 2 2 ξ =T α23 = 12 (I + R)B(ξ )|ξ =T , α31 = (I + R) 1 − 12 B(ξ ) ξ =∞ = 1, ∞ 1 α32 = (I + R) 2 T 1 1 1 KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) 4 2 2 ξ =∞ 1 = B(T ), 2 α33 = 12 (I + R)B(ξ )|ξ =∞ = 0. If we take elementary row operations, we get ⎛ 1 + α11 det ⎝ α21 α31 ⎛ α12 1 + α22 α32 1 + α11 − α13 ⎝ = det α21 − α23 0 = det 1 + β11 β21 ⎞ α13 α23 ⎠ 1 + α33 α12 − 12 B(T )α13 1 + α22 − 12 B(T )α23 0 β12 , 1 + β22 ⎞ α13 α23 ⎠ 1 1321 QUATERNIONIC WISHART where β11 = (I + R) 1 − B(ξ ) , − Ai(ξ ) T , ) ∞ 1 β12 = (I + R) 2 T KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) , − Ai(ξ ) , T β21 = (I + R) 1 − B(ξ ) ξ =T , ∞ 1 β22 = (I + R) 2 KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) T * ξ =T . Using (71) and (72), we observe [s (1) (ξ ) = 1 − B(ξ )] det I − KAiry (ξ, η)χ(η) det 1 + β11 β21 β12 1 + β22 = det I − KAiry (ξ, η)χ(η) + s (1) (ξ ) Ai(η) χ (η) 1 + 2 ∞ T KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) δT (η) . If we denote R̃(ξ, η) as the resolvent of (KAiry (ξ, η)χ(η) + s (1) (ξ ) Ai(η))χ(η), so that as operators −1 I + R̃(ξ, η) = I + KAiry (ξ, η)χ(η) + s (1) (ξ ) Ai(η) χ (η) and Q(ξ ) = (I + R̃) then ∞ T KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) , FGSE1 = det I − KAiry (ξ, η)χ(η) + s (1) (ξ ) Ai(η) χ (η) det I + 12 Q(ξ )δT (η) . To prove Theorem 2, we need only (6) and det I + 12 Q(ξ )δT (η) = 1, which by (72) is equivalent to Q(T ) = 0. (73) If we take f (ξ ) = Q(ξ ) + 1, then (73) is = ∞ T which is equivalent to (74) I − KAiry (ξ, η)χ(η) + s (1) (ξ ) Ai(η) χ (η) f (ξ ) − 1 KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ), I − KAiry (ξ, η)χ(η) + s (1) (ξ ) Ai(η) χ (η) f (ξ ) = s (1) (ξ ). 1322 D. WANG The integral equation (74) is solvable, and the solution is f (ξ ) = (I + R)s (1) (ξ ) . 1 − (I + R)s (1) (ξ ), Ai(ξ )T Therefore to prove Theorem 2 we need only to prove f (T ) = 1, which is equivalent to (I + R)s (1) (T ) = 1 − (I + R)s (1) (ξ ), Ai(ξ ) T . This is a nontrivial result, but it can be derived by results in [30]. In Section VII of [30] Tracy and Widom√define function q̄ and ū for both GOE and GSE. Our (I + R)s (1) (T ) is equal to 2 times their q̄ in GOE and our (I + R)s (1) (ξ ), Ai(ξ )T is equal to 2 times their ū in GOE. With (I + R)s (1) (T ) = e− (75) +∞ T q(s) ds (I + R)s (1) (ξ ), Ai(ξ ) T = 1 − e− (76) +∞ T , q(s) ds , where q is the Painlevé II function determined by the differential equation q (s) = sq(s) + 2q 3 (s) together with the condition q(s) ∼ Ai(s) as s → ∞. We can give a proof of (75) and (76), based on the method and results in [29]. First, assume T is fixed, then (I + R)s (1) is a function, and we have ' ( d ds (1) (ξ ) d (I + R)s (1) (ξ ) = (I + R) + , (1 + R) s (1) (ξ ). dξ dξ dξ Since ' d (1) dξ s (ξ ) = Ai(ξ ) and we have (2.13) in [29], which is ( d , (1 + R) = −(2 + R) Ai(ξ ) · (1 − K t )−1 (Ai(η)χ(η)) + R(η, T ) · ρ(T , η), dξ where ρ(x, y) = δ(x − y) + R(x, y) is the distribution kernel of 1 + R, and K t is the transpose (as an operator) of KAiry (ξ, η)χ(η), we have d (I + R)s (1) (ξ ) = (1 + R) Ai(ξ ) − (1 + R) Ai(ξ ) · (I + R)s (1) (ξ ), Ai(ξ ) T dξ + R(ξ, T ) · (1 + R)s (1) (T ). If we regard T as a parameter, then we have d (I + R)s (1) (ξ ; T ) = −R(ξ, T ) · (1 + R)s (1) (T ), dT because (2.16) in [29] gives (77) 1 (1 + R) = R(ξ, T ) · ρ(T , η). dT 1323 QUATERNIONIC WISHART Therefore, if we set ξ = T and take the derivative with respect to the parameter T , we have d d d (1) (1) + (1 + R)s (T ) = (1 + R)s (T ) dT dξ dT ξ =T = (1 + R) Ai(T ) · 1 − (I + R)s (1) (ξ ), Ai(ξ ) T . On the other hand, by (77) we have d (I + R)s (1) (ξ ), Ai(ξ ) T dT = −(1 + R)s (1) ) * d (I + R)s (1) (ξ ), Ai(ξ ) (T ) · Ai(T ) + dT = −(1 + R)s (1) (T ) · Ai(T ) + ∞ T R(ξ, T ) Ai(ξ ) dξ T = −(1 + R)s (1) (T ) · (1 + R) Ai(T ). (1.11) and (1.12) in [29] give the result (1 + R) Ai(T ) = q(T ), and now if we denote (I + R)s (1) (T ) = sT and (I + R)s (1) (ξ ), Ai(ξ )T = wT , we have ⎧ d ⎪ ⎪ sT = q(1 − wT ) ⎨ dT ⎪ d ⎪ ⎩ (1 − wT ) = qsT . dT Now we can get (75) and (76) by boundary conditions. APPENDIX: DISCUSSION ON THE TRACE NORM CONVERGENCE OF INTEGRAL OPERATORS RELATED TO LUE For convenience, we write (44) as % e2My z2(M−N)+j e−2Myz dz, 2πi C (z − 1)j +1 where C is a contour around 1, and we have another integral representation of Laguerre polynomials (78) (2(M−N)) Lj (2My) = (2(M−N)) Lj (2Mx) = (79) 1 (2(M − N) + j )! 2(M−N) 2(M−N) j !(2M) x 2πi × where D is a contour around 0. % D e2Mxz (z − 1)j z2(M−N)+j +1 dz, 1324 D. WANG Recall the integral operator K(x, y) [11] for the rescaled LUE with parameters 2N and 2M, and by (78) and (79) we have K(x, y) = 2N−1 j =0 j! (2M)2(M−N)+1 (2(M − N) + j )! × L(2(M−N)) (2Mx)L(2(M−N)) (2My)x M−N y M−N e−x+y j j (80) = % 2(M−N)+j % 2M y M−N eMy 2N−1 −2Myz z dz dw e (2πi)2 x M−N eMx j =0 C (z − 1)j +1 D × e2Mxw (w − 1)j w2(M−N)+j +1 . We can write the sum of integrands in (80) as 2N−1 e2Mxz j =0 z2(M−N)+j −2Myw (w − 1)j e (z − 1)j +1 w2(M−N)+j +1 j = e2Mxz e−2Myw 2N−1 z(w − 1) z2(M−N) 1 w2(M−N) (z − 1)w j =0 (z − 1)w = e2Mxz e−2Myw 1 − (z(w − 1)/((z − 1)w))2N z2(M−N) 1 w2(M−N) (z − 1)w 1 − z(w − 1)/((z − 1)w) = 1 1 e2Mxz z2(M−N) e−2Myw 2(M−N) z−w w 2N z2M 1 −Myw (w − 1) e2Mxz e . z−w (z − 1)2N w2M By the residue theorem, let C and D be disjoint, then for the variable z, the pole z = w is outside of C, % % 1 1 e2Mxz z2(M−N) e−2Myw 2(M−N) = 0. dz dw z−w w C D On the other side, we assume (w − z) to be less than 0, and get − 1 = 2M z−w so that we have 2M (2πi)2 (81) = % % dz dw C D % 2 (2M) (2πi)2 C ∞ et2M(w−z) dt, 0 2N z2M 1 2Mxw (w − 1) e−2Myz e z−w (z − 1)2N w2M % dz dw D ∞ 0 e−2M(y+t)z z2M (z − 1)2N 1325 QUATERNIONIC WISHART × e2M(x+t)w = (2M) 2 ∞ % 1 2πi 0 × e −2M(y+t)z C % 1 2πi z2M dz (z − 1)2N e2M(x+t)w D (w − 1)2N w2M (w − 1)2N dw dt. w2M Put (80)–(81) together, we get the result K(x, y) = −(2M)2 × (82) y M−N eMy x M−N eMx ∞ % 1 2πi 0 D 1 × 2πi e2M(x+t)w % e (w − 1)2N dw w2M −2M(y+t)z C z2M dz dt. (z − 1)2N To find the probability that the largest eigenvalue ≥ T in the LUE, we need to consider the integral operator from L2 ([0, ∞)) to L2 ([0, ∞)) with the kernel χ (x)K(x, y)χ (y). We can decompose it into the product of two integral operators by (82): χ (x)K(x, y)χ (y) = −(2M)2 χ (x)J (x, t)χ[0,∞) (t) ◦ χ[0,∞) (t)H (t, y)χ(y), where χ (x)J (x, t)χ[0,∞) (t) and χ[0,∞) (t)H (t, y)χ(y) stands for two integral operators with these kernels, and J (x, t) = 1 x M−N eMx H (t, y) = y M−N eMy 1 2πi 1 2πi % % e2M(x+t)w (w − 1)2N dw, w2M e−2M(y+t)z z2M dz. (z − 1)2N D C Since we consider the limiting distribution of the largest eigenvalue around (1 + )4/3 γ −1 )2 , we take p = (1 + γ −1 )2 , q = γ(1+γ , x = p + qξ , y = p + qη and t = qτ . (2M)2/3 # Then for the rescaled kernel χ (ξ )K(ξ, η)χ (η), we have # #(τ, η)χ (η), η)χ (η) = χ (ξ )J#(ξ, τ )χ[0,∞) (τ ) ◦ χ[0,∞) (τ )H χ (ξ )K(ξ, where J#(ξ, τ ) = #(τ, η) = H (γ + 1)4/3 1/3 γ 2N eN−M 1 M γ x M−N eMx 2πi (γ + 1)4/3 γ M 1/3 y M−N eMy γ 2N eN−M 1 2πi % e2M(p+q(ξ +τ ))w D % C (w − 1)2N dw, w2M e−2M(p+q(ξ +τ ))z z2M dz. (z − 1)2N 1326 D. WANG We want to prove the trace norm convergence # η)χ (η) lim χ (ξ )K(ξ, M→∞ (83) = χ (ξ )KAiry (ξ, η)χ(η) = χ (ξ ) Ai(ξ + τ )χ[0,∞) (τ ) ◦ χ[0,∞) (τ ) Ai(τ + η)χ (η). By results in functional analysis, we need only to prove the convergence in Hilbert– Schmidt norm of (e.g., see [20]) (84) (85) lim χ (ξ )J#(ξ, τ )χ[0,∞) (τ ) = χ (ξ )Kf Airy (ξ, η)χ(η), M→∞ #(τ, η)χ (η) = χ[0,∞) (τ ) Ai(τ + η)χ (η). lim χ[0,∞) (τ )H M→∞ Since for integral operators, the convergence in Hilbert–Schmidt norm is equivalent to the convergence in L2 norm of their kernels as two variable functions, we can verify (84) and (85) by asymptotic analysis similar to that in Section 3. #4a1 (ξ, η)χ(η) in (39), we have For the integral operator χ (ξ )S √ # #(τ, η)χ (η), #4a1 (ξ, η)χ(η) = 1 χ (ξ ) √1 # χ (ξ )S J#(ξ, τ )χ[0,∞) (τ ) ◦ χ[0,∞) (τ ) y H 2 x # # in the same way as J# and J#, but use parameters 2N − 2 J# and H where we define # #4a1 part of (61), we have and 2M − 2 instead of 2N and 2M. Similarly, in the S #4a1 (ξ, η)e−εη χ (η) χ (ξ )eεξ S √ y # eεξ # 1 #(τ, η)χ (η). = χ (ξ ) √ # J (ξ, τ )χ[0,∞) (τ ) ◦ χ[0,∞) (τ ) −εη H 2 e x We can give rigorous proofs to (39) and (61) in the same way as (83). Acknowledgments. The author is most grateful to his advisor Mark Adler, who pointed out the spiked model problem and gave warm encouragement to the author. I also thank Ira Gessel for help in combinatorics, especially his suggestions in proofs in Section 2, and Jinho Baik for his suggestions on related models. REFERENCES [1] A DLER , M., F ORRESTER , P. J., NAGAO , T. and VAN M OERBEKE , P. (2000). Classical skew orthogonal polynomials and random matrices. J. Statist. Phys. 99 141–170. MR1762659 [2] A NDERSON , T. W. (1963). Asymptotic theory for principal component analysis. Ann. Math. Statist. 34 122–148. MR0145620 [3] A NDERSSON , S. (1975). Invariant normal models. Ann. Statist. 3 132–154. MR0362703 [4] BAIK , J., B EN A ROUS , G. and P ÉCHÉ , S. (2005). Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. Ann. Probab. 33 1643–1697. MR2165575 [5] BAIK , J., B ORODIN , A., R AINS , E. M. and S UIDAN , T. M. (2007). Unpublished results. In Conference Random and Integrable Models in Mathematics and Physics, September 11– 15, Brussels. QUATERNIONIC WISHART 1327 [6] BAIK , J. and R AINS , E. M. (2001). The asymptotics of monotone subsequences of involutions. Duke Math. J. 109 205–281. MR1845180 [7] BAIK , J. and R AINS , E. M. (2001). Symmetrized random permutations. In Random Matrix Models and Their Applications. Mathematical Sciences Research Institute Publications 40 1–19. Cambridge Univ. Press, Cambridge. MR1842780 [8] BAIK , J. and S ILVERSTEIN , J. W. (2006). Eigenvalues of large sample covariance matrices of spiked population models. J. Multivariate Anal. 97 1382–1408. MR2279680 [9] DE B RUIJN , N. G. (1955). On some multiple integrals involving determinants. J. Indian Math. Soc. (N.S.) 19 133–151 (1956). MR0079647 [10] D UMITRIU , I., E DELMAN , A. and S HUMAN , G. (2007). MOPS: Multivariate orthogonal polynomials (symbolically). J. Symbolic Comput. 42 587–620. MR2325917 [11] F ORRESTER , P. J. (1993). The spectrum edge of random matrix ensembles. Nuclear Phys. B 402 709–728. MR1236195 [12] F ORRESTER , P. J. (2000). Painlevé transcendent evaluation of the scaled distribution of the smallest eigenvalue in the Laguerre orthogonal and symplectic ensembles. Available at http://www.citebase.org/abstract?id=oai:arXiv.org:nlin/0005064. [13] F ORRESTER , P. J. (2006). Hard and soft edge spacing distributions for random matrix ensembles with orthogonal and symplectic symmetry. Nonlinearity 19 2989–3002. MR2275509 [14] F ORRESTER , P. J., NAGAO , T. and H ONNER , G. (1999). Correlations for the orthogonalunitary and symplectic-unitary transitions at the hard and soft edges. Nuclear Phys. B 553 601–643. MR1707162 [15] G EMAN , S. (1980). A limit theorem for the norm of random matrices. Ann. Probab. 8 252–261. MR566592 [16] G OHBERG , I. C. and K RE ĬN , M. G. (1969). Introduction to the Theory of Linear Nonselfadjoint Operators. Translations of Mathematical Monographs 18. Amer. Math. Soc., Providence, RI. Translated from the Russian by A. Feinstein. MR0246142 [17] H ANLON , P. J., S TANLEY, R. P. and S TEMBRIDGE , J. R. (1992). Some combinatorial aspects of the spectra of normally distributed random matrices. In Hypergeometric Functions on Domains of Positivity, Jack Polynomials, and Applications (Tampa, FL, 1991). Contemporary Mathematics 138 151–174. Amer. Math. Soc., Providence, RI. MR1199126 [18] J OHANSSON , K. (2001). Discrete orthogonal polynomial ensembles and the Plancherel measure. Ann. of Math. (2) 153 259–296. MR1826414 [19] J OHNSTONE , I. M. (2001). On the distribution of the largest eigenvalue in principal components analysis. Ann. Statist. 29 295–327. MR1863961 [20] L AX , P. D. (2002). Functional Analysis. Pure and Applied Mathematics (New York). Wiley, New York. MR1892228 [21] M ACDONALD , I. G. (1995). Symmetric Functions and Hall Polynomials, 2nd ed. Oxford Univ. Press, New York. MR1354144 [22] M AR ČENKO , V. A. and PASTUR , L. A. (1967). Distribution of eigenvalues in certain sets of random matrices. Mat. Sb. (N.S.) 72 507–536. MR0208649 [23] M EHTA , M. L. (2004). Random Matrices, 3rd ed. Pure and Applied Mathematics (Amsterdam) 142. Academic Press, Amsterdam. MR2129906 [24] M UIRHEAD , R. J. (1982). Aspects of Multivariate Statistical Theory. Wiley, New York. MR652932 [25] PAUL , D. (2007). Asymptotics of sample eigenstructure for a large dimensional spiked covariance model. Statist. Sinica 17 1617–1642. MR2399865 [26] P RÄHOFER , M. and S POHN , H. (2002). Current fluctuations for the totally asymmetric simple exclusion process. In In and Out of Equilibrium (Mambucaba, 2000). Progress in Probability 51 185–204. Birkhäuser, Boston. MR1901953 [27] S TANLEY, R. P. (1989). Some combinatorial properties of Jack symmetric functions. Adv. Math. 77 76–115. MR1014073 1328 D. WANG [28] S ZEG Ő , G. (1975). Orthogonal Polynomials, 4th ed. Colloquium Publications XXIII. Amer. Math. Soc., Providence, RI. MR0372517 [29] T RACY, C. A. and W IDOM , H. (1994). Level-spacing distributions and the Airy kernel. Comm. Math. Phys. 159 151–174. MR1257246 [30] T RACY, C. A. and W IDOM , H. (1996). On orthogonal and symplectic matrix ensembles. Comm. Math. Phys. 177 727–754. MR1385083 [31] T RACY, C. A. and W IDOM , H. (1998). Correlation functions, cluster functions, and spacing distributions for random matrices. J. Statist. Phys. 92 809–835. MR1657844 D EPARTMENT OF M ATHEMATICS B RANDEIS U NIVERSITY WALTHAM , M ASSACHUSETTS 02454 USA E- MAIL : [email protected]