The cleanest route is to turn both terms on the right into inverse tangents, then add them with the tangent addition formula.
Step 1 — convert sin − 1 5 13 \sin^{-1}\frac{5}{13} sin − 1 13 5 . Let α = sin − 1 5 13 \alpha=\sin^{-1}\frac{5}{13} α = sin − 1 13 5 , so sin α = 5 13 \sin\alpha=\frac{5}{13} sin α = 13 5 with α ∈ [ − π 2 , π 2 ] \alpha\in\left[-\frac{\pi}{2},\frac{\pi}{2}\right] α ∈ [ − 2 π , 2 π ] . The sine is positive, so α \alpha α is acute and the cosine is positive:
cos α = 1 − 25 169 = 12 13 , tan α = 5 12 . \cos\alpha=\sqrt{1-\frac{25}{169}}=\frac{12}{13},\qquad \tan\alpha=\frac{5}{12}. cos α = 1 − 169 25 = 13 12 , tan α = 12 5 .
Hence α = tan − 1 5 12 \alpha=\tan^{-1}\frac{5}{12} α = tan − 1 12 5 (the 5 5 5 –12 12 12 –13 13 13 triangle).
Step 2 — convert cos − 1 3 5 \cos^{-1}\frac35 cos − 1 5 3 . Let β = cos − 1 3 5 \beta=\cos^{-1}\frac35 β = cos − 1 5 3 , so cos β = 3 5 \cos\beta=\frac35 cos β = 5 3 with β ∈ [ 0 , π ] \beta\in[0,\pi] β ∈ [ 0 , π ] . The cosine is positive, so β \beta β is acute and
sin β = 1 − 9 25 = 4 5 , tan β = 4 3 , \sin\beta=\sqrt{1-\frac{9}{25}}=\frac45,\qquad \tan\beta=\frac43, sin β = 1 − 25 9 = 5 4 , tan β = 3 4 ,
giving β = tan − 1 4 3 \beta=\tan^{-1}\frac43 β = tan − 1 3 4 (the 3 3 3 –4 4 4 –5 5 5 triangle).
Step 3 — check the condition before adding. The addition formula
tan − 1 u + tan − 1 v = tan − 1 u + v 1 − u v \tan^{-1}u+\tan^{-1}v=\tan^{-1}\frac{u+v}{1-uv} tan − 1 u + tan − 1 v = tan − 1 1 − uv u + v
requires u v < 1 uv<1 uv < 1 if the sum is to stay inside ( − π 2 , π 2 ) \left(-\frac{\pi}{2},\frac{\pi}{2}\right) ( − 2 π , 2 π ) with no π \pi π correction. Here
u v = 5 12 ⋅ 4 3 = 20 36 = 5 9 < 1 , uv=\frac{5}{12}\cdot\frac43=\frac{20}{36}=\frac59<1, uv = 12 5 ⋅ 3 4 = 36 20 = 9 5 < 1 ,
so the plain form is the right one.
Step 4 — add.
tan − 1 5 12 + tan − 1 4 3 = tan − 1 ( 5 12 + 4 3 1 − 5 9 ) = tan − 1 ( 5 + 16 12 4 9 ) = tan − 1 ( 21 12 ⋅ 9 4 ) = tan − 1 189 48 = tan − 1 63 16 . \tan^{-1}\frac{5}{12}+\tan^{-1}\frac43=\tan^{-1}\left(\frac{\frac{5}{12}+\frac43}{1-\frac59}\right)=\tan^{-1}\left(\frac{\frac{5+16}{12}}{\frac49}\right)=\tan^{-1}\left(\frac{21}{12}\cdot\frac94\right)=\tan^{-1}\frac{189}{48}=\tan^{-1}\frac{63}{16}. tan − 1 12 5 + tan − 1 3 4 = tan − 1 ( 1 − 9 5 12 5 + 3 4 ) = tan − 1 ( 9 4 12 5 + 16 ) = tan − 1 ( 12 21 ⋅ 4 9 ) = tan − 1 48 189 = tan − 1 16 63 .
Therefore
sin − 1 5 13 + cos − 1 3 5 = tan − 1 63 16 , \sin^{-1}\frac{5}{13}+\cos^{-1}\frac35=\tan^{-1}\frac{63}{16}, sin − 1 13 5 + cos − 1 5 3 = tan − 1 16 63 ,
which is what was to be proved.
Step 5 — numerical check. tan − 1 63 16 = 1.32209 \tan^{-1}\frac{63}{16}=1.32209 tan − 1 16 63 = 1.32209 radians, while sin − 1 5 13 + cos − 1 3 5 = 0.39479 + 0.92730 = 1.32209 \sin^{-1}\frac{5}{13}+\cos^{-1}\frac35=0.39479+0.92730=1.32209 sin − 1 13 5 + cos − 1 5 3 = 0.39479 + 0.92730 = 1.32209 . Both sides also lie in ( 0 , π 2 ) \left(0,\frac{\pi}{2}\right) ( 0 , 2 π ) , so no branch adjustment was hiding in the algebra.