Let sec−1x=θ, so x=secθ. For x<−1 the principal value of sec−1 lies in the second quadrant:
2π<θ<π
The step where sign matters. From sec2θ−1=tan2θ,
x2−1=tan2θ=∣tanθ∣
and in the second quadrant tanθ is negative, so ∣tanθ∣=−tanθ. Hence
x2−1−1=−tanθ−1=tanθ1=cotθ
The expression is therefore cot−1(cotθ), and since θ∈(2π,π) already lies inside the principal range (0,π) of cot−1,
cot−1(cotθ)=θ=sec−1x
cot−1(x2−1−1)=sec−1x,x<−1
Check at x=−2: the expression evaluates to 2.0944, and sec−1(−2)=32π=2.0944. (The value π−sec−1x would give 1.0472, which is not what the expression returns.)