[tex]\cos^2\dfrac{9\pi}8=\dfrac{1+\cos\frac{9\pi}4}2=\dfrac{1+\cos\frac\pi4}2=\dfrac{\sqrt2+1}{2\sqrt2}=\dfrac{2+\sqrt2}4[/tex]
[tex]\implies \cos\dfrac{9\pi}8=\pm\sqrt{\dfrac{2+\sqrt2}4}[/tex]
Because [tex]\pi<\dfrac{9\pi}8<\dfrac{3\pi}2[/tex] (third quadrant), and [tex]\cos x<0[/tex] for [tex]x[/tex] in this interval, you must take the negative root, so
[tex]\cos\dfrac{9\pi}8=-\sqrt{\dfrac{2+\sqrt2}4}=-\dfrac{\sqrt{2+\sqrt2}}2[/tex]
