[tex]x^2\dfrac{\mathrm d^2y}{\mathrm dx^2}-3x\dfrac{\mathrm dy}{\mathrm dx}+20y=0[/tex]
This is your standard Euler-Cauchy ODE, which means we can substitute [tex]y=x^r[/tex], then determine the possible values of [tex]r[/tex] to generate the set of fundamental solutions.
[tex]y=x^r[/tex]
[tex]y'=rx^{r-1}[/tex]
[tex]y''=r(r-1)x^{r-2}[/tex]
[tex]x^2r(r-1)x^{r-2}-3xrx^{r-1}+20x^r=0[/tex]
[tex]r(r-1)x^r-3rx^r+20x^r=0[/tex]
[tex]r(r-1)-3r+20=0[/tex]
[tex]r^2-4r+20=0\implies r=2\pm4i[/tex]
For [tex]r=2+4i[/tex], we get the solution
[tex]x^{2+4i}=x^2x^{4i}=x^2e^{4i\ln x}=x^2(\cos(4\ln x)+i\sin(4\ln x))[/tex]
We'll get something almost identical with [tex]r=2-4i[/tex] - namely, two fundamental solutions [tex]x^2\cos(4\ln x)[/tex] and [tex]x^2\sin(4\ln x)[/tex], and so the general solution will be
[tex]y_c=C_1x^2\cos(4\ln x)+C_2x^2\sin(4\ln x)[/tex]
Given [tex]y(1)=2[/tex], we have
[tex]2=C_1[/tex]
and from [tex]y'(1)=-8[/tex] we get
[tex]{y_c}'=2x^2\cos(4\ln x)+C_2x^2\sin(4\ln x)[/tex]
[tex]\implies-8=4+4C_2\implies C_2=-3[/tex]
so that the particular solution is
[tex]y_c=2x^2\cos(4\ln x)-3x^2\sin(4\ln x)[/tex]
