Step 1 :First, we recognize that this is an integral of the form \(\int u dv\), where \(u = \sqrt{x} = x^{\frac{1}{2}}\) and \(dv = (5x+3)^2 dx\).
Step 2 :We need to find \(du\) and \(v\). The derivative of \(u\) with respect to \(x\) is \(\frac{1}{2}x^{-\frac{1}{2}}\), so \(du = \frac{1}{2}x^{-\frac{1}{2}} dx\).
Step 3 :To find \(v\), we need to integrate \(dv\). The integral of \((5x+3)^2\) with respect to \(x\) is \(\frac{5}{3}x^3 + 3x^2 + C\), so \(v = \frac{5}{3}x^3 + 3x^2 + C\).
Step 4 :Now we can use the integration by parts formula, \(\int u dv = uv - \int v du\). Substituting our expressions for \(u\), \(v\), and \(du\), we get \(\int \sqrt{x}(5x+3)^2 dx = x^{\frac{1}{2}}(\frac{5}{3}x^3 + 3x^2 + C) - \int (\frac{5}{3}x^3 + 3x^2 + C)\frac{1}{2}x^{-\frac{1}{2}} dx\).
Step 5 :Simplify the integral on the right to get \(\int \sqrt{x}(5x+3)^2 dx = x^{\frac{1}{2}}(\frac{5}{3}x^3 + 3x^2 + C) - \frac{5}{6}x^2 - \frac{3}{2}x + C\).
Step 6 :Finally, simplify the expression to get the final answer: \(\boxed{\frac{175}{2} x^{\frac{7}{2}}+75 x^{\frac{5}{2}}+\frac{27}{2} x^{\frac{3}{2}}+c}\).