Calculus Optimization Problem - Need Serious Help

[Endless]

Let's call X the angle between c and h (look at Del's picture).

c = 8*cos(X) [cos(X) = c/8]
b/2 = c*sin(X) [sin(X) = (b/2)/c]
b/2 = 8*cos(X)*sin(X)
h = 8*cos(X)*sqrt(1-sin(X)^2)
h = 8*cos(X)^2

So, A = h*b/2 = 64*cos(X)^3*sin(X)
Transformate that horror:
A = 64*cos(X)^2*(0.5*sin(2*X))
A = 64*(1/2)*(1+cos(2*X))*(1/2)*sin(2*X)
A = 16*sin(2*X)+8*sin(4*X)
(find your trig book, there are a lot of ways to manipulate cos and sin, and you do end up with this)

Derivate that (fun xD)
A' = 32*(cos(2*X)+cos(4*X)) = 0
cos(2*X)+cos(4*X) = 0
Another transformation
cos(pi-2*X) = cos(4*X)
pi-2*X = 4*X
X = pi/6

From this you find:
c=4*sqrt(3)
b=4*sqrt(3)
h=6
A=12*sqrt(3)

We now have the mathematical justification of the obvious (to me) result to your question: the biggest isoceles triangle that fits in a circle is an equilateral triangle.

Edit: just reread Del's posts. I think you get:
h = (32-(b/2)^2-(b/2)^4)^(1/4)
A = hb/2
A = (1/2)*b*(32-(b/2)^2-(b/2)^4)^(1/4)
And have fun with solving that derivate = 0.

Edit^2:
Reread the orginal post, whose method is right.
A=x*(4+sqrt(16-x^2))

Derivate that, you get:
4+sqrt(16-x^2)-x*(x/(sqrt(16-x^2))) = 0
4+sqrt(16-x^2) = (x^2)/(sqrt(16-x^2))
x != 0, x != -4, x != 4
4*sqrt(16-x^2) + 16 - x^2 = x^2
sqrt(16-x^2) = 1/2 * x^2 - 4
16-x^2 = (1/2 * x^2 - 4)^2
16 = 1/4 * x^4 - 3* x^2 +16
(1/4) x^4 = 3 * x^2
So... 1/4 x^2 = 3
x^2=12
x=2*sqrt(3)