The answer.
For a cost function of 
C(x) = 0.5*x'*Lxx*x + 0.5*u'*Luu*u
The value function is
V(x) = V0 + Vx*x + 0.5*x'*Vxx*x

For a cost function of 
C(x) = x'*Lxx*x + u'*Luu*u
The value function is
V(x) = V0 + Vx*x + x'*Vxx*x
But the first derivative is screwed up a little bit.

So for dp-grid, dp-bal, and dp-random you were doing the wrong thing.
Not sure how far back this goes.

/************************************************************************/

What is the correct value function (1/2 or no 1/2)?
V(x) = V0 + Vx*x + x'*Vxx*x?
V(x) = V0 + Vx*x + 1/2*(x'*Vxx*x)?
What is the correct derivative?
Vx = Vxx*x
Vx = 2*Vxx*x

/************************************************************************/

  /* Init DDP stuff */
  lqrd = init_ddp( the_d );
  /* Set default/global LQR S and K */
  for ( i = 0; i < N_STATE_DIMENSIONS; i++ )
    {
      for( j = 0; j < N_STATE_DIMENSIONS; j++ )
        lqr_s[i][j] = lqrd->P_ss[i][j];
    }

/************************************************************************/
/* Value function defined by LQR controller */

float lqr_value( float *state )
{
...
  /* Compute value */
  for ( i = 0; i < N_STATE_DIMENSIONS; i++ )
    {
      temp[i] = 0;
      for( j = 0; j < N_STATE_DIMENSIONS; j++ )
        temp[i] += lqr_s[i][j]*( state[j] - the_desired_state[j] );
    }
  value = 0;
  for( j = 0; j < N_STATE_DIMENSIONS; j++ )
    value += temp[j]*( state[j] - the_desired_state[j] );


>> a = [   0.999291963315052  0.00962740370979986 
  -0.141607337761918    0.925480741747187 
]

a =

    0.9993    0.0096
   -0.1416    0.9255

>> b = [ 0.000149625933865138 
   0.029925187032419 
]

b =

    0.0001
    0.0299

>> q = 0.1*0.01*eye(2)

q =

   1.0e-03 *

    1.0000         0
         0    1.0000

>> r = 0.01*eye(1)

r =

    0.0100

>> [k,s,e] = dlqr( a, b, q, r )

k =

    0.0090    0.0208


s =

    0.1288    0.0034
    0.0034    0.0074


e =

    0.9577
    0.9665

>> s0 = [1
2]

s0 =

     1
     2

>> s0'*s*s0

ans =

    0.1722


s1 = s0;
cost = 0;
for i = 1:1000
 u = - k*s1;
 cost = cost + s1'*q*s1 + u'*r*u;
 s1 = a*s1 + b*u;
end;

cost =

    0.1722

For a cost function s1'*Q*s1 + u'*R*u;
So matlab V(x) = x'*S*x

In my lqrd thing.
Lxx = 2nd derivative of cost function, so
L(X) = 1/2*(x'*Lxx*x + u'*Luu*u)

