<div dir="ltr">yes - there is something you are missing. read the code below: <div><br></div><div><div style="font-family:arial,sans-serif;font-size:13px"> /* Begin threaded gradient modification.</div><div style="font-family:arial,sans-serif;font-size:13px">
* Scale by gradient scales, then estimate the learning</div><div style="font-family:arial,sans-serif;font-size:13px"> * rate if options are set to (using the scaled gradient),</div><div style="font-family:arial,sans-serif;font-size:13px">
* then modify by learning rate. The m_Gradient variable</div><div style="font-family:arial,sans-serif;font-size:13px"> * is modified in-place. */</div><div style="font-family:arial,sans-serif;font-size:13px"> this->ModifyGradientByScales();</div>
<div style="font-family:arial,sans-serif;font-size:13px"> this->EstimateLearningRate();</div><div style="font-family:arial,sans-serif;font-size:13px"> this->ModifyGradientByLearningRate();</div></div><div><br></div>
<div>the call to <span style="font-family:arial,sans-serif;font-size:13px"> this->ModifyGradientByScales()</span><span style="font-family:arial,sans-serif;font-size:13px">; changes the gradient according the scales, as the name suggests. the v4 optimizers all behave in this general manner although this is taken from the gradient descent class.</span></div>
<div><br></div><div>so - the transform expects that the update was already modified by the scales so that the only thing the transform needs to do ( if anything at all) is multiply by a scalar</div><div><br></div>
<div>also, you are looking at the base class which is only used if the derived class did not implement UpdateTransformParameters. for instance, the GaussianDisplacementField transform will also smooth the parameters when this function is called.</div>
<div><br></div><div>is this clear enough?</div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div><br></div>brian<br><div><br></div><div><br></div></div>
<br><br><div class="gmail_quote">On Tue, May 7, 2013 at 11:59 AM, Joël Schaerer <span dir="ltr"><<a href="mailto:joel.schaerer@gmail.com" target="_blank">joel.schaerer@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>I spent a while looking at the v4
optimization framework. I can follow your reasoning until
UpdateTransformParameters is called on the transform. However, at
this state, the old scaling is still done:<br>
<br>
itkTransform.hxx:<br>
if( factor == 1.0 )<br>
{<br>
for( NumberOfParametersType k = 0; k < numberOfParameters;
k++ )<br>
{<br>
this->m_Parameters[k] += update[k];<br>
}<br>
}<br>
else<br>
{<br>
for( NumberOfParametersType k = 0; k < numberOfParameters;
k++ )<br>
{<br>
this->m_Parameters[k] += update[k] * factor;<br>
}<br>
}<br>
<br>
which makes sense, since parameters scales are an optimizer
concept that transforms know nothing about.<br>
<br>
So (if I understand correctly), the code has been shuffled around
quite a bit, but the behavior is still the same.<br>
<br>
Is there something I'm missing?<br>
<br>
joel<div><div class="h5"><br>
<br>
<br>
On 07/05/2013 16:40, brian avants wrote:<br>
</div></div></div><div><div class="h5">
<blockquote type="cite">
<div dir="ltr">also - to take away a bit of the "mystery"
surrounding v4 optimization, let's see how the gradient descent
AdvanceOneStep function works:
<div><br>
</div>
<div>
<div>void</div>
<div>GradientDescentOptimizerv4</div>
<div>::AdvanceOneStep()</div>
<div>{</div>
<div> itkDebugMacro("AdvanceOneStep");</div>
<div><br>
</div>
<div> /* Begin threaded gradient modification.</div>
<div> * Scale by gradient scales, then estimate the learning</div>
<div> * rate if options are set to (using the scaled
gradient),</div>
<div> * then modify by learning rate. The m_Gradient
variable</div>
<div> * is modified in-place. */</div>
<div> this->ModifyGradientByScales();</div>
<div> this->EstimateLearningRate();</div>
<div> this->ModifyGradientByLearningRate();</div>
<div><br>
</div>
<div> try</div>
<div> {</div>
<div> /* Pass graident to transform and let it do its own
updating */</div>
<div> this->m_Metric->UpdateTransformParameters(
this->m_Gradient );</div>
<div> }</div>
<div> catch ( ExceptionObject & )</div>
<div> {</div>
<div> this->m_StopCondition = UPDATE_PARAMETERS_ERROR;</div>
<div> this->m_StopConditionDescription <<
"UpdateTransformParameters error";</div>
<div> this->StopOptimization();</div>
<div><br>
</div>
<div> // Pass exception to caller</div>
<div> throw;</div>
<div> }</div>
<div><br>
</div>
<div> this->InvokeEvent( IterationEvent() );</div>
<div>}</div>
<div><br>
</div>
<div><br>
</div>
<div>i hope this does not look too convoluted.
then the base metric class does this:</div>
<div><br>
</div>
<div>
<div>template<unsigned int TFixedDimension, unsigned int
TMovingDimension, class TVirtualImage></div>
<div>void</div>
<div>ObjectToObjectMetric<TFixedDimension,
TMovingDimension, TVirtualImage></div>
<div>::UpdateTransformParameters( const DerivativeType &
derivative, ParametersValueType factor )</div>
<div>{</div>
<div> /* Rely on transform::UpdateTransformParameters to
verify proper</div>
<div> * size of derivative */</div>
<div>
this->m_MovingTransform->UpdateTransformParameters(
derivative, factor );</div>
<div>}</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>so the transform parameters should be updated in
a way that is consistent with: </div>
</div>
<div><br>
</div>
<div><span style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:13px">newPosition[j]
= currentPosition[j] + transformedGradient[j] * factor /
scales[j];</span><br style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:13px">
</div>
<div><span style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<div><font face="arial, sans-serif" color="#500050">factor
defaults to 1 .... anyway, as you can infer from the above
discussion, even the basic gradient descent optimizer can be
used to take " regular steps " if you want.</font></div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div><br>
</div>
brian<br>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<br>
<div class="gmail_quote">On Tue, May 7, 2013 at 10:23 AM, brian
avants <span dir="ltr"><<a href="mailto:stnava@gmail.com" target="_blank">stnava@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">brad
<div><br>
</div>
<div>did this issue ever go up on jira? i do remember
discussing with you at a meeting. our solution is in
the v4 optimizers.</div>
<div><br>
</div>
<div>the trivial additive parameter update doesnt work in
more general cases e.g. when you need to compose
parameters with parameter updates. </div>
<div><br>
</div>
<div>to resolve this limitation, the v4 optimizers pass
the update step to the transformations </div>
<div><br>
</div>
<div>this implements the idea that " the transforms know
how to update themselves " </div>
<div><br>
</div>
<div>there are several other differences, as nick pointed
out, that reduce the need for users to experiment with
scales .</div>
<div><br>
</div>
<div>for basic scenarios like that being discussed by
joel, i prefer the conjugate gradient optimizer with
line search. </div>
<div><br>
</div>
<div>
<div>itkConjugateGradientLineSearchOptimizerv4.h</div>
<div><br>
</div>
<div>when combined with the scale estimators, this leads
to registration algorithms with very few parameters to
tune. 1 parameter if you dont consider
multi-resolution.</div>
<span><font color="#888888">
</font></span></div>
</div>
<div class="gmail_extra"><span><font color="#888888"><br clear="all">
<div>
<div><br>
</div>
brian<br>
<div><br>
</div>
<div><br>
</div>
</div>
</font></span>
<div>
<div>
<br>
<br>
<div class="gmail_quote">On Tue, May 7, 2013 at 9:27
AM, Nick Tustison <span dir="ltr"><<a href="mailto:ntustison@gmail.com" target="_blank">ntustison@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Brad,<br>
<br>
I certainly don't disagree with Joel's findings.
It seems like a<br>
good fix which should be put up on gerrit. There
were several<br>
components that we kept in upgrading the
registration framework.<br>
The optimizers weren't one of them.<br>
<br>
Also, could you elaborate a bit more on the
"convoluted" aspects<br>
of parameter advancement? There's probably a
reason for it and<br>
we could explain why.<br>
<span><font color="#888888"><br>
Nick<br>
</font></span>
<div>
<div><br>
<br>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div>