<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Bill,<div><br></div><div>After my tests I agree that reading the headers in DICOM files is a surprisingly expensive operation as such it should be minimized. The coping of the MDAs is insignificant performance wise. I believe that the best solution would be to have a dedicated DICOM series readers, which also removes the extra header reads needed for the name generation as well as the extra one in the UpdateOutputInformation.</div><div><br></div><div>If we assume that the usually way to utilize the reader is to just Update, or stream Update, then the additional read of the headers appears unnecessary. </div><div><br></div><div>I believe a solution would be to make the GetMDDA method smarter, and by default update this MDDA in the UpdateData. A time stamp would need to be used for the MDDA to check when it needs to be updated in the UpdateData methods. For streaming, the first time through would require reading all of the headers for the MDDA, this should bring the time stamp up to date. The GetMDDA methods could also check this timestamp and perform the reading of the headers if it's out of date. This is my best current idea on how to maintain the 1) and 2) I previously mentioned.</div><div><br></div><div>Brad</div><div><br><div><div>On Mar 23, 2010, at 12:33 PM, Bill Lorensen wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Brad,<br><br>I have an itk 2.8 checkout. The difference is due to the processing of<br>all files in the GenerateOutputInformation method. In the past, only<br>two files were processed. If I restrict the number of files to 2<br>rather that number of files, I get pretty reasonable speeds.<br><br>Roger,<br><br>As an experiment (and definitely not a fix!), can you in the method<br>void ImageSeriesReader<TOutputImage><br>::GenerateOutputInformation(void)<br><br>change the line:<br>for ( int i = 0; i != numberOfFiles; ++i )<br>to<br> for ( int i = 0; i != 2; ++i )<br><br>and rerun your tests.<br><br>Bill<br><br><br>On Tue, Mar 23, 2010 at 8:59 AM, Bradley Lowekamp<br><<a href="mailto:blowekamp@mail.nih.gov">blowekamp@mail.nih.gov</a>> wrote:<br><blockquote type="cite">Bill,<br></blockquote><blockquote type="cite">That is only the half of it. Every time an ImageFileReader is used 3 MDDs<br></blockquote><blockquote type="cite">(meta data dictionaries) are created, one in the ImageIO, one in the<br></blockquote><blockquote type="cite">ImageFileReader, and one in the output Image. This is in addition to the two<br></blockquote><blockquote type="cite">copies, you pointed out in ImageSeriesReader. Clearly reading with an<br></blockquote><blockquote type="cite">ImageFileReader the MDD scales very poorly as the it's size increases. I<br></blockquote><blockquote type="cite">still have the remaining performance questions:<br></blockquote><blockquote type="cite">How much time is spent coping the MDD vs reading? (leaning towards reading<br></blockquote><blockquote type="cite">as very expensive)<br></blockquote><blockquote type="cite">As pointed out in Roger's most recent performance tests, there appears to be<br></blockquote><blockquote type="cite">some additional performance problems in the UpdateData, part. This is<br></blockquote><blockquote type="cite">independent of the additional MDD read in the UpdateOutputInformation. This<br></blockquote><blockquote type="cite">is definitely another problem, perhaps inside the DICOM library.<br></blockquote><blockquote type="cite">The change of moving (apparently duplicating) the copying to MDDs to the MDD<br></blockquote><blockquote type="cite">array was added over a year ago, when streaming support was added. If I<br></blockquote><blockquote type="cite">recall correctly the two motivating factors were 1) the MDD array is output<br></blockquote><blockquote type="cite">information and logically should be updating during the<br></blockquote><blockquote type="cite">UpdateOutputInformation part of the pipeline 2) when streaming each file<br></blockquote><blockquote type="cite">should not need to be read to create the MMD array. I don't recall where<br></blockquote><blockquote type="cite">this discussion took place right now.<br></blockquote><blockquote type="cite">I will run some performance test to try to figure out where the time is<br></blockquote><blockquote type="cite">being spent. Without changing 1 from above, I am not sure how much could be<br></blockquote><blockquote type="cite">gained.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Looking at the performance numbers of the Read Directory part, I would guess<br></blockquote><blockquote type="cite">that the meta data is also read there. I believe that an idea solution would<br></blockquote><blockquote type="cite">only read this information once. But that is beyond this scope.<br></blockquote><blockquote type="cite">Brad<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On Mar 22, 2010, at 11:20 PM, Bill Lorensen wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Brad,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">It looks like the meta data array is populated in both the<br></blockquote><blockquote type="cite">GenerateOutputInformation and GenerateData. Also all slices are<br></blockquote><blockquote type="cite">processed in GenerateOutputInformation. In 2.8, only 2 slices were<br></blockquote><blockquote type="cite">processed.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Why were these changes made? We are also seeing bad dicom performance<br></blockquote><blockquote type="cite">in Slicer3.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Bill<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On Mon, Mar 22, 2010 at 6:24 AM, Bradley Lowekamp<br></blockquote><blockquote type="cite"><<a href="mailto:blowekamp@mail.nih.gov">blowekamp@mail.nih.gov</a>> wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Hello,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Can you please tell us a little more about your test data and computer. What<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">kind of file system is the data on ( locale or network)? How much memory<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">does the computer have? What is the size of the data? What is the native<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">pixel type of the data? What are the actual timings? Does the execution seem<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">to be CPU or IO bound?<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">One of the changes that was made to the class was to populate the<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">MetaDataArray in the UpdataOutputInformation phase of the instead of the<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">UpdateOutputData part. This should be just reading the headers of the files<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">in the series. There were several reasons this change was made. To help<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">determine the cause of your slowness, lets break up the timing a little<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">further.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Could you please call:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">start timer<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">reader->UpdateOutputInformation();<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">lap timer<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">reader->UpdateLargestPossibleRegion();<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">stop timer<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">And post the timing results.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Thanks,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Brad<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On Mar 21, 2010, at 2:52 PM, Roger Bramon Feixas wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">This week we updated our ITK version from 2.8 to 3.16 and we noticed the<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">medical models are loading 2x slower using the 3.16 ITK version. We use<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">itk::ImageSeriesReader and the problem is focused in its Update() method.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I attached a simple test program which reproduces the problem and where we<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">can see that the Update() method is 2 times slower using ITK 3.16 vs. ITK<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">2.8.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">We compiled both versions using Visual Studio 2008 on Windows XP 32bits and<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> we don't known if this problem also occurs in other platforms.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I wonder if other itk users have this same performance problem and if there<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">is anybody can help us in order to solve it.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Thanks!<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Roger<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><test.zip><ATT00001..txt><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">========================================================<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Bradley Lowekamp<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Lockheed Martin Contractor for<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Office of High Performance Computing and Communications<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">National Library of Medicine<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><a href="mailto:blowekamp@mail.nih.gov">blowekamp@mail.nih.gov</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">_____________________________________<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Powered by <a href="http://www.kitware.com">www.kitware.com</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Visit other Kitware open-source projects at<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><a href="http://www.kitware.com/opensource/opensource.html">http://www.kitware.com/opensource/opensource.html</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Kitware offers ITK Training Courses, for more information visit:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><a href="http://www.kitware.com/products/protraining.html">http://www.kitware.com/products/protraining.html</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Please keep messages on-topic and check the ITK FAQ at:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><a href="http://www.itk.org/Wiki/ITK_FAQ">http://www.itk.org/Wiki/ITK_FAQ</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Follow this link to subscribe/unsubscribe:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><a href="http://www.itk.org/mailman/listinfo/insight-users">http://www.itk.org/mailman/listinfo/insight-users</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">========================================================<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Bradley Lowekamp<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Lockheed Martin Contractor for<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Office of High Performance Computing and Communications<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">National Library of Medicine<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><a href="mailto:blowekamp@mail.nih.gov">blowekamp@mail.nih.gov</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote></div></blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Helvetica" size="3" style="font: 12.0px Helvetica">========================================================</font></p><p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Helvetica" size="3" style="font: 12.0px Helvetica">Bradley Lowekamp<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font></p><p style="margin: 0.0px 0.0px 0.0px 0.0px"><font class="Apple-style-span" face="Arial"><span class="Apple-style-span" style="font-family: Arial; "><span class="Apple-style-span" style="font-family: Arial; ">Lockheed Martin </span></span></font><font face="Helvetica" size="3" style="font: 12.0px Helvetica">Contractor for</font></p><p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Helvetica" size="3" style="font: 12.0px Helvetica">Office of High Performance Computing and Communications</font></p><p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Helvetica" size="3" style="font: 12.0px Helvetica">National Library of Medicine<span class="Apple-converted-space"> </span></font></p><p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Helvetica" size="3" style="font: 12.0px Helvetica"><a href="mailto:blowekamp@mail.nih.gov">blowekamp@mail.nih.gov</a></font></p><br class="Apple-interchange-newline"></span></span>
</div>
<br></div></body></html>