Multi-thread access to the Simple Viewer DLL

PDF-XChange Viewer SDK for Developer's
(ActiveX and Simple DLL Versions)

Moderators: TrackerSupp-Daniel, Tracker Support, Vasyl-Tracker Dev Team, Sean - Tracker, Chris - Tracker Supp, Tracker Supp-Stefan, Ivan - Tracker Software

Post Reply
johnr@etakeoff.com
User
Posts: 57
Joined: Thu Jan 01, 1970 12:00 am

Multi-thread access to the Simple Viewer DLL

Post by johnr@etakeoff.com » Tue Apr 26, 2011 1:24 pm

We have an application to generate TIFF files from PDF files/pages. It uses three threads. The main thread displays the current page for the user to view. A second thread opens the next file/page in the background. After the user enters some information, a third thread generates the TIFF file by writing the page to a large bitmap.

Once all three threads are going, it all bogs down. What would normally take 10-20 seconds takes 10-20 MINUTES. Is this something I shouldn't even attempt?

We're using the simple non-ActiveX viewer DLL. We also use the toolkit to read bookmarks. Memory is not a problem. When things are bogged down, Task Manager shows 50% CPU usage and plenty of memory.

Lzcat - Tracker Supp
Site Admin
Posts: 718
Joined: Thu Jun 28, 2007 8:42 am

Re: Multi-thread access to the Simple Viewer DLL

Post by Lzcat - Tracker Supp » Wed Apr 27, 2011 10:30 am

Hi John.
pxcview was not specially designed to work with the same document using several threads, using several documents in different threads is completely safe. You may work with the same document in several threads, but with serious restrictions: do not call PXCV_ReleasePageCachedData, PXCV_ReleaseCachedData and PXCV_Delete functions for document when other thread may work with this document to prevent GPF or other unexpected behaviour; when any thread renders page content document will be blocked and therefore most other functions will wait until rendering operation is complete.
Regarding to your workflow there is one potential problem which I can see:
johnr@etakeoff.com wrote:third thread generates the TIFF file by writing the page to a large bitmap
Generating bitmap from PDF is time consuming operation in case of complex context and / or large output bitmap. For example to render page with Letter size at 300 DPI you will get about 32 Mbytes HBITMAP and at least 32 Mbytes (same size as HBITMAP + some memory needed for rendering) for internal usage before conversion to HBITMAP. And if you want to use 600 DPI - replace 32 with 128 and a lot of additional time for simple copying data from internal storage to HBITMAP and from HBITMAP to your TIFF writer. Possible way to reduce this overhead - use PXCV_DrawPageToIStream function, this will skip HBITMAP creation and directly will save (encode) from internal storage to IStream. However anyway this will be time consuming operation and for that time document will be blocked, so all other threads will be not able to acccess to it (they will wait). Other possible solution - do not render complete page, but divide it into blocks and call PXCV_DrawPageToDIBSection/PXCV_DrawPageToDC several times (one for each block), and only after compose and save page. This will reduce memory allocation during rendering and may improve performance in cost of additional coding (actually Viewer and pxcview also render large pages using similar technique to maintain reasonable memory usage and performance).
All recomendations above were theoretical, so maybe better solution is to get sample project (sources, compiled code and test set of files) to illustrate a problem and target PC specifications (OS, CPU, memory and harddrive - because some temporary files may be used during rendering). In case of posting here please remove any serials and other sensitive informations from your code before compilation. Or you may send it to support@tracker-software.com with some description and link for this topic in message body (strongly recommended).
HTH.
Victor
Tracker Software
Project manager

Please archive any files posted to a ZIP, 7z or RAR file or they will be removed and not posted.

johnr@etakeoff.com
User
Posts: 57
Joined: Thu Jan 01, 1970 12:00 am

Re: Multi-thread access to the Simple Viewer DLL

Post by johnr@etakeoff.com » Wed Apr 27, 2011 10:53 am

I'm aware of all the issues with writing to a large bitmap. We do that regulalrly and it typically takes 10-20 seconds. What we're seeing here is quite different. It's easily 50 time longer.

But you've answered my basic question that it should be multi-thread safe. I'll dig into it further. BTW, we only read and display the files. We never modify them. And the different threads don't work on the same document simultaneously.

Thanks for your help.

Lzcat - Tracker Supp
Site Admin
Posts: 718
Joined: Thu Jun 28, 2007 8:42 am

Re: Multi-thread access to the Simple Viewer DLL

Post by Lzcat - Tracker Supp » Wed Apr 27, 2011 10:59 am

johnr@etakeoff.com wrote:the different threads don't work on the same document simultaneously
That's strange, in such case there should be no problems described above. Can you provide one or more typical files which you process and say typical bitmap dimentions?
Victor
Tracker Software
Project manager

Please archive any files posted to a ZIP, 7z or RAR file or they will be removed and not posted.

johnr@etakeoff.com
User
Posts: 57
Joined: Thu Jan 01, 1970 12:00 am

Re: Multi-thread access to the Simple Viewer DLL

Post by johnr@etakeoff.com » Wed Apr 27, 2011 11:29 am

We generate large 30"x40", 400 DPI, monochrome bitmaps. But let me dig deeper into this before I use any more of your time. My preliminary investigation indicated a possible multi-threading problem but that may not be the case.

Thanks for your help.

Lzcat - Tracker Supp
Site Admin
Posts: 718
Joined: Thu Jun 28, 2007 8:42 am

Re: Multi-thread access to the Simple Viewer DLL

Post by Lzcat - Tracker Supp » Wed Apr 27, 2011 11:53 am

I think problem is in size - your image require 732 MBytes to be generated as HBITMAP, regardless TIFF format, and internal representation will use temporary file to keep such large data (instead of memory), and this may seriously degrade performance in case when two or more threads work with harddrive. Hardcoded limit is 128 Mb, please try split page into parts for rendering (each part should have less than 33 554 432 pixels).
Victor
Tracker Software
Project manager

Please archive any files posted to a ZIP, 7z or RAR file or they will be removed and not posted.

johnr@etakeoff.com
User
Posts: 57
Joined: Thu Jan 01, 1970 12:00 am

Re: Multi-thread access to the Simple Viewer DLL

Post by johnr@etakeoff.com » Wed Apr 27, 2011 12:02 pm

Actually, because the bitmaps are monchrome, it only requires 24 MB per bitmap (30*40*400*400/8). That's not much memory for today's machines. Back in 1996 when we started it was a serious challenge, even at 200 DPI.

Lzcat - Tracker Supp
Site Admin
Posts: 718
Joined: Thu Jun 28, 2007 8:42 am

Re: Multi-thread access to the Simple Viewer DLL

Post by Lzcat - Tracker Supp » Wed Apr 27, 2011 12:10 pm

Sorry, but the internal representation uses 32 bit's per pixel, not one, so the internal buffer uses 732 MBytes and they are stored on harddrive.

You may do an easy test - use 150 DPI instead of 400 and see whether the performance will be reduced or not.
Victor
Tracker Software
Project manager

Please archive any files posted to a ZIP, 7z or RAR file or they will be removed and not posted.

johnr@etakeoff.com
User
Posts: 57
Joined: Thu Jan 01, 1970 12:00 am

Re: Multi-thread access to the Simple Viewer DLL

Post by johnr@etakeoff.com » Wed Apr 27, 2011 1:37 pm

What you're saying is not true in our case. To create the TIFF file, we create a monchrome bitmap, select it into the DC, draw and save it as a TIFF file. There's no 32 bit bitmap.

To make sure, I ran our application on a pdf file that was 18x24 300 dpi. By your calculation this would be 311 MB. By my calculation it would be 10. I used Task Manager to monitor memory use while the TIFF file was generated. Memory usage started at 990 MB before I started the application. During generation it jumped to 1020 MB. A total change of 30 MB, not the 311 you predicted.

None the less, I'll keep an eye out for memory usage when diagnosing this problem. Thanks for your help.

John - Tracker Supp
Site Admin
Posts: 8204
Joined: Tue Jun 29, 2004 10:34 am
Location: Vancouver Island - Canada
Contact:

Re: Multi-thread access to the Simple Viewer DLL

Post by John - Tracker Supp » Wed Apr 27, 2011 2:23 pm

feel free to send an actual file 'Zipped' if you want us to take a look and analyze.

thanks
If posting files to this forum - you must archive the files to a ZIP, RAR or 7z file or they will not be uploaded - thank you.

Best regards
Tracker Support
http://www.tracker-software.com

Lzcat - Tracker Supp
Site Admin
Posts: 718
Joined: Thu Jun 28, 2007 8:42 am

Re: Multi-thread access to the Simple Viewer DLL

Post by Lzcat - Tracker Supp » Wed Apr 27, 2011 3:03 pm

johnr@etakeoff.com wrote:What you're saying is not so in our case. To create the TIFF file, we create a monchrome bitmap, select it into the DC, draw and save it as a TIFF file. There's no 32 bit bitmap.
You do not create such a bitmap, but we create for internal usage, and this is true (I'm developer of that code, so trust me - I know what I am saying). Yes, for some parameters (if you specify pxvrpf_RenderAsGray or pxvrpf_BlackAndWhite flags in PXV_CommonRenderParameters structure) we may use 16 bits per pixel (2 bytes), but no less.
johnr@etakeoff.com wrote:To make sure, I ran our application on a pdf file that was 18x24 300 dpi. By your calculation this would be 311 MB. By my calculation it would be 10. I used Task Manager to monitor memory use while the TIFF file was generated. Memory usage started at 990 MB before I started the application. During generation it jumped to 1020 MB. A total change of 30 MB, not the 311 you predicted.
As you can see in my comment above all internal buffers greater than 128 MB are stored in a temporary file, so memory is not used. Please inspect the %Temp% folder during TIFF creation - there will be an additional "ixc???.tmp" file(s) during creation.

When the buffer is stored on disk - memory usage is very small - several KB (typically 64KB or 128 KB), so you will not see it in the task manager. When you use just one buffer it is not a problem because Windows caching is in effect and data operations are sequential, most of the time.

However when using two of them simultaneously caching is not so effective and disk operations are not sequential - because access for file 1 and file 2 are mixed in the HDD queue.

Please try the following: split a complete page rect into small stripes or rects (to get the size below 128 MB using my calculation) and call PXCV_DrawPageToDC several times to draw all the parts.

For one file you will not see significant performance difference except increased memory usage, but when you run your multithreaded test - your app freezes should be resolved.
Victor
Tracker Software
Project manager

Please archive any files posted to a ZIP, 7z or RAR file or they will be removed and not posted.

johnr@etakeoff.com
User
Posts: 57
Joined: Thu Jan 01, 1970 12:00 am

Re: Multi-thread access to the Simple Viewer DLL

Post by johnr@etakeoff.com » Wed Apr 27, 2011 5:53 pm

You're saying that whenever you draw, you draw to a bitmap first, then bitblt that image to the DC?

That would explain A LOT.

Any way to turn that off - other than the banding.

John - Tracker Supp
Site Admin
Posts: 8204
Joined: Tue Jun 29, 2004 10:34 am
Location: Vancouver Island - Canada
Contact:

Re: Multi-thread access to the Simple Viewer DLL

Post by John - Tracker Supp » Wed Apr 27, 2011 7:15 pm

John,

Using any method - provided by any conversion process from any toolkit there will always be a bitmap generated at some point in the process - there has to be.

Additonally - what do you think a TIFF file is 'under the hood' ?

Its a Bitmap - just with different encoding to provide and support the specail properties assigned to a TIFF file!

Taking all the information provided above (both ways) - it would appear the most efficient method for you to achieve the desired result is to do as Victor suggested above;
we create a monchrome bitmap, select it into the DC, draw and save it as a TIFF file.
e.g. use vector based drawing: set the flags pxvrpf_UseVectorRenderer and pxvrpf_EmbeddedFontAsCurves (and also maybe pxvrpf_RenderAsGray as you use monochrome) into the PXV_CommonRenderParameters structure used in the PXCV_DrawPageToDC function.

that should get you the best possible results when taking all matters into account.

HTH
If posting files to this forum - you must archive the files to a ZIP, RAR or 7z file or they will not be uploaded - thank you.

Best regards
Tracker Support
http://www.tracker-software.com

johnr@etakeoff.com
User
Posts: 57
Joined: Thu Jan 01, 1970 12:00 am

Re: Multi-thread access to the Simple Viewer DLL

Post by johnr@etakeoff.com » Wed Apr 27, 2011 10:15 pm

Thanks for all the information. I'd always wondered why your viewer was blindingly fast when drawing to the screen but a dog when drawing to a large bitmap. Now I know.

Yes, I know that a TIFF file is a bitmap. I'm not an idiot. What I didn't know was that you draw to an internal bitmap then draw that bitmap to the DC (containing MY bitmap.) I use the same technique to avoid flicker.

I'll implement the banding next week and let you know. Hopefully it will solve a number of problems for us.

Thanks for the great support.

User avatar
Tracker Supp-Stefan
Site Admin
Posts: 13931
Joined: Mon Jan 12, 2009 8:07 am
Location: London
Contact:

Re: Multi-thread access to the Simple Viewer DLL

Post by Tracker Supp-Stefan » Thu Apr 28, 2011 9:38 am

Sure John,

We would love to hear how it goes for you!

And as for the support - all the kudos are going to Victor on this one ;)

Cheers,
Stefan

johnr@etakeoff.com
User
Posts: 57
Joined: Thu Jan 01, 1970 12:00 am

Re: Multi-thread access to the Simple Viewer DLL

Post by johnr@etakeoff.com » Tue May 03, 2011 9:50 pm

Thanks to all.

I already had banding logic so it only took a few minutes to make sure no band was more than 4M pixels (128MB). The result was that what previously took 28 seconds to generate now takes less than 5 seconds.

I greatly appreciate your assistance.

User avatar
Paul - Tracker Supp
Site Admin
Posts: 4990
Joined: Wed Mar 25, 2009 10:37 pm
Location: Chemainus, Canada
Contact:

Re: Multi-thread access to the Simple Viewer DLL

Post by Paul - Tracker Supp » Tue May 03, 2011 9:53 pm

:D
_________________
If posting files to this forum - you must archive the files to a ZIP, RAR or 7z file or they will not be uploaded - thank you.

Best regards

Paul O'Rorke
Tracker Support North America
http://www.tracker-software.com

Post Reply