Page 1 of 1

Limitations with PXC_AddPage and PXC_WriteDocumentExA

Posted: Wed Jun 21, 2006 11:23 am
by juergen

Until reciently we used our application to create multipage tiff files from our customers' print spool text files. The magic file size of 4GB (ca. 100000-500000 Pages with G4 compression) had not yet been reached.
Since PDF is the better format for files of that size, we decided to use PDF-Toolkit to generate PDF files.

Unfortunatly, it turned out that your toolkit can't create PDF files larger than 15624 pages. The tookit generates a "not enough memory" error as soon as one tries to add page 16128 with "PXC_AddPage".

With "PXC_AddPage", this error always turns up at the magic number 16128.
When trying to save a PDF with "PXC_WriteDocumentExA", the failure arises earlier, at page 15624.

This cannot be caused by the RAM. The computer is equipped with 4GB of RAM, of which 2GB were available when the errors occured.

The errors occured while trying to create documents of about 21000 pages. The toolkit version is 3.40.079.

We do, however, have exactly the same error with version 3.60.102.
Tests run on Windows 2000/XP/2003-Server all deliver the same result.

You can reproduce this error in your sample application "PDF-XChange PRO 3 SDK\Examples\SDKExamples\CExamples\PXCSample\ex_text.cpp", by adding a large enough for loop to create a document with "PXC_AddPage".

If required I could supply you with the modified source file.

Of course, you might ask who in the world needs PDF files with 100,000 - 500,000 pages?
The answer of course is: customers - they require it for their data volume and insist on having it.

Can you please check this and let us know whether you have a solution and could supply a hotfix?

Thanks in advance for your kind assistance

Posted: Wed Jun 21, 2006 11:56 am
by John - Tracker Supp

to allow us to test and as closely as possible replicate the type of file with 'typical' sample content - can you provide a few image files for us to work with and we will use these in our tests - as this could have some bearing on the results.

Are they typically 1 TIFF file per page etc ?


Posted: Wed Jun 21, 2006 1:24 pm
by John - Tracker Supp

Having looked at this it is necessary to explain a little of the way the library works and how resources are used - and how best to overcome this issue.

Firstly - even though you have 4GB of memory - a 32 bit APP cannot fully use 4GB, only 2GB can be used - even though 4GB can be addressed - even then though - 2GB of address space is reserved by 32 bit windows for internal use.

Of the 2GB available for use by your application - this must contain your application code and data, so in reality for data your 32-bit application can use up to almost 2Gb of address space (it matters not if this is physical or virtual memory from the swap file).

Our libraries will retain in memory large elements of your file as it is being written - this allows us to optimize the file when written to disk far more efficiently - the drawback being memory is consumed by so doing.

There are 2 workarounds which will extend the number of pages you can write with the file.

Firstly set an arbitrary limit to the pages being processed and then write the file to disk - then process the next batch of pages and append to the original file.

This will extend the limit somewhat - but most likely will fail well before reaching you desired total of 100,000 pages+, hard to say in your precise circumstances when this will be.

Or the best alternative which will allow you to achieve the desired limits and beyond -:)

Compile you application as a 64-bit app and use our 64 bit libraries (64-bit Windows supports up to 1 terabyte of physical memory with 8 terabytes of address space for each process).

You would need to then run on this on a 64 bit Windows enabled PC of course.

Hope that helps

Posted: Wed Jun 21, 2006 3:52 pm
by John - Tracker Supp

Thanks for the file - but this does not change I am afraid the advice given in my post above.

It is possible in V4 we can tune the memory usage to some degree - how much help that will be in such a situation it is hard to foretel - but I doubt it would be sufficient to overcome the situation you describe - for now the 2 work arounds suggested above are the best we can offer.

Posted: Wed Jun 21, 2006 4:04 pm
by juergen
Tracker Support wrote:Hi,

to allow us to test and as closely as possible replicate the type of file with 'typical' sample content - can you provide a few image files for us to work with and we will use these in our tests - as this could have some bearing on the results.

Are they typically 1 TIFF file per page etc ?


We don't add/insert TIFF-images to the pdf-files. We create PDF files from text files instead of creating TIFF files from text files. The PDF files are generated in the same way as in your sample application under the point "Text and Fonts". Please try ist with the modified attached source file from your sample application. And you can see what's going wrong.

Posted: Wed Jun 21, 2006 4:32 pm
by John - Tracker Supp

Whilst the content variation may extend the limits somewhat - it does not alter the fundamental fact that a limit will be reached at some point.

There are some addiitonal steps you can take to 'eek' out the maximum tolerances from the library/32 bit Windows - but this will not overcome entirely the limitations regarding memory usage :

Increase the Compression algorithm if not already set to 'Zip' with maximum compression and call the PXC_EndPage function for each page for which content creation is finished.

This should extend the limits - but to what I cannot give a specific answer - but the core advice, if possible - is to migrate the application to 64-bit Windows for such tasks.