InsertPagesFromDoc Issue

PDF-XChange Editor SDK for Developers

Moderators: TrackerSupp-Daniel, Tracker Support, Paul - Tracker Supp, Vasyl-Tracker Dev Team, Chris - Tracker Supp, Sean - Tracker, Ivan - Tracker Software, Tracker Supp-Stefan

Forum rules
DO NOT post your license/serial key, or your activation code - these forums, and all posts within, are public and we will be forced to immediately deactivate your license.

When experiencing some errors, use the IAUX_Inst::FormatHRESULT method to see their description and include it in your post along with the error code.
Post Reply
jeffp
User
Posts: 914
Joined: Wed Sep 30, 2009 6:53 pm

InsertPagesFromDoc Issue

Post by jeffp »

I'm having an issue with inserting lots of files/pages into a document. I have one example where I'm inserting over 1,500 one page PDF files into a new PDF using InsertPagesFromDoc. It seems the inserting so many pages is causing the memory to climb so high that it crashes my program.

Is there a call I can make after every 200 files or so that will compact or reduce the memory usage? I found a work around if I close the file and reopen it, and proceed with inserting more files. I do this every 200 files or do and it seems to get rid of the bloat in memory.

Sometimes it's not convenience to save and reopen so I was wondering if there was another call I can make while the file is still open and receiving inserted pages that would do the same thing as a save, close and reopen does?

Thanks.

--Jeff
Sasha - Tracker Dev Team
User
Posts: 5522
Joined: Fri Nov 21, 2014 8:27 am
Contact:

Re: InsertPagesFromDoc Issue

Post by Sasha - Tracker Dev Team »

Hello jeffp,

Well if you want precise memory control then the best language for it would be C++.
In your case there a two thing that you can do:
1) Divide the insertion process into portions (for example 200 pages max).
2) Move to x64 and solve all of your problems.

Cheers,
Alex
Subscribe at:
https://www.youtube.com/channel/UC-TwAMNi1haxJ1FX3LvB4CQ
jeffp
User
Posts: 914
Joined: Wed Sep 30, 2009 6:53 pm

Re: InsertPagesFromDoc Issue

Post by jeffp »

Again, as a workaround, I can save out to temp after combining so many files, but when combining one file with say 2000 pages, we often get the memory crash as well.

The funny thing about this is it wasn't happening just a few builds ago. Seems to be something that has crept in just recently.

We'll try it again on the next build.

--Jeff
Sasha - Tracker Dev Team
User
Posts: 5522
Joined: Fri Nov 21, 2014 8:27 am
Contact:

Re: InsertPagesFromDoc Issue

Post by Sasha - Tracker Dev Team »

Hello jeffp,

The V9 should be out soon - maybe it will fix the problem you mentioned.

Cheers,
Alex
Subscribe at:
https://www.youtube.com/channel/UC-TwAMNi1haxJ1FX3LvB4CQ
jeffp
User
Posts: 914
Joined: Wed Sep 30, 2009 6:53 pm

Re: InsertPagesFromDoc Issue

Post by jeffp »

When do you expect V9 and what are the major changes going to be?
Sasha - Tracker Dev Team
User
Posts: 5522
Joined: Fri Nov 21, 2014 8:27 am
Contact:

Re: InsertPagesFromDoc Issue

Post by Sasha - Tracker Dev Team »

Hello jeffp,

The release should be out in a month or so, if everything is OK.

Cheers,
Alex
Subscribe at:
https://www.youtube.com/channel/UC-TwAMNi1haxJ1FX3LvB4CQ
jeffp
User
Posts: 914
Joined: Wed Sep 30, 2009 6:53 pm

Re: InsertPagesFromDoc Issue

Post by jeffp »

Has there been any similar issues like this reported? Has any of fixes for V9 addressed this specifically?

There seems to be some large memory bloats now when doing certain pdf actions with the SDK. Again, you have to save, close, and re-open to knock out the memory.

--Jeff
Sasha - Tracker Dev Team
User
Posts: 5522
Joined: Fri Nov 21, 2014 8:27 am
Contact:

Re: InsertPagesFromDoc Issue

Post by Sasha - Tracker Dev Team »

Hello jeffp,

Again, using the RAM speeds up the overall execution process drastically. Like I said before, if you would use the C++ like language that allows managing the memory then the Garbage Collector would be in your hands and would not store the tons of garbage objects. You can see what you can done in your language to free the used memory manually. Like using Dispose() for the given object or something like that. If you had been using the x64 version - the memory problems (before the garbage collector decides to free the memory) would not be as critical as in the x86 versions.

Cheers,
Alex
Subscribe at:
https://www.youtube.com/channel/UC-TwAMNi1haxJ1FX3LvB4CQ
jeffp
User
Posts: 914
Joined: Wed Sep 30, 2009 6:53 pm

Re: InsertPagesFromDoc Issue

Post by jeffp »

It's obviously not feasible to switch programming languages for a product that is over 10 years old. So I'm trying to deal with it the best I can in Delphi. Again, the interesting thing here is that this wasn't happening several builds ago.

I feel I'm doing all I can to free the objects and interfaces used in my code. I can't do anything about what happen in the API.

Take a look at the following Delphi code. I'm just looking through your InsertPagesFromDoc procedure. I'm not sure what there is to free on my end that will stop the bloat from looping through this call.

if (ASrcPagesAsCommaText <> '') then
begin
with TStringList.Create do
try
CommaText := ASrcPagesAsCommaText;
for i := 0 to Count - 1 do
begin
iPage := StrToIntDef(Strings, 0);
if (iPage > 0) then
begin
FDoc.Pages.InsertPagesFromDoc(ASrc.Doc, ABeforeIndex + i, iPage - 1, 1, AFlags, nil);
Inc(AInsertedCount);
end;

end;
finally
Free;
end;
end;
Sasha - Tracker Dev Team
User
Posts: 5522
Joined: Fri Nov 21, 2014 8:27 am
Contact:

Re: InsertPagesFromDoc Issue

Post by Sasha - Tracker Dev Team »

Hello jeffp,

We will need to recreate this behavior from our part - I'll try creating the C# sample for this one and see whether I experience this behavior. Will any large sample file work? Like 1.7 PDF Specification or something (700+ pages).

Cheers,
Alex
Subscribe at:
https://www.youtube.com/channel/UC-TwAMNi1haxJ1FX3LvB4CQ
jeffp
User
Posts: 914
Joined: Wed Sep 30, 2009 6:53 pm

Re: InsertPagesFromDoc Issue

Post by jeffp »

Here's the test file I'm working with.

http://files.filecenterdms.com/TestFile.zip

Tell me when you have it and I'll pull it down.

--Jeff
Sasha - Tracker Dev Team
User
Posts: 5522
Joined: Fri Nov 21, 2014 8:27 am
Contact:

Re: InsertPagesFromDoc Issue

Post by Sasha - Tracker Dev Team »

Hello jeffp,

Got it, thanks.

Cheers,
Alex
Subscribe at:
https://www.youtube.com/channel/UC-TwAMNi1haxJ1FX3LvB4CQ
Sasha - Tracker Dev Team
User
Posts: 5522
Joined: Fri Nov 21, 2014 8:27 am
Contact:

Re: InsertPagesFromDoc Issue

Post by Sasha - Tracker Dev Team »

Hello jeffp,

Tested this on the C# application. Here's what I got (after I've inserted all 4000 pages):
image.png
I've used the regsvr32 on our latest release of 341 version.
I'll attach the project that you can try for yourself:
TestApp.zip
(2.28 KiB) Downloaded 87 times
Cheers,
Alex
Subscribe at:
https://www.youtube.com/channel/UC-TwAMNi1haxJ1FX3LvB4CQ
jeffp
User
Posts: 914
Joined: Wed Sep 30, 2009 6:53 pm

Re: InsertPagesFromDoc Issue

Post by jeffp »

Again, I can't switch to C# but was just hoping you may have found some places where you can manage/free memory better internal to the SDK.

--Jeff
User avatar
Lzcat - Tracker Supp
Site Admin
Posts: 677
Joined: Thu Jun 28, 2007 8:42 am

Re: InsertPagesFromDoc Issue

Post by Lzcat - Tracker Supp »

Hi, jeffp

Unfortunately we cannot do your job for you. We do not know which component(s) use too much memory in YOUR product. Even if it looks that is is our SDK this may be not true. For example there is common problem with all .NET languages, where is no strict control on objects lifetime. In some cases this not big problem, because objects doe not use much memory, but if that objects holds references for complex objects (like our SDK document) this will cause big problems, especially in 32-bit application. This happens because all objects are not immediately released in .NET programs, but just marked as unused and keep alive until garbage collector will release them (which happens not so frequently). So for example if you are opening same document in 100 times loop this may cause that 100 copies of document remains in memory even if you hoping that you are not holding them. Unfortunately tin such languages there is no methods to force releasing COM objects, so precise control is not possible :( . Sometimes it is require to use hidden/undocumented functions to force garbage collector to free unused objects, but even this is may be not enough because many components does not free used memory completely, but keep it reserved for them (this may cause that 32-bit process address space is fully reserved, even when program is using only several MB of RAM). So in most cases it is require to switch to 64-bit code, even for non-.NET programs it you need to handle large documents or a lot of small simultaneously.

What you should to is to monitor your program memory usage after each processing step and check where too much memory is used. For example if you find that after opening document, inserting pages from it to other document and closing it memory usage is constantly grow (which is normal, by the way :D), you should leave just open and close document code. After this modification memory should not grow constantly (it may grow a bit for several
first iterations, and than keep approximately on same level). If this is true, problem is somewhere in insertion code (your or our side, maybe both, this will require future investigation). But if memory usage remains to grow constantly - this means that you simply do not release opened documents. This may happen because you keep documents objects in array of processed documents (for example), or because garbage collector does not release them, or because someone added not needed reference to it (in C++ code). And we can do nothing there except advise you to find and fix problem on your side.

And I want to remind two things:
1. Our SDK is COM based, so there is no direct methods to free any of objects, instead you should make sure that none of objects are keep referenced after they nor needed.
2. In many programing languages which use garbage collection objects are not released immediately after exiting out of scope, so you should find how to release them (this all .NET languages, JAVA, an some others).

Kind regards,
Lzcat - Tracker Supp
Victor
Tracker Software
Project manager

Please archive any files posted to a ZIP, 7z or RAR file or they will be removed and not posted.
jeffp
User
Posts: 914
Joined: Wed Sep 30, 2009 6:53 pm

Re: InsertPagesFromDoc Issue

Post by jeffp »

Was playing around with another PDF component written in C# and they have a call in their SDK called

ForceGarbageCollection

May I suggest adding this at some point to your SDK. Thanks.

--Jeff
User avatar
Lzcat - Tracker Supp
Site Admin
Posts: 677
Joined: Thu Jun 28, 2007 8:42 am

Re: InsertPagesFromDoc Issue

Post by Lzcat - Tracker Supp »

Hi, jeffp

Looks like you misunderstood. Our SDK is written using C++, not C#. We cannot call ForceGarbageCollection because it is language-specific function, and it must be called by application which use our SDK (if needed).
And as I told before it may not help if your code keep array of processed documents (for example). So you should make some testing with your code, find on which step memory is not released properly and try to find workaround (call ForceGarbageCollection or whatever needed, depending on your programing language). And only if this not help - you should make some small example illustrating problem and provide it to us for investigation (we will need working application with all required components except our SDK, source code of example, testing file set and small instruction how to use it if needed).

Kind regards,
Lzcat - Tracker Supp
Victor
Tracker Software
Project manager

Please archive any files posted to a ZIP, 7z or RAR file or they will be removed and not posted.
jeffp
User
Posts: 914
Joined: Wed Sep 30, 2009 6:53 pm

Re: InsertPagesFromDoc Issue

Post by jeffp »

My bad. Sorry.
User avatar
Tracker Supp-Stefan
Site Admin
Posts: 17812
Joined: Mon Jan 12, 2009 8:07 am
Location: London
Contact:

InsertPagesFromDoc Issue

Post by Tracker Supp-Stefan »

:)
Post Reply