Missing text / fonts / images in PXC_WriteDocument files

This Forum is for the use of Software Developers requiring help and assistance for Tracker Software's PDF-Tools SDK of Library DLL functions(only) - Please use the PDF-XChange Drivers API SDK Forum for assistance with all PDF Print Driver related topics or PDF-XChange Viewer SDK if appropriate.

Moderators: TrackerSupp-Daniel, Tracker Support, Vasyl-Tracker Dev Team, Chris - Tracker Supp, Sean - Tracker, Tracker Supp-Stefan

Post Reply
omascia
User
Posts: 48
Joined: Thu Mar 11, 2010 7:07 pm

Missing text / fonts / images in PXC_WriteDocument files

Post by omascia »

We experience a strange problem on Server 2008 R2 since this monday.
There have been a bunch of Windows Updates on sunday 17 Aug and since monday, after *some* time our binaries start producing PDF files where either only the images are missing or images and all text. Lines and boxes are the only visual elements that seem to persist.
The code producing the PDFs had been untouched for months and the binaries had not even been upgraded in 2 months when this started to occur on monday.
Restarting the binaries solve the issue for *some* time (can be hundreds of PDF produced without issue), then all of a sudden this strange behavior starts happening and persists (until restart).
All this had been flawless for countless documents for more than a year and we're puzzled as to what might trigger this awful and strange behavior.
What is especially puzzling is that we apparently detect no failure of any PXC_ API during building the PDF, including the PXC_WriteDocument.
It just happen that from some point in time, all those built PDFs are (apparently) missing text and images (sometimes only images). By missing I'm only meaning that viewing the PDF with any PDF reader we see no text and no images in it. If there are lines (PXC_MoveTo / PXC_LineTo / PXC_StrokePath), they seem to be the only visual elements remaining in the viewed PDF. The Fonts property page shown by Adobe Reader lists nothing : no fonts. This is very strange. I wonder if the text themselves are still present in compressed streams within the file and only fonts are missing, or not. I have some samples of these files to send you if you gave me instructions where to send them for analysis.
Having considered a lot of things and our code which didn't change for long, it is tempting to relate this behavior with the bunch of Windows updates which installed on sunday.
But what subtle details of the Windows behavior might have changed to trigger this kind of not-easily reproducible issue when building PDF files (as it does not occur systematically until some point in time), is really puzzling. The most logical thing would be some logic error in our code, and we're reviewing it since monday, but as it did not change for so long and the problem started to occur even though binaries had not been updated, it is difficult to pursue this path. We hope that anything you might discover in the problematic PDF files produced might give a clue to what to look for next.
Thanks for your help.
omascia
User
Posts: 48
Joined: Thu Mar 11, 2010 7:07 pm

Re: Missing text / fonts / images in PXC_WriteDocument files

Post by omascia »

I can simulate a similar behavior by doing the following in our ordinary listing engine.
This one PXC_AddFont(..."Courier New"...), which I comment out.
We normally AddFont "Courier New" 400 and 700, non-italic.

Then, assuming we do not test error returns of most other APIs, the whole printing process goes on without crash, and the resulting PDF has no text content and no font (which is expected in such a test).
So... We have to assume that a binary, building a lot of such PDFs along the day, all of a sudden start to suffer from some Windows denial to load a font. An issue which fixes itself by a restart of the process, until it re-appears at some point in time later in the day.

We have the suspicion it might be related to bad recent Microsoft fixes to some security breaches. They have temporarily withdrawn an update (KB2982791) after it has been discovered bad side-effects can happen *when* KB2976897 is also present.

The key thing here being that we just received the KB2976897 this sunday (our software mostly unused) and the issue appeared the first time on monday morning.

We will see to uninstall the KB2976897 overnight because we suspect those issues they have with the next update (KB2982791) might very well be build on some other error introduced by KB2976897.

To tell the truth, we are short of ideas about this strange issue and want to believe this is the cause of it.

It is by the way our understanding that if during the run of one process we build hundreds of PDFs, there is no housekeeping to do in regards with these fonts other than properly PXC_ReleaseDocument() after we have called PXC_WriteDocument() to save the resulting output. And that each thread building one PDF does it through its own PXC_NewDocument(), ..., PXC_ReleaseDocument(), there are no threading / contention issues to consider. Right? (Well it at least looked fine for more than 14 months of the same implementation running unchanged until this issue started to hit us this week.)
User avatar
Ivan - Tracker Software
Site Admin
Posts: 3549
Joined: Thu Jul 08, 2004 10:36 pm
Location: Vancouver Island - Canada
Contact:

Re: Missing text / fonts / images in PXC_WriteDocument files

Post by Ivan - Tracker Software »

Can you send us (you can use ivan@tracker-software.com or support@pdf-xchange.com) some of files that have problems?
Tracker Software (Project Director)

When attaching files to any message - please ensure they are archived and posted as a .ZIP, .RAR or .7z format - or they will not be posted - thanks.
omascia
User
Posts: 48
Joined: Thu Mar 11, 2010 7:07 pm

Re: Missing text / fonts / images in PXC_WriteDocument files

Post by omascia »

Thanks! Done. With some added information.
omascia
User
Posts: 48
Joined: Thu Mar 11, 2010 7:07 pm

Re: Missing text / fonts / images in PXC_WriteDocument files

Post by omascia »

We uninstalled the KB2976897 and... we have had only one case since 08:00 today (it is 15:45 here at time of posting). This case we had minutes ago is the limited one (where only the images are missing). And as we added some trace code yesterday evening, we can show the decoded error code returned by PXC_AddImage : "Error [Operating system]: The system cannot find the file specified."

Checked again : those files "not found" (which happen to be .jpg files) are indeed present and can be opened live in any image viewer, and were found many times during the previous hours by the same lines of code in the same running binary. Merely stopping and starting the process fixes it. It looks like a resource exhaustion issue where the OS wouldn't be able to open any new file. But that is not as simple and is located within PXC_AddImage because we have some other files which are opened right before and right after the call to PXC_AddImage. It's not that the whole process starts going mad when this happens : everything works flawlessly (the binary produces PDFs but that is not its purpose, just a byproduct) and if it weren't for missing images and eventually texts, those PDF are fine as well as every other result (database updates, external files reading and son on) is perfectly fine. Really really puzzling.

I still have to detect today one of those cases (seen no later than yesterday afternoon) where the whole text is missing. So maybe the widows update removal helped regarding PXC_AddFont (also accessing file resources), but not PXC_AddImage (maybe these are two similar issues, whose origin is different).

Tonight we will go further and uninstall all the 6 or 7 windows updates which got installed last sunday : before that day we never had such issues.
omascia
User
Posts: 48
Joined: Thu Mar 11, 2010 7:07 pm

Re: Missing text / fonts / images in PXC_WriteDocument files

Post by omascia »

Just got informed of one of those cases with missing texts and fonts and indeed, as I expected after seeing PXC_AddImage returning an unexplainable 'not found', the PXC_AddFont calls start returning : PXC_AddFont('Courier New'): Error [PdfLib]: Invalid font.
And this for font names as "Arial" or "Courier New" which where successfully used tens of times over the previous hours.

I forgot to mention this is a 64 bits process, so using the 64 bits DLLs of PDF-Tools. The DLLs are 4.0.309.0 (pxclib40.dll for instance).
omascia
User
Posts: 48
Joined: Thu Mar 11, 2010 7:07 pm

Re: Missing text / fonts / images in PXC_WriteDocument files

Post by omascia »

We're still fighting this issue and having a hell of a time with every of our customers running Windows Server 2008 R2, fully updated (since mid-august). By fully updated, I mean in the state where Windows Update reports nothing to install : every service pack, important or recommend update is installed. After some time the processes (running non stop) which have to sometimes build a PDF (all our printing architecture is PDF based - first producing a PDF then printing it (which allows for an archival architecture)) start getting "Invalid font format" and "Invalid image format" from PXC_AddFont and PXC_AddImage.

Regarding PXC_AddFont with have decided 24h ago to code a temptative workaround. As we mainly use Arial and Consolas, we now temporarily use PXC_AddFontFromFile referring directly to the adequate TTF file, instead of relying on PXC_AddFont to do the so-called font matching. At time of this writing we have not yet had returns from field that the problem re-appeared. But it sometimes takes more than 4 or 5 hours of work during the day before it triggers, so we cannot conclude anything for now.

Regarding pictures inserted in the PDF, we only insert jpeg files. Here again we have tried to workaround by loading the jpeg files in memory (using another imaging library) and then using PXC_AddImageFromHBITMAP instead of PXC_AddImage. Here we have new real results to share. When the issue start occurring (apparently much more quickly than the font issue), we now get a black rectangle in the PDF instead of the jpeg, while not getting any error code from PXC_AddImageFromHBITMAP. Again, for a same binary running, a same jpeg file, it works fine, then after some time it starts to output those black rectangles.

We're today coding a sample test binary which we could share the source code to repetitively create simple PDF the very way we build them in the released application. We are confident that letting this run for some time on the same Windows Server 2008 R2 setup should trigger the same issue. If it does we intend to retry with fully debug build and start reverse engineering your DLL interactions with windows and files when either adding an image or a font to a PDF file. We suspect some specific GDI actions are triggering this issue. Not saying there is a bug in your code : just saying there is something common to your code adding fonts and adding images which starts failing in the context of the current OS state.

We have images of last mi-july of Server 2008 R2 where this does not happen. So we can really workaround the issue for customers where we have control on their hardware (or those running hosted on ours), but we're still stuck for the other using their own resources and not trained to revert their Windows Server to some point back in time.

This is the very first time we encounter a major issue with PDF-Tools SDK, it has always been flawless for us, except maybe some documentation errors or imprecisions leading sometimes to questions before finding the right way to do things (review your explanations regarding PXC_CloseImage and PXC_PlaceImage for instance and your comments about calling multiple times PXC_AddFont for a same font while building a same PDF file). Though we understand these are side-effects from a Microsoft bug or behavior change, you may help us by reviewing your code, especially portions which might have similarities in the context of adding fonts or images.
omascia
User
Posts: 48
Joined: Thu Mar 11, 2010 7:07 pm

Re: Missing text / fonts / images in PXC_WriteDocument files

Post by omascia »

I'll post a conclusion here within the next 24h (when we will have enough returns from field) but it looks like we have suffered from a very inextricable cross libraries issue, which only started after receiving some Windows update fixes, and that PDF-Tools (and we're so happy if this is so) was NOT the trigger in itself (as much as it looked like), but in only in combination with some other tools/libraries. If the outcome within the next 24h stays as it was for the last 6h, then we'll be *very* happy this was isolated and properly worked around. :)
User avatar
Tracker Supp-Stefan
Site Admin
Posts: 17823
Joined: Mon Jan 12, 2009 8:07 am
Location: London
Contact:

Re: Missing text / fonts / images in PXC_WriteDocument files

Post by Tracker Supp-Stefan »

Hello omascia,

Glad the fonts workaround is working for you!

I've asked one of our Tools SDK devs to take a look at this topic and his comments are as follows:
All I can guess with images is that this other library also has problems with reading images and creates empty bitmaps (filled all components with zero - black color). But PXC_AddImageFromHBITMAP receives that bitmap and converts it successfully (therefore no error).
And i have no idea what may cause such behaviour.

Looks like the application has problems accessing files (maybe too many handles open or too much system memory is used, or something else).
Regards,
Stefan
omascia
User
Posts: 48
Joined: Thu Mar 11, 2010 7:07 pm

Re: Missing text / fonts / images in PXC_WriteDocument files

Post by omascia »

What we were experiencing, now that we have some background and a solution is this. Using GDI API which are NOT meant to deal with actual windows painting is possible from a NON-GUI (command-line type) executable, without even a Windows messages loop (typically a blind service application). This has been flawless for us for years. Until a sad bunch of security fixes deployed by MS mid-august. They even have had to withdraw one of those updates soon after release due to reports of crashes, and other weird things, including the need to delete FNTCACHE file and fixup some registry keys. We got the first batch of these updates but not the one which was withdrawn quickly. This is on Server 2008 R2, fully patched.

What we were experiencing in our process turned out to be really unexpected errors returned by GDI calls after some hundreds or thousands of them. Checking the GDI and USER handles count at key moments and including at time of those errors turned up nothing unusual : rarely more than 30 to 40 GDI objects in use at any time (out of ~10000 allowable by process). We had no memory shortage issues, no file handles issues of any sort : these processes do a lot of work and building some PDF (with images) account for a very low percentage of that : the only thing which was starting to go wrong after some time was the PDF building process. Everything else was running smoothly.

Both PDF-Tools and our old imaging library were impacted by these errors. The imaging library sometimes failed to build a correct image and the PDF-Tools sometimes failed to insert (an otherwise perfectly correct) image file to a pdf. Both these actions, as they were programmed involved GDI calls by these libraries. The remaining 99% parts of our process were totally unaffected because there is absolutely no GDI calls. At least this is the picture as we see it now after "the dust settled". We also had issues with adding fonts to the pdf through PXC_AddFont, and again that stopped occurring as soon as we worked around by using PXC_AddFontFromFile, which I presume has fewer if any dependency on GDI APIs.

Yesterday we had excellent success by :

1) no more using the imaging library to convert the images our users give as input to a cleaner JPEG file with quality set at 70% (our only use of that library). We use GDI+ to read the files and output a jpg 70%. It helped a lot. The only thing which was still failing on us after some time was PXC_AddImageFromHBITMAP, sometimes leading to a black box instead of the actual image, and sometimes leading to error core returned ("gdi error") - hence no image in the PDF.

2) no more using PXC_AddImage or PXC_AddImageFromHBITMAP, but passing pixels straight using PXC_AddImageFromMemory().

In addition, we received yesterday new updates from MS : they re-instantiated the fixes that were held off. On a separate system which was affected by the same conditions, we reverted all the code changes to the point it was before this issue started (so with none of our workarounds to lesser the GDI calls). And all is now well too on that test system, though some more hours of running might be better before drawing a conclusion.

These were 11 very interesting days. :)
User avatar
Tracker Supp-Stefan
Site Admin
Posts: 17823
Joined: Mon Jan 12, 2009 8:07 am
Location: London
Contact:

Re: Missing text / fonts / images in PXC_WriteDocument files

Post by Tracker Supp-Stefan »

Hello omascia,

Glad to hear that both your workaround and the MS fix for their patches are now working!

Cheers,
Stefan
Post Reply