Fixing "out of memory" in VS2005

Main development forum.

Fixing "out of memory" in VS2005

Postby sorcerer6000 » Sun Dec 05, 2010 8:31 pm

I'm comparing two large text files (~300 MB), and I get "out of memory". So I'd like to investigate why this is happening, and possibly fix it.

I downloaded the sources for 2.12.4 and compiled using VS2005, which converted the solution file (WinMerge.sln) successfully.

The build fails for several reasons:
1) 1>Performing Pre-Link Event...
1>* Configure
1>c:\Program Files\WinMerge\WinMerge-2.12.4-src\Src
1>Could Not Find c:\program
This is probably because there's a blank in the path, and I can fix that.

2) 1>LINK : fatal error LNK1181: cannot open input file 'libexpat.lib'
This library was not included in the source distribution, although libexpat.def is.

In addition, there are many warnings from the compiler, most of which can be easily fixed (e g, signed/unsigned mismatch).

So, my question is, is it useful for me to fix any or all of these issues, and, if so, where can I get libexpat.lib? And which set of sources should I be working on?

VS2005 is the only compiler I have, so I would produce a .sln in that format.
sorcerer6000
 
Posts: 2
Joined: Sun Dec 05, 2010 8:20 pm

Re: Fixing "out of memory" in VS2005

Postby kimmov » Sun Dec 05, 2010 8:59 pm

Have you read the Docs/Developers/Compiling.html-file in source distribution? That should help with compiling.

The problem is probably because you only converted the main project/solution file, not ones in Externals-folder. Yes, this cumbersome that not all projects are in that same solution file, historical reasons.. :( If you have Python installed you can run Tools/Scripts/UpgradeProjects.py to update all the solution/project files.

I'm afraid there is not much you can do for this problem. The problem is in architecture of WinMerge and how we load and compare files. Probably we can really solve that in WinMerge 3 if that happens someday...
kimmov
 
Posts: 562
Joined: Thu Sep 11, 2008 8:51 pm
Location: Finland

Re: Fixing "out of memory" in VS2005

Postby sorcerer6000 » Mon Dec 06, 2010 2:49 am

I have solved the build problems, and built a debug version to figure out what is going on.

The main problems with the large files are (1) You are mapping a view of the entire file at once, which eventually will use all the RAM; (2) you are not allowing for files >2^31 bytes. The first can be remedied by processing the file in sections of, say 1MB; the second, by using more up-to-date file handling routines. Of course, I am not familiar with the complete WinMerge architecture, so it may not be as simple as I think to fix these issues. Perhaps you can comment on them.

BTW, one problem with the build was the command given to the PreLink.bat. When a path $(TargetPath) can contain spaces, it must be enclosed in quotes on the command line. So I changed

PreLink.bat $(IntDir) $(TargetPath)

to

PreLink.bat $(IntDir) "$(TargetPath)"

It is also possible for $(IntDir) to contain blanks, but since the batch file uses this string as a label in the batch file (bad practice!), it cannot just be enclosed in quotes; they are removed by a file path parser, but not by the batch interpreter.
sorcerer6000
 
Posts: 2
Joined: Sun Dec 05, 2010 8:20 pm

Re: Fixing "out of memory" in VS2005

Postby kimmov » Mon Dec 06, 2010 9:38 am

sorcerer6000 wrote:(1) You are mapping a view of the entire file at once, which eventually will use all the RAM;

Yes, this is the main problem. And it is tricky problem to solve if you look further and see how we interact with diffutils as diffengine. Diffutils understands only whole files or streams. Now, it would be possible to give parts of file, compare them and then combine the results. But since diffutils already "optimizes" diffs it creates this can cause different results than comparing whole files/streams. And you also need to handle cases where the split happens in middle of diff etc. Yes, it is doable. But very hard to do without breaking the results since we have no good test set. I wouldn't modify that code without tests. And that again is a bit waste of time for the old diffutils code...

sorcerer6000 wrote:(2) you are not allowing for files >2^31 bytes. The first can be remedied by processing the file in sections of, say 1MB; the second, by using more up-to-date file handling routines.

That is not entirely true. We handle bigger files in folder compare when not using diffutils ("full contents compare"). And indeed switch automatically to use byte-per-byte compare for large files. But diffutils compare is the problem.

sorcerer6000 wrote:BTW, one problem with the build was the command given to the PreLink.bat. When a path $(TargetPath) can contain spaces, it must be enclosed in quotes on the command line. So I changed

PreLink.bat $(IntDir) $(TargetPath)

to

PreLink.bat $(IntDir) "$(TargetPath)"

It is also possible for $(IntDir) to contain blanks, but since the batch file uses this string as a label in the batch file (bad practice!), it cannot just be enclosed in quotes; they are removed by a file path parser, but not by the batch interpreter.

Yep. I've never build (any) software (in/to) paths containing spaces... And those batch files were just trick we get something sane for VC6 and its "projects" and "workspaces". I guess modern solution files would work without these tricks but nobody has bothered trying it...

About the compare thing.. My idea for WinMerge 3 is to revisit whole thing (doing lots of things from scratch). Different compare engine and way to handle files etc. For WinMerge 2 there is just too much cruft piled up over the years that trying to fix it seems to be desperate task. But unfortunately my time for WinMerge development is very limited and so both the old and new versions are quite much in limbo state now...
kimmov
 
Posts: 562
Joined: Thu Sep 11, 2008 8:51 pm
Location: Finland


Return to Developers

Who is online

Users browsing this forum: No registered users and 2 guests

cron