This is the mail archive of the cygwin-developers mailing list for the Cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Performance optimization in av::fixup - use buffered IO, not mapped file


On 12/12/2012 10:39 AM, Ryan Johnson wrote:
On 12/12/2012 9:04 AM, Eric Blake wrote:
On 12/12/2012 06:22 AM, Corinna Vinschen wrote:
On Dec 12 06:11, Eric Blake wrote:
On 12/11/2012 08:13 PM, Daniel Colascione wrote:
Considering the horrible and
unexpected performance implications of sparse files, I don't think generating
them automatically from a sequence of seeks and writes is the right thing to do.
Why can't we instead use posix_fallocate() as a means of identifying a
file that must not be sparse, and then just patch the compiler to use
posix_fallocate() to never generate a sparse executable (but let all
other sparse files continue to behave as normal)?


posix_fallocate is not allowed to generate sparse files, due to the following restriction: "If posix_fallocate() returns successfully, subsequent writes to the specified file data shall not fail due to the lack of free space on the file system storage media." See http://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_fallocate.html Therefore only ftruncate and lseek potentially generate sparse files. On second thought, I don't quite understand what you mean by "use posix_fallocate() as a means of identifying a file that must not be sparse". Can you explain, please?
Since we know that an executable must NOT be sparse in order to make it
more efficient with the Windows loader, then gcc should use
posix_fallocate() to guarantee that the file is NOT sparse, even if it
happens to issue a sequence of lseek() that would default to making it
sparse without the fallocate.

In other words, I'm proposing that we delete nothing from cygwin1.dll,
and instead fix the problem apps (gcc, emacs unexec) that actually
create executables, so that the files they create are non-sparse because
we have proven that they should not be sparse for performance reasons.
Meanwhile, all non-executable files (such as virtual machine disk
images, which are typically much bigger than executables, and where
being sparse really does matter) do not have to jump through extra hoops
of using ftruncate() when plain lseek() would do to keep them sparse.
Does gcc/ld/whatever know the final file size before the first write?

You have to posix_fallocate the entire file before any write that might create a hole, because the sparse flag poisons the loader, and persists even if all gaps are later filled.
Heh... hit send too soon.

Alternatively, a quick experiment verifies that calling pwrite instead of lseek+write bypasses the sparse-ifying "optimization." If emacs and binutils always seek before writing, it might be as simple as patching them to use pwrite instead. That would even improve performance on other platforms with pwrite, by cutting the syscall count in half.

A quick scan of binutils sources suggests that all*** file writes go through libiberty/simple-object.c:simple_object_internal_write, which indeed uses an lseek+write pair. As long as gcc uses libiberty to write out executables as well, it should pick up the fix automatically.

The emacs (24.0.96) unexecw.c copies the entire executable file over using read/write pairs (= not sparse), and then patches the output using seek/write pairs. Again, an easy conversion to use pread instead.

*** I'm actually not sure what gold does, but we don't care because it doesn't target cygwin anyway.

Ryan


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]