This is the mail archive of the cygwin-developers mailing list for the Cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Performance optimization in av::fixup - use buffered IO, not mapped file


On Dec 12 09:47, Eric Blake wrote:
> On 12/12/2012 08:39 AM, Ryan Johnson wrote:
> 
> > Does gcc/ld/whatever know the final file size before the first write?
> 
> No, but does it need to?  posix_fallocate() does not change file
> contents; it merely says that anywhere there was previously a hole must
> now be guaranteed to be backed by disk.  So gcc would write the file as
> usual, and then just before close()ing the fd, do a final
> posix_fallocate(fd, 0, len) with len determined by the final file size.
> 
> > 
> > You have to posix_fallocate the entire file before any write that might
> > create a hole, because the sparse flag poisons the loader,
> 
> Is there really a flag stuck into the file when it becomes sparse?

Yes.  And, as I wrote, you can't remove it pre-Vista.

> > cp --sparse=never $(which emacs-nox) dense
> > for f in sparse dense; do echo $f; time ./$f -Q --batch --eval
> > '(kill-emacs)'; done
> > cp --sparse=never dense sparse
> > for f in sparse dense; do echo $f; time ./$f -Q --batch --eval
> > '(kill-emacs)'; done
> > du dense sparse
> 
> This doesn't point to a flag in the file, so much as cached information
> (the file system is remembering that 'sparse' used to be sparse, even if
> it is no longer sparse).  But your point about a file being cached at
> some point while it is sparse, even if it is later made non-sparse, is
> interesting.
> 
> > 
> > The relevant output is:
> >> sparse
> >> real    0m1.791s
> >>
> >> dense
> >> real    0m0.606s
> >>
> >> sparse
> >> real    0m3.158s
> >>
> >> dense
> >> real    0m0.081s
> >>
> >> 16728   dense
> >> 16768   sparse
> > 
> > Given that we're talking about cygwin-specific patches for emacs and
> > binutils anyway, would it be better to add a cygwin-specific fcntl call
> > that clears the file's sparse flag?
> 
> What flag is there to clear?  Your cp demonstration showed that even
> when we do a byte-for-byte copy of every byte (and the file is
> non-sparse), the file system cache remembers that it used to be sparse.
>  How do we defeat that file system cache?

Another question is, is that behaviour reproducible?  Does it happen the
second time the "new" non-sparse sparse file is called?  You don't even
know if the slowness is a result of writing the file is still in flight.
Windows caching can be pretty slow at times, but it recovers quickly
if a file is used again, usually.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]