This is the mail archive of the cygwin mailing list for the Cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Rsync over SSH dies before it is able to send with lots of files


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to Bob Brown on 8/9/2005 10:25 PM:
> 
>>Hmm, you're making bash expand a wildcard into 45000 files.  This
>>shouldn't crash bash, but I'm guessing there is some of memory
>>corruption or malloc failure when consuming so much memory all before
>>forking(). I'll have to see if I can try and reproduce the failure.
> 
> Do you think the win32 path length will be included for every file?  If this is 
> the case you can probably safely assume 30-40 bytes per filename.  If this 
> were the case it would quickly add up to about 1.7MB in just passed 
> filenames alone.  Considering that this will ultimately process a directory 
> with 500,000 files in it it's not hard to work out that things are going to go 
> pear shaped in a hurry.

I still want to figure out why you were getting a crash.  I tried the
following:

$ cd /tmp
$ mount -m
...
mount -f -s -b -X "c:/cygwin/bin" "/bin"
$ mkdir lots
$ # 45000 invocations of touch is slow, so do 10 files at a time...
$ for i in `seq 4500` ; do echo $i ; touch longfilename${i}0
longfilename${i}1 longfilename${i}2 longfilename${i}3 longfilename${i}4
longfilename${i}5 longfilename${i}6 longfilename${i}7 longfilename${i}8
longfilename${i}9 ; done
$ echo /cygdrive/c/cygwin/tmp/lots/*
... # Works, echo is a builtin
$ /bin/echo /cygdrive/c/cygwin/tmp/lots/*
... # Works for me, since my /bin is mounted executable
$ /cygdrive/c/cygwin/bin/echo /cygdrive/c/cygwin/tmp/lots/*
bash: /cygdrive/c/cygwin/bin/echo: Argument list too long

Hmm, I got a reasonable error message instead of a crash.  I'll keep
playing with it, to throw more data at the command line, to see if the
amount of data contributes to a crash.  Since echo is a built-in, you
should be able to do "echo rsync --progress -rtv
192.168.0.1::mydirectory/* c:\localcopy | wc" as an indication of how many
bytes were being attempted on the command line.

- --
Life is short - so eat dessert first!

Eric Blake             ebb9@byu.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFC+fyD84KuGfSFAYARAowfAJ4oyt+h4zZyxBzg5DhVJ+6E6vHNagCfZ1Yw
ysUikKj057gZkNKgQrmQ0w4=
=3Ubk
-----END PGP SIGNATURE-----

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]