...making Linux just a little more fun!

### rsync not working when trying to update LG mirror

Suramya Tomar [security at suramya.com]

Wed, 03 Oct 2007 22:33:18 +0530

Hey everyone, I maintain a mirror for LG at my site and recently it was brought to my attention that the mirror had gone out of date. I had the update script set up as a cron job so didn't monitor it. But apparently it has been failing with the following errors:

u38576182:~/public_html/suramya.com/linux/gazette > ./sync_mirror.sh rsync: failed to connect to linuxgazette.net: Connection refused rsync error: error in socket IO (code 10) at clientserver.c(97) rsync: failed to connect to linuxgazette.net: Connection refused rsync error: error in socket IO (code 10) at clientserver.c(97)

The sync script contains the following code:

u38576182:~/public_html/suramya.com/linux/gazette > cat sync_mirror.sh RSYNC_RSH=/usr/bin/ssh export RSYNC_RSH rsync -avz --exclude /ftpfiles/ linuxgazette.net::lg-all www_root rsync -avz linuxgazette.net::lg-ftp ftpfiles

This only happens on this one particular system. When I try it on my local system it runs without issues. Any idea why the lingazette.net is rejecting the rsync connection? If I try to ssh to linuxgazette.net from the server I get a password prompt.

This server has rsync version 2.5.6cvs protocol version 26 installed. Its a shared hosting account so I don't have root access on the server.

Any help would be appreciated

Thanks, Suramya

    --

    Name : Suramya Tomar
    Homepage URL: http://www.suramya.com

Top Back

René Pfeiffer [lynx at luchs.at]

Wed, 3 Oct 2007 19:20:37 +0200

Hello, Suramya!

On Oct 03, 2007 at 2233 +0530, Suramya Tomar appeared and said:

    > [...]
    > u38576182:~/public_html/suramya.com/linux/gazette > ./sync_mirror.sh
    >       rsync: failed to connect to linuxgazette.net: Connection refused
    > rsync error: error in socket IO (code 10) at clientserver.c(97)
    > rsync: failed to connect to linuxgazette.net: Connection refused
    > rsync error: error in socket IO (code 10) at clientserver.c(97)
    >
    > The sync script contains the following code:
    >
    > u38576182:~/public_html/suramya.com/linux/gazette > cat sync_mirror.sh
    > RSYNC_RSH=/usr/bin/ssh
    > export RSYNC_RSH
    > rsync -avz --exclude /ftpfiles/ linuxgazette.net::lg-all www_root
    > rsync -avz linuxgazette.net::lg-ftp ftpfiles

Do you always use "RSYNC_RSH=/usr/bin/ssh" when rsyncing? My mirror script simply contains the following:

#!/bin/sh

rsync -avz --delete --exclude /ftpfiles/ linuxgazette.net::lg-all \ /home/web/linuxgazette/ > /tmp/lg_mirror.log

The rsync command has this version:

rsync version 2.6.9 protocol version 29 Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others. <http://rsync.samba.org/> Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, inplace, IPv6, ACLs, 64-bit system inums, 64-bit internal inums

rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details.

Have you tried omitting RSYNC_RSH?

Best, Ren?.

Top Back

Ben Okopnik [ben at linuxgazette.net]

Wed, 3 Oct 2007 13:52:15 -0500

On Wed, Oct 03, 2007 at 10:33:18PM +0530, Suramya Tomar wrote:

    > Hey everyone,
    >   I maintain a mirror for LG at my site and recently it was brought to
    > my attention that the mirror had gone out of date. I had the update
    > script set up as a cron job so didn't monitor it. But apparently it has
    > been failing with the following errors:
    >
    > u38576182:~/public_html/suramya.com/linux/gazette > ./sync_mirror.sh
    >       rsync: failed to connect to linuxgazette.net: Connection refused
    > rsync error: error in socket IO (code 10) at clientserver.c(97)
    > rsync: failed to connect to linuxgazette.net: Connection refused
    > rsync error: error in socket IO (code 10) at clientserver.c(97)
    >
    > The sync script contains the following code:
    >
    > u38576182:~/public_html/suramya.com/linux/gazette > cat sync_mirror.sh
    > RSYNC_RSH=/usr/bin/ssh
    > export RSYNC_RSH
    > rsync -avz --exclude /ftpfiles/ linuxgazette.net::lg-all www_root
    > rsync -avz linuxgazette.net::lg-ftp ftpfiles
    >
    > This only happens on this one particular system. When I try it on my
    > local system it runs without issues. Any idea why the lingazette.net is
    > rejecting the rsync connection? If I try to ssh to linuxgazette.net from
    > the server I get a password prompt.

It sounds to me like the RSYNC_RSH variable is being ignored by your version of 'rsync'. You might want to try making it an explicit commandline argument, e.g.

    rsync -avz --rsh=ssh linuxgazette.net::lg-ftp ftpfiles

    > This server has rsync version 2.5.6cvs  protocol version 26 installed.
    > Its a shared hosting account so I don't have root access on the server.

Mine is

    rsync  version 2.6.6  protocol version 29

    --
    * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *

Top Back

Suramya Tomar [security at suramya.com]

Thu, 04 Oct 2007 00:39:48 +0530

Hey Ren?,

    > Do you always use "RSYNC_RSH=/usr/bin/ssh" when rsyncing? My mirror
    > script simply contains the following:

I was using it because it said to do so on the LG Mirror guidelines page.

    >
    > #!/bin/sh
    >
    > rsync -avz --delete --exclude /ftpfiles/ linuxgazette.net::lg-all \
    >  /home/web/linuxgazette/ > /tmp/lg_mirror.log
    >

<Snip>

    > Have you tried omitting RSYNC_RSH?

I tried using omitting the RSYNC_RSH and I got the following message:

u38576182:~/public_html/suramya.com/linux/gazette > rsync -avz --delete --exclude /ftpfiles/ linuxgazette.net::lg-all www_root/ rsync: failed to connect to linuxgazette.net: Connection refused rsync error: error in socket IO (code 10) at clientserver.c(97)

On my local machine I am running version 2.6.9 protocol version 29. Could it be an issue with version 26? A search on Google didn't return any relevant hits.

Thanks, Suramya

    --

    Name : Suramya Tomar
    Homepage URL: http://www.suramya.com

Top Back

Suramya Tomar [security at suramya.com]

Thu, 04 Oct 2007 00:43:02 +0530

Hey Ben,

    > It sounds to me like the RSYNC_RSH variable is being ignored by your
    > version of 'rsync'. You might want to try making it an explicit
    > commandline argument, e.g.
    >
    > rsync -avz --rsh=ssh linuxgazette.net::lg-ftp ftpfiles
    > '''

When I try the above command I get the following:

u38576182:~/public_html/suramya.com/linux/gazette > rsync -avz --rsh=ssh linuxgazette.net::lg-ftp ftpfiles WARNING: --rsh or -e option ignored when connecting to rsync daemon rsync: failed to connect to linuxgazette.net: Connection refused rsync error: error in socket IO (code 10) at clientserver.c(97) u38576182:~/public_html/suramya.com/linux/gazette >

so it looks like you are right, it is ignoring the RSYNC_RSH variable. Is there any other way I can force it to use ssh? Otherwise I will have to call and yell at my hosting provider to upgrade their rsync version (which I am pretty doubtful they would do).

Thanks, Suramya

    --

    Name : Suramya Tomar
    Homepage URL: http://www.suramya.com

Top Back

René Pfeiffer [lynx at luchs.at]

Wed, 3 Oct 2007 21:21:32 +0200

Re!

On Oct 04, 2007 at 0039 +0530, Suramya Tomar appeared and said:

    > > Do you always use "RSYNC_RSH=/usr/bin/ssh" when rsyncing? My mirror
    > > script simply contains the following:
    >
    > I was using it because it said to do so on the LG Mirror guidelines page.

Oops. I just "retrofitted" my script with RSYNC_RSH and it works. The error should be somewhere else then.

    > [...]
    > On my local machine I am running version 2.6.9  protocol version 29.
    > Could it be an issue with version 26? A search on Google didn't return
    > any relevant hits.

This could be the case. I have no idea how far the rsync versions can differ. I always try to match rsync server and client as close as possible.

Best regards, Ren?.

Top Back

Ben Okopnik [ben at linuxgazette.net]

Wed, 3 Oct 2007 14:46:14 -0500

On Thu, Oct 04, 2007 at 12:43:02AM +0530, Suramya Tomar wrote:

    > Hey Ben,
    >
    > > It sounds to me like the RSYNC_RSH variable is being ignored by your
    > > version of 'rsync'. You might want to try making it an explicit
    > > commandline argument, e.g.
    > >
    > > rsync -avz --rsh=ssh linuxgazette.net::lg-ftp ftpfiles
    > > '''
    >
    > When I try the above command I get the following:
    >
    > u38576182:~/public_html/suramya.com/linux/gazette > rsync -avz --rsh=ssh
    > linuxgazette.net::lg-ftp ftpfiles
    > WARNING: --rsh or -e option ignored when connecting to rsync daemon
    > rsync: failed to connect to linuxgazette.net: Connection refused
    > rsync error: error in socket IO (code 10) at clientserver.c(97)
    > u38576182:~/public_html/suramya.com/linux/gazette >
    >
    > so it looks like you are right, it is ignoring the RSYNC_RSH variable.
    > Is there any other way I can force it to use ssh? Otherwise I will have
    > to call and yell at my hosting provider to upgrade their rsync version
    > (which I am pretty doubtful they would do).

Heck with it; since you're getting both the HTTP and the FTP content, just use 'wget' and make life simple.

    wget --mirror http://linuxgazette.net/

That should do it.

    --
    * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *

Top Back

Jim Jackson [jj at franjam.org.uk]

Thu, 4 Oct 2007 14:21:02 +0100 (BST)

On Thu, 4 Oct 2007, Suramya Tomar wrote:

    > Hey Ben,
    >
    > > It sounds to me like the RSYNC_RSH variable is being ignored by your
    > > version of 'rsync'. You might want to try making it an explicit
    > > commandline argument, e.g.
    > >
    > > rsync -avz --rsh=ssh linuxgazette.net::lg-ftp ftpfiles
    > > '''
    >
    > When I try the above command I get the following:
    >
    > u38576182:~/public_html/suramya.com/linux/gazette > rsync -avz --rsh=ssh
    > linuxgazette.net::lg-ftp ftpfiles
    > WARNING: --rsh or -e option ignored when connecting to rsync daemon
    > rsync: failed to connect to linuxgazette.net: Connection refused
    > rsync error: error in socket IO (code 10) at clientserver.c(97)
    > u38576182:~/public_html/suramya.com/linux/gazette >
    >
    > so it looks like you are right, it is ignoring the RSYNC_RSH variable.
    > Is there any other way I can force it to use ssh?

reading Manual pages is useful and fun

Top Back

Ben Okopnik [ben at linuxgazette.net]

Thu, 4 Oct 2007 08:52:59 -0500

On Thu, Oct 04, 2007 at 02:21:02PM +0100, Jim Jackson wrote:

    > On Thu, 4 Oct 2007, Suramya Tomar wrote:
    >
    > > so it looks like you are right, it is ignoring the RSYNC_RSH variable.
    > > Is there any other way I can force it to use ssh?
    >
    > reading Manual pages is useful and fun
    >
    > >From the rsync manual page....
    >
    > ..... Contacting an rsync daemon directly happens when the source or
    > destination path contains a double colon (: separator after a host
    > specification.....
    >
    > and you are using a double colon seperator. Use the
    >
    >        rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
    >
    > form - with a single operator to use ssh.

The only problem with that is, he won't be able to use the targets that are specified for retrieving the content. At that point, he might as well use 'wget' as I suggested - although using 'rsync' does have some extra benefits.

    --
    * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *

Top Back

Jim Jackson [jj at franjam.org.uk]

Thu, 4 Oct 2007 15:42:12 +0100 (BST)

On Thu, 4 Oct 2007, Ben Okopnik wrote:

    > On Thu, Oct 04, 2007 at 02:21:02PM +0100, Jim Jackson wrote:
    > > On Thu, 4 Oct 2007, Suramya Tomar wrote:
    > >
    > > > so it looks like you are right, it is ignoring the RSYNC_RSH variable.
    > > > Is there any other way I can force it to use ssh?
    > >
    > > reading Manual pages is useful and fun
    > >
    > > >From the rsync manual page....
    > >
    > > ..... Contacting an rsync daemon directly happens when the source or
    > > destination path contains a double colon (: separator after a host
    > > specification.....
    > >
    > > and you are using a double colon seperator. Use the
    > >
    > >        rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
    > >
    > > form - with a single operator to use ssh.
    >
    > The only problem with that is, he won't be able to use the targets that
    > are specified for retrieving the content. At that point, he might as
    > well use 'wget' as I suggested - although using 'rsync' does have some
    > extra benefits.

Indeed.

But I did answer the specific question asked re. rsync

Top Back

Ben Okopnik [ben at linuxgazette.net]

Thu, 4 Oct 2007 10:08:46 -0500

On Thu, Oct 04, 2007 at 03:42:12PM +0100, Jim Jackson wrote:

    > On Thu, 4 Oct 2007, Ben Okopnik wrote:
    >
    > > The only problem with that is, he won't be able to use the targets that
    > > are specified for retrieving the content. At that point, he might as
    > > well use 'wget' as I suggested - although using 'rsync' does have some
    > > extra benefits.
    >
    > Indeed.
    >
    > But I did answer the specific question asked re. rsync

"The operation was successful, but the patient died." That's logic for ya.

    --
    * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *

Top Back

Suramya Tomar [security at suramya.com]

Fri, 05 Oct 2007 21:39:38 +0530

Hey Ben,

    > Heck with it; since you're getting both the HTTP and the FTP content,
    > just use 'wget' and make life simple.
    >
    > ``
    > wget --mirror http://linuxgazette.net/
    > ''
    >
    > That should do it.

That would get me a copy of the LG website as long as there are no dynamic scripts involved otherwise I would end up getting static copies of the processed pages.

I think I will use it for now till I get the rsync issue sorted out, but with this I would we re-downloading ~400 MB of data every month that I already have... I would like to avoid that if I could.

- Suramya

    --

    Name : Suramya Tomar
    Homepage URL: http://www.suramya.com

Top Back

Suramya Tomar [security at suramya.com]

Fri, 05 Oct 2007 21:43:51 +0530

Hey Jim,

    >> so it looks like you are right, it is ignoring the RSYNC_RSH variable.
    >> Is there any other way I can force it to use ssh?
    >
    > reading Manual pages is useful and fun

Yeah. Reading it is fun, figuring out what it means usually gets interesting.

    > ..... Contacting an rsync daemon directly happens when the source or
    > destination path contains a double colon (: separator after a host
    > specification.....
    >
    > and you are using a double colon seperator. Use the
    >
    >        rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
    >
    > form - with a single operator to use ssh.
    >

When I try using a single colon separator it asks me for a password:

u38576182:~/public_html/suramya.com/linux/gazette > RSYNC_RSH=/usr/bin/ssh u38576182:~/public_html/suramya.com/linux/gazette > export RSYNC_RSH u38576182:~/public_html/suramya.com/linux/gazette > rsync -avz --exclude /ftpfiles/ linuxgazette.net:lg-all www_root u38576182 at linuxgazette.net's password: rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(229)

I terminated the session because I know that the user u38576182 doesn't exist on the LG.net site.

Thanks for the help.

- Suramya

    --

    Name : Suramya Tomar
    Homepage URL: http://www.suramya.com

Top Back

Rick Moen [rick at linuxmafia.com]

Fri, 5 Oct 2007 10:30:42 -0700

Quoting Suramya Tomar (security at suramya.com):

    > Hey Jim,
    >
    > >> so it looks like you are right, it is ignoring the RSYNC_RSH variable.
    > >> Is there any other way I can force it to use ssh?
    > >
    > > reading Manual pages is useful and fun
    >
    > Yeah. Reading it is fun, figuring out what it means usually gets
    > interesting.

FWIW, I too have found rsync to frequently ignore the environment variable, even though I have it always set in my shell environment:

$ set | grep RSYNC RSYNC_RSH=/usr/bin/ssh

Frustrating, isn't it? It's so messy to junk up scripts (and command lines) with "-e ..." stuff, that it's really tempting to just do:

$ su - # cd /usr/bin ; mv rsh rsh.old ; ln ssh rsh # exit

Grrr!

Top Back

Jim Jackson [jj at franjam.org.uk]

Sun, 7 Oct 2007 10:42:14 +0100 (BST)

On Fri, 5 Oct 2007, Suramya Tomar wrote:

    > > ..... Contacting an rsync daemon directly happens when the source or
    > > destination path contains a double colon (: separator after a host
    > > specification.....
    > >
    > > and you are using a double colon seperator. Use the
    > >
    > >        rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
    > >
    > > form - with a single operator to use ssh.
    > >
    >
    > When I try using a single colon separator it asks me for a password:
    >
    > u38576182:~/public_html/suramya.com/linux/gazette > RSYNC_RSH=/usr/bin/ssh
    > u38576182:~/public_html/suramya.com/linux/gazette > export RSYNC_RSH
    > u38576182:~/public_html/suramya.com/linux/gazette > rsync -avz --exclude
    > /ftpfiles/ linuxgazette.net:lg-all www_root
    > u38576182 at linuxgazette.net's password: rsync error: received SIGUSR1 or
    > SIGINT (code 20) at rsync.c(229)
    >
    > I terminated the session because I know that the user u38576182 doesn't
    > exist on the LG.net site.

errr.... yes it will do. That is how ssh works, you need an account at the far end to use.

If you have one then use

remoteaccountname at remote.host.name:dirs/etc

It will still ask for a passwrod unless you have set up you keys to allow access. This is agood overview of ssh

http://www.ibm.com/developerworks/aix/library/au-satopenssh.html

If you don't have a remote account, then why were you trying to use ssh???

Jim

Top Back

Suramya Tomar [security at suramya.com]

Fri, 12 Oct 2007 01:33:17 +0530

Hey,

    >> I terminated the session because I know that the user u38576182 doesn't
    >> exist on the LG.net site.
    >
    > errr.... yes it will do. That is how ssh works, you need an account at the
    > far end to use.

<snip>

    > If you don't have a remote account, then why were you trying to use ssh???

I tried it because it was suggested I try doing so. It didn't work so was letting everyone know that it didn't work.

Anyways, as a temp fix I 'solved' the issue by syncing the mirror to my local system and then syncing that with the www server.

I did try using the wget --mirror as suggested by Ben but that didn't work. wget died after a while with the following error message:

wget: realloc: Failed to allocate 1048576 bytes; memory exhausted.

I will try calling up the tech support and see if they would be willing to upgrade the rsync version on the server. Not too hopeful though.

Thanks for the help everyone.

- Suramya

    --

    Name : Suramya Tomar
    Homepage URL: http://www.suramya.com

Top Back