How to mirror/sync a remote ftp folder with a local folder

Blog
Linux
Sys Administration

lftp is one of the best command line utilities currently available for working with ftp servers, One of its useful features is mirroring which is suitable for backup purposes and very useful for slow connections. In this article i'm going to show you how to use this utility to backup and mirror your files a ftp server.

Simple Usage

The following script mirros your local folder to a remote ftp server, it only updates the changes you've made since the last execution so only the first time requires lots of bandwidth :

#!/bin/bash
HOST='mysite.com'
USER='myuser'
PASS='myuser'
TARGETFOLDER='/new'
SOURCEFOLDER='/home/myuser/backups'

lftp -f "
open $HOST
user $USER $PASS
lcd $SOURCEFOLDER
mirror --reverse --delete --verbose $SOURCEFOLDER $TARGETFOLDER
bye
"

You can save it as upload.sh and execute it using the following command :

bash upload.sh

Advanced Usage

lftp offers many parameters, here i'll explain some of the most useful ones :

  • --reverse : Indicates that you want to update your remote folder. if you want to update your local folder then you should remove this option and also swap values of TARGETFOLDER and SOURCEFOLDER variables
  • --delete : It tells lftp that you want to remove all the files/folders that no longer exist on source folder
  • --use-cache : If you're about to sync very large number of files, this option can speed up things a lot. It does not have any effect the first time but in the next sessions it uses local cache instead of scanning the remote server
  • --exclude : Sometimes there are folder/files on both source and target folders that have nothing to do with the mirroring, using this option we can tell lftp to ignore this sort of files

Tips

FTP servers' version and configuration is not always the same , so the above scripts might not always work,

The usual issue is Passive Mode, and occurs when the client (you) is behind proxy or firewall.

Windows ?

It's possible to use this utility under windows by downloading this files from  Cygwin Information and Installation :

cygcrypto-0.9.7.dll
cygiconv-2.dll
cygintl-2.dll
cygncurses7.dll
cygreadline5.dll
cygssl-0.9.7.dll
cygwin1.dll
lftp.exe

I've attached executable in case you had trouble downloading them from cygwin server. BUT CHECK THEM FILES WITH YOUR ANTIVIRUS FIRST

Resources

AttachmentSize
lftp-install.zip821.06 KB
lftp.zip1.79 MB
Your rating: None Average: 3.1 (17 votes)

Comments

DanyC's picture

Hi, Have you tried this

Hi,

Have you tried this configuration over http proxy?

Thx,
Dani

admin's picture

No, but lftp seem to support

No, but lftp seem to support http proxy http://lftp.yar.ru/lftp-man.html
look for ftp:proxy

Tobias Rådenholt's picture

Thanks alot. Works like a

Thanks alot. Works like a charm!

Lucy's picture

Awesome, this is what exactly

Awesome, this is what exactly I was searching for! Good One, thanks for sharing!

sigma's picture

Works like a charm, very

Works like a charm, very interesting with the sync job.
But if the ftp server is down it shouldn.t retry over and over again.
How to limit the number of tries and then finish the script?

Thanks in advance!

Ramesh's picture

Thanks..lot

Thanks..lot