LFTP -- an excellent, scriptable FTP/SFTP client

LFTP is available on Unix systems (Solaris, Linux, etc)  as well as Windows (it is included in the standard list of packages for Cygwin). Very well designed program that offers dramatic productivity boost for knowledgeable admin and advanced users who need to use ftp often.  It was written by  Alexander V. Lukyanov .

This is an excellent, really excellent client. Since version 3.0 it supports sftp.  It will reconnect and continue transfers in the event of a disconnection. if you quit the program while transfers are still in progress, it will switch to nohup mode and finish the transfers in the background. Mirroring capabilities are not that important: most mirroring program like wget support ftp

I instantly discarded ncftp when I learned about lftp.

• lftp has shell-like command syntax allowing you to launch several commands in parallel in background
• Every operation in lftp is reliable, that is any not fatal error is ignored and the operation is repeated. So if downloading breaks, it will be restarted from the point automatically. Even if ftp server does not support REST command, lftp will try to retrieve the file from the very beginning until the file is transferred completely.
• lftp can download or update a whole directory tree. There is also reverse mirror (mirror -R) which uploads or updates a directory tree on server. Mirror can also synchronize directories between two remote servers, using FXP if available.

There is the author web site devoted to this program: LFTP - sophisticated file transfer program

• Description
• Feature list  very impressive; must read ! --NNB
• Man page
• Some FAQs with answers
• Tutorial (by Peter Matulis)
• Download
• Changes
• Mailing lists
• Author
• Related RFC documents
• Contribute!

LFTP is sophisticated file transfer program with command line interface. It supports FTP, HTTP, FISH, SFTP, HTTPS and FTPS protocols. GNU Readline library is used for input.

Every operation in lftp is reliable, that is any non-fatal error is handled and the operation is retried automatically. So if downloading breaks, it will be restarted from the point automatically. Even if ftp server does not support REST command, lftp will try to retrieve the file from the very beginning until the file is transferred completely. This is useful for dynamic-ip machines which change their IP addresses quite often, and for sites with very bad internet connectivity.

If you exit lftp when some jobs are not finished yet, lftp will move itself to nohup mode in background. The same happens when you have a real modem hangup or when you close an xterm.

lftp has shell-like command syntax allowing you to launch several commands in parallel in background (&). It is also possible to group commands within () and execute them in background. All background jobs are executed in the same single process. You can bring a foreground job to background with ^Z (c-z) and back with command `wait' (or `fg' which is alias to `wait'). To list running jobs, use command `jobs'. Some commands allow redirecting their output (cat, ls, ...) to file or via pipe to external command. Commands can be executed conditionally based on termination status of previous command (&&, ||).

Examples:

	lftp> cat file | gzip > file.gz
	lftp> get file &
	lftp> (cd /path && get file) &

The first command retrieves file from ftp server and passes its contents to gzip which in turn stores compressed data to file.gz. Other commands show how to start commands or command groups in background.

lftp has builtin mirror which can download or update a whole directory tree. There is also reverse mirror (mirror -R) which uploads or updates a directory tree on server.

There is command `at' to launch a job at specified time in current context, command `queue' to queue commands for sequential execution for current server, and much more.

LFTP supports IPv6 for both FTP and HTTP protocols. For FTP protocol it uses method described in RFC2428.

Other low level stuff supported: ftp proxy, http proxy, ftp over http, opie/skey, fxp transfers, socks.

LFTP supports secure versions of the protocols FTP and HTTP: FTPS (explicit and implicit) and HTTPS. LFTP needs to be libked with an SSL library to support them. GNU TLS and OpenSSL are both supported as SSL backend.

If lftp was compiled with OpenSSL library, then it includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit. (http://www.openssl.org/)

See FEATURES for more detailed list of features.

See man page lftp(1) for more details.

NEWS CONTENTS

• 20111025 : lftp-4.3.3 released ( lftp-4.3.3 released, Oct 25, 2011 )
• 20071204 : CLI Magic Quick and easy backup with lftp by Dmitri Popov ( December 04 , Linux.com )

Old News ;-)

[Oct 25, 2011] lftp-4.3.3 released

• new setting ftp:use-tvfs (yes, no, auto).
• improved ftp path handling for servers without TVFS feature.
• improved closure matching, now *.EXT matches URLs ending with ".EXT".
• updated man page.
• updated translations.
• fixed mirror target directory naming.

[Dec 04, 2007] CLI Magic Quick and easy backup with lftp by Dmitri Popov

wget has similar functionality and is more versatile (can use HTTP).

December 04 | Linux.com

No matter what Linux distribution you are using, chances are you'll find more than one graphical FTP client in its repositories, but if you are looking for a powerful command-line FTP tool, your best bet is lftp. Of course, you can always use the good old ftp command, but lftp takes the task of managing files and directories using the FTP protocol to a new level. To see what I mean, let's use lftp to write a script that creates a local backup copy of a Web site.

To write the script, you need to know how to use lftp to connect to an FTP server and synchronize a remote directory with a local one. If your FTP server supports anonymous connections, you can connect to it using the simple command lftp ftpsite. If the server requires a user name and password, the connection command would look like lftp -u username,password ftpsite.

To synchronize a remote directory with a folder on your hard disk, lftp utilizes the mirror command. Used without switches, this command syncs the current local and remote directories. You can also specify explicitly the source and target directories:

mirror path/to/source_directory path/to/target_directory

The mirror command offers a comprehensive set of switches, which you can use to control the synchronization process. For example, used with the --delete switch, the mirror command deletes the files in the local folder that are not present in the remote directory, while the --only-newer option forces lftp to download only newer files. Another handy switch is --exclude; it allows you to specify which files and directories to skip during synchronization. And if you prefer to keep an eye on the syncing process, you can use the --verbose switch.

Typing all those switches every time you want to synchronize two directories can be a bit of a bother. Fortunately, lftp understands complex commands that can perform several actions in one fell swoop. All you have to do is to use the -e switch, so lftp stays connected and runs the specified commands:

lftp -u username,password  -e "mirror --delete --only-newer --verbose path/to/source_directory path/to/target_directory" ftpsite

Using this command, lftp connects to the FTP server using the provided credentials, and then runs the command(s) in the quotes. You can save the entire command in a text file, then run it by pointing lftp to it using the -f switch:

lftp -f /home/user/ftpscript.txt

lftp has a few other clever tricks up its sleeve. The at switch can come in handy when you want to run the backup at a specific time. The following command, for example, runs at midnight:

lftp at 00:00 -u username,password -e "mirror --delete --only-newer --verbose path/to/source_directory path/to/target_directory" ftpsite &

Notice the ampersand, which sends the command to the background so you don't have to keep the terminal window open.

Now you know how to create local backup of files and directories stored on an FTP server. But how do you restore the data if disaster strikes? Quite easily, actually. All you have to do is to add the --reverse switch to the mirror command:

lftp -u username,password -e "mirror --reverse --delete --only-newer --verbose path/to/source_directory path/to/target_directory" ftpsite

As the name suggests, the switch reverses the source and target directories, so lftp uploads files from the local directory to the remote FTP server.

That's all there is to it. Check lftp's man pages to get an overview of lftp's other useful options, and start FTPing like a pro.

Dmitri Popov is a freelance writer whose articles have appeared in Russian, British, US, German, and Danish computer magazines.

Recommended Links

• LFTP - sophisticated file transfer program
• lftp: a better FTP client by Peter Matulis
• lftp - Wikipedia, the free encyclopedia
• Linux Command Directory lftp - O'Reilly Media
• LinuxPlanet - Tutorials - Moving Files In Linux - lftp
• lftp to accelerate ftp - http download speed under Linux and UNIX
• Linux - Unix lftp Command Mirror Files and Directories
• Linux - Unix ncftp Upload Directory Tree To Remote FTP Server Recursively - nixCraft

Reference

lftp [options] [url]

File transfer program with more features than ftp. The lftp command allows FTP and HTTP protocol transfers, plus other protocols including FISH (SSH based), FTPS, and HTTPS. It uses a shell-like command interface and offers job control in a manner similar to bash. lftp has two important reliability features: it resumes failed or interrupted transactions, and it goes into the background automatically if it is quit in the middle of a file transfer.

Options

Run in debug mode.

Start, execute the specified commands, and then wait for further instructions.

Connect to the specified port number.

Login to the server with the username (and, optionally, password) you specify.

Run the specified script file of lftp commands, then exit.

Run the commands specified, then exit.

Commands

The lftp commands are similar to those for ftp. However, lftp lacks or uses different mechanisms for a number of commands, including $, ascii, binary, case, and macdef. It also adds the following:

Create an alias for a command. For example, you could set dir to be an alias for ls -lf.

Set the username to anonymous. This is the default username.

Execute a command at a given time, as with the at command in an actual shell.

The lftp bookmark command used with the following arguments will add, delete, edit, import, or list bookmarks, respectively:

add name url

del name

edit

import type

list

Work with the local memory cache. This command should be followed by the arguments:

stat

Display the status for the cache.

on|off

Turn caching on or off.

flush

Empty the cache.

size n

Set the maximum size for the cache. Setting it to -1 means unlimited.

expire nu

Set the cache to expire after n units of time. You can set the unit to seconds (s), minutes (m), hours (h), or days (d). For example, for a cache that expires after an hour, use the syntax cache expire 1h.

Where the ftp version of this command just stops all sessions, this version closes idle connections with the current server. If you have connections to multiple servers and wish to close all idle connections, add the -a flag.

Execute the specified lftp command, with the specified arguments, ignoring any aliases created with the alias command.

Copy a directory exactly. The mirror command accepts the following arguments:

-c, --continue

If mirroring was interrupted, resume it.

-e, --delete

Delete local files that are not present at the remote site.

-s, --allow-suid

Keep the suid/sgid bits as set on the remote site.

-n, --only-newer

Get only those files from the remote site that have more recent dates than the files on the local system. Cannot be used with the -c argument.

-r, --no-recursion

Do not get any subdirectories.

--no-umask

Do not use umask when getting file modes. See umask for more information about file modes.

-R, --reverse

Mirror files from the local system to the remote system. With this argument, make sure that you specify the local directory first and the remote directory second. If you do not specify both directories, the second is assumed to be the same as the first. If you choose neither, the operation occurs in the current working directories.

-L, --dereference

When mirroring a link, download the file the link points to rather than just the link.

-N, --newer-than filename

Get all files newer than the file filename.

-P, --parallel[=n]

Download n files in parallel.

-i, --include regex

Get only the files whose names match the regular expression regex. See grep for more information about regular expressions.

-x, --exclude regex

Do not get the files whose names match regex. See grep for more information about regular expressions.

-t, time-prec n

Set the precision of time measurement for file comparison; if file dates differ by amounts less than n, they are assumed to be the same. You can specify n in seconds (s), minutes (m), hours (h), or days (d).

-T, --loose-time-prec n

Set the precision for loose time comparisons. You can specify n in seconds (s), minutes (m), hours (h), or days (d).

-v, --verbose=n

Set the verbose level. You can set n from 0 (no output) to 3 (full output) using a number or by repeating the v. For example, -vvv is level 3 verbose mode.

--use-cache

Use the cache to get directory listings.

--remove-source-files

Move, rather than copy, files when mirroring.

Set one of the preference variables for lftp. If run without arguments, list the variables that have been changed; without arguments and with the -a or -d flags, list all values or default values, respectively.

See the lftp manpage for a complete list of preference variables that can be set.

Wait for the job or jobs you specify by number, or all jobs, to terminate.

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: July 28, 2019