======== Goal
- Transfer full trees of files from M1 to/from M2 with rsync
	- Excellent for files that are not being modified
	- Good for files that are being appended to, just remember to run once more when the appending is done so you send the final version.
	- Not great for database-type files where changes are actively being made (anywhere in the file(s)).
		- rsync will do its best, but you run a risk of sending a file where the changes are not atomic.
	- Works on Linux/Mac/Windows/pretty much anything
		- Pre-installed - or at least available via the package manager - on all Linuxes
		- Installed on mac
		- Available for Windows via cygwin https://www.cygwin.com/
	- https://rsync.samba.org/
		- One major advantage; if you're sending a file that has an old version on the target system, rsync will only send the changes need to bring the old file current, potentially saving a lot of bandwidth.


======== Before you start
- Set up ssh keys so you can ssh from M1 to M2
	- Once keys are created, run "quick-ssh-keyinstall M2" ( http://www.stearns.org/ssh-keyinstall/quick-ssh-keyinstall )
	- All these instrustions assume that this is getting run from M1 (command line or cron) and M1 can ssh to M2
	- The account on M1 needs read privileges on the files it will be transferring
	- The account on M2 needs write privileges on the target directory and the underlying files if any.
- These should _not_ be between root accounts, both accounts should be non-root
	- If you allow ssh between 2 root accounts, you've destroyed any security between the 2 systems.


======== To push files from M1 to M2
rsync -a -e ssh /top/level/directory/ user@m2s_hostname:/remote/top/level/directory/
	- Both directories should end in a "/" - that tells rsync that these should look identical
		- If you're only looking to send a single file, end both paths with that filename instead of a slash.
	- The ":" in the target indicates that this is the remote system.
	- Note that this is a _one-way_ push; files on M1 are sent to M2; nothing on M2 is carried back
		- To get true synchronized directories, look at unison
		- Think of the first directory as the master copy, and you want the second directory to look like it
	- "-a" - preserve ownership, permissions, etc.
		- rsync is smart enough to handle the case where a user account is under 2 different pids on the two systems
	- "-e ssh": rsync does all its work over an ssh connection.  Leave off if you're copying 2 trees on this system and not interacting with other machines.
	- "user@" is optional and only needed if you're ssh'ing to a different account on M2
	- the entire "user@m2s_hostname" block is anything you can ssh to, including any label on a "host" line in ~/.ssh/config .


======== To pull files from M2 to M1
rsync -a -e ssh [user@]m2s_hostname:/remote/top/level/directory/ /top/level/directory/
	- Same notes as above


======== Parameters to add if needed
"-v"		verbose and list all files transferred
"--progress"	Show progress, useful if invoked by hand, not a good choice from cron
"--bwlimit NNN"	Restrict the total bandwidth to NNN kilobytes/second.  Good if you want to reserve the majority of the bandwidth to other tasks
			Without this, run at full speed
"--delete"	Remove unneeded files from the destination (M2).  Any files in the target tree that don't exist in the source will be deleted.  No effect if the target directory is empty.
			Without "--delete", the tree from M1 will be sent over, and will be mixed with any files that were already on M2 but not on M1.
"-z"		Compress all data sent/received.  Leave this off if all the files being sent are already compressed or encrypted, or if your ssh connection is forced to compress (~/.ssh/config stanza contains "Compression yes" - discouraged).
			Without this, data will be sent in uncompressed form, increasing bandwidth needed but reducing CPU load.
"--temp-dir=/remote/top/level/tempdir/ --delay-updates"
		These two options force the new files to show up in ..../tempdir/ while rsync is running, and only move the files into their final directory at the end.


======== Notes
	Also, if you want to make certain that rsync has essentially no impact on CPU or disk bandwidth on either system, run it under nice and ionice:
nice -n 19 ionice -c 3 rsync --rsync-path="nice -n 19 ionice -c 3 rsync" {normal rsync parameters from above}
	Obviously this will slow the transfer between the systems as rsync has to wait until there's both free processor time and free disk bandwidth.


	This can be run out of cron as long as there's a passphrase-less ssh keypair set up between M1 and M2 (private key in M1 ~/.ssh/ , public key copied to M2 ~/.ssh/authorized_keys)
	Setting this up to use a key stored in an ssh-agent is possible, but tricker; contact me if you want to use this approach.

	To the best of my knowledge, rsync cannot directly copy from one _remote_ computer to another _remote_ computer, but you can do 2 copies (remote1->me, me->remote2)

	If the act of ssh'ing from M1 to M2 requires ssh'ing to a gateway system first then ssh'ing to M2, set up the connection to M2 in ~/.ssh/config like the following and rsync will be able to "directly" access M2:

Host M2
	Hostname		M2s.ip.address
	Port			22
	ProxyCommand		nohup ssh gateway.system.ip.address nc %h %p
	HostKeyAlias		M2