--- title: "File Transfer" canonical: "https://kb.uconn.edu/space/SH/26033783688/File%20Transfer" format: markdown --- This page provides instructions on how to use various programs to copy files to and from the HPC Cluster using Secure Copy (SCP), which transfers files between computers using the Secure Shell (SSH) protocol. In order to use any of these methods to transfer files, you must either be on the Storrs campus or using the UConn VPN. See [this page](https://wiki.hpc.uconn.edu/index.php/HPC_Getting_Started#Off-campus_Access) for VPN instructions. ## FileZilla FileZilla is a graphical file transfer utility for all operating systems. [Download](https://filezilla-project.org/download.php?platform=win64) and run Filezilla Client from their webpage. or from Sourceforge: [https://sourceforge.net/projects/filezilla/](https://sourceforge.net/projects/filezilla/) 1. In the top menu, click on *File > Site Manager* or with the shortcut key stroke *Ctrl + s *to open the Site Manager window. 2. Click on "New Site" and call it, say, "HPC". Fill in/select the following values: 1. Host: *hpc2.storrs.hpc.uconn.edu* 2. Protocol: SFTP - SSH File Transfer Protocol 3. Port: 22 4. Logon Type: Interactive 5. User: ![image-20240813-183508.png](media://7dc30934-6ac5-4430-8870-e72ba0afbe98) 3. Under Transfer Settings 1. Check “Limit number of simultaneous connections. Set max to “1”. ![image-20240813-183825.png](media://f930b2e9-965e-49da-be34-c448920d3215) Setting the simultaneous connections option to 1 will tell FileZilla to only allow 1 connection to be established for transfers instead of opening a connection/authentication for each file transfer. 4. When Filezilla connects, you will be prompted for your password and DUO authentication. 5. Once connection is established, you will see the remote site in the right pannel. 6. To transfer files, you can drag and drop between the left (local) and right (remote) computers. 7. The next time you start FileZilla, you can quickly login with the Site Manager button. 8. It is also recommended to disable to default timeout setting within FileZilla under **Edit > Settings > Connection **and set the value from 20 to 0 ![FileZillaTimeOutDisable.png](media://6874eff1-6f95-4910-8bb0-75df16069f7e) Click OK to confirm the change and restart FileZilla > ⚠️ If the connection to [hpc2.storrs.hpc.uconn.edu](http://hpc2.storrs.hpc.uconn.edu) hangs or does not connect, specifying one of the login node(s) directly can be used as a workaround. > ⚠️ > ⚠️ The “Host” field will be one of the following: > ⚠️ > ⚠️ Login4: > ⚠️ > ⚠️ [login4.storrs.hpc.uconn.edu](http://login4.storrs.hpc.uconn.edu) > ⚠️ > ⚠️ Login5: > ⚠️ > ⚠️ [login5.storrs.hpc.uconn.edu](http://login5.storrs.hpc.uconn.edu) > ⚠️ > ⚠️ Login6: > ⚠️ > ⚠️ [login6.storrs.hpc.uconn.edu](http://login6.storrs.hpc.uconn.edu) ## SCP (Mac OS or GNU/Linux) To transfer large files between 2 computers, it is useful to directly transfer between those computers instead of copying to your local computer. One can use `scp` on the cluster to copy files. `scp` is a command line utility bundled with UNIX operating systems. For example, to upload file(s): ``` scp @hpc2.storrs.hpc.uconn.edu:/path/to/remote/destination/directory ``` To download a directory: ``` scp -r @hpc2.storrs.hpc.uconn.edu:/path/to/remote/source/directory . # The dot means current directory; -r means recursively copy everything in the directory ``` ## WinSCP (Windows only) [WinSCP](http://winscp.net/). Use the same connection settings as FileZilla. ## Using SSH Keys To Bypass DUO [https://uconn.atlassian.net/wiki/spaces/SH/pages/27104772140](https://uconn.atlassian.net/wiki/spaces/SH/pages/27104772140) ## Transferring your data You can transfer data to your local machine in two ways. 1. The slow way uses the programs listed above. This is most suitable for small transfers. 2. The fast way uses the *Globus service*. Globus is about two to five times faster, depending on system traffic, and it can reach a transfer speed of about 50MB per second. It should be used for large transfers. A NOTE ABOUT GLOBUS: Globus does more than transfer data between the Storrs HPC cluster and local systems - it can transfer data within a network of facilities. Globus comprises large network of *endpoints* that span the US, and it transfers data rapidly between any two endpoints. You [read about Globus here](https://kb.uconn.edu/space/SH/26033653003). ## Preparing Your Data For Transfer ***If your data contains many small files***, then you should **tar** your files up into one or more **tarballs** and store the tarballs. Although it involves an extra step, ***this will make it faster and easier to transfer your files***, because the system can handle the transfer of large tarballs much more easily than the transfer of many small files. ***It will also make more efficient use of space***. Here's an example of using tar. Suppose your data is in 3 directories. You may find it convenient to create a tarball for each directory, as show in this example: ``` # List directory % ls -l drwxr-xr-x 5 aaa0000 Domain_Users 4096 Jun 22 2018 data1 drwxr-xr-x 5 aaa0000 Domain_Users 4096 Jun 22 2018 data2 drwxr-xr-x 5 aaa0000 Domain_Users 4096 Jun 22 2018 data2 # Make tarballs % tar cf data1.tar data1 % tar cf data2.tar data2 % tar cf data3.tar data3 ``` You can then transfer the files. To recover your original directories: ``` # Unpack tarball % tar xf data1.tar % tar xf data2.tar % tar xf data3.tar ``` You can also compress the tarballs to decrease size and make transferring them faster. ``` # List directory % ls -l drwxr-xr-x 5 aaa0000 Domain_Users 4096 Jun 22 2018 data1 # Make compressed tarballs % tar czf data1.tgz data1 ``` You can then transfer the files. To recover your original directories: ``` # Unpack compressed tarball % tar xzf data1.tgz ``` ## Moving large files If your data is mostly in large files, then you may want to copy your data directly, and not as tarballs. ## Transfering Files Using Globus See the page [Globus Connect](https://kb.uconn.edu/space/SH/26033653003) for instructions on how to use Globus