Lotterhos Lab vs. Personal Storage on Discovery and GitHub

Edit this page

  • We have active storage (10 TB) that should be used for ‘active’ projects, and archival storage for storing raw files and outputs from projects. Please be smart about how you use archival storage and work with Dr. Lotterhos to make sure only relevant files are being archived.

  • In addition each user has a home directory and a scratch directory.

Storage unix tricks

df -h /work/lotterhos #total space useage for lotterhos

pwd | du -h --max-depth=1 get disk use for folders in the current directory

ls -lah view owners of files and file sizes (the total size of the folders is not displayed)

Working with github:

In Github Settings, go down to “Developer settings” and create a personal access token:

image

Make sure it is “Classic” and then “configure SSO”.

More information here:

Authorizing personal access token

Then in the command line in Discovery use your token when it asks for a password.

It is good practice to create a repo on Github and then clone it to Discovery to start a project. You can follow instructions in Buffalo’s Bioinformatics book to work with git on linux.

In addition, you may want to collaborate on Discovery through your git repo.

You may also need to set up ssh keys:

example: When person A started working on his git branch in Discovery, he cloned that branch to Discovery (e.g in the folder and repo /work/lotterhos/MVP-NonClinalAF). To give person B access,

1. Person-B Go to the terminal and copy your public key file content.
cat ~/.ssh/id_rsa.pub
(If you don't have a key pair generated already, generate a new one using command "ssh-keygen" )
2. Go to the github website and login with Person-B credentials.
3. In the upper-right corner of the github page, click Person-B profile photo, then click Settings.
4. Click New SSH key or Add SSH key.
5. Paste your key into the "Key" field.
6. Click Add SSH key.
7. Test the connection by running the following command in the terminal on person B Discovery:
ssh -T git@github.com

Giving someone access to a folder and all containing folders

General code:

setfacl -R -m u:username:rwx myfolder

Example for user rumberger.c in folder MVP_Genotypes_DataMerge:

setfacl -R -m u:rumberger.c:rwx MVP_Genotypes_DataMerge

Your home directory on Discovery**.

  • Path: /~/ or /home/username

  • Note that only you have access to these files.

Your scratch directory on Discovery**. Use this for intermediate outputs that are not backed up.

  • Path: /scratch/username

  • Note that only you have access to these files.

Lotterhos shared storage on Discovery

  • /work/lotterhos/

  • Katie will create a folder for each lab project in this directory. For each folder, Katie will submit a ticket to research computing to ask them to create a group for the project with members who will have access to that folder.

Example:

mkdir BlackSeaBass                  # make the folder
chgrp -R lotterhos-bsb 2020_NOAA_BlackSeaBass_ddRADb    # change the group on the directory. I forget what the -R does
chmod -R 775 2020_NOAA_BlackSeaBass_ddRADb            # give group permissions to the directory
ls -la 2020_NOAA_BlackSeaBass_ddRADb # check permissions

  • Groups

    • lotterhos - all Lotterhos Lab members

    • lotterhos-bsb - access to Black Sea Bass project

    • lotterhos-cod

  • Groups are created by research computing

Find the groups of a user: groups tbittar will give the groups for user tbittar

Find the users in a group: users lotterhos-bsb lotterhos will give the users for group lotterhos-bsb and group lotterhos

usermod -a -G groupname username I forget what this does

Sometimes I have still had trouble making the code work on existing directories, but it can work if you copy the directory to a new one

Archival Lotterhos Lab storage** (25 TB for lab). Northeastern NESE collection.

  • Path: /lotterhosarchive/

  • Access through GLOBUS endpoint that Katie can Share with you

Outdated storage

Active Lotterhos Lab storage: more information** (? TB for lab).

  • smb://nunet.neu.edu/rc-shares/lotterhos/

  • Ask Katie to share a Globus endpoint with you. The endpoint with include the words “smb Volumes/lotterhos”

  • Within this storage space we have a shared folder for each project, that only the people on that project have access to.

  • For this endpoint to work, it has to be mapped through Katie’s laptop or the iMac in the lab. Katie’s laptop from home: sign into NU VPN, make sure Globus Connect Personal App is running, plug in ethernet cord

  • Because accessing this server is done through a computer, it makes transferring files specific to the computer that the server is mapped to. Typically, you will ask Katie to share the Globus endpoint for this server that is set up through the lab iMac. As long as the server is mapped to the computer (typically lab iMac) and that connection is not interrupted, you should have no problems.

  • From a laptop or over wifi, transfers of large files are very slow. This is why we map to the server through the iMac. The point of all the steps we outline below is so we can use Globus to transfer files from the smb drive to Discovery.

  • To connect to the smb server, follow these directions to map to the server on a computer this document

(Typically Katie only)

  • Setting up a Globus endpoint for the folder to share (typically Katie only)

  • Intitial step if Globus personal endpoint for the computer is not set up: Download and install the Globus Connect Personal App. In Globus web app, click on “Add a personal endpoint” and enter the key in the Desktop app.

  • Intial setup if working from a home laptop: Use GlobalProtect to establish a VPN connection (vpn.northeastern.edu) (if you are having trouble establishing the VPN connection on a Mac, check your Security and Permissions settings and give the app permissions, see image). Then you should be able to map to the server.

  • Step 1: In the Globus Desktop App, then go to Preferences-> Access and click the “+” sign. On a Mac, at the top center click on the bar, click on the name of the computer, and you should see the mapped server there as a volume Volumes/lotterhos. Click on it and add it to the list.

  • Step 2: In the Globus Web App, go to the endpoint for the computer. Then type Volumes/lotterhos/ in the path. Save it as a bookmark.

  • Step 3: Before you can conduct file transfers from a personal endpoint, you need to have a “Globus Plus Account”. Email James Macalino at NU research computing to get added. You will then receive an email.