RU Big Data 2023 (NWI-IBC036) FAQ

Cluster Admin Instruction for Students

Students on REDBAD

You receive the key to use for access to the cluster via email, follow these instructions to use it and login to the cluster under your personal account.

Master node

IP address of the master (this value may change, announcement in Matrix room if it does!).

export REDBADMASTER=18.232.146.133

First login

Students can then login as themselves.

ssh -i id_redbad_${USER} ${USER}@${REDBADMASTER}

The passphrase for the key is announced in the Matrix room and Brightspace; your student number is the username.

(You might setup ssh-agent and ssh-add for convenience in handling the passphrase.)

You will run into the following ASCII art upon a successful login:

Last login: Tue Nov  3 19:48:15 2020

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
                                                                    
EEEEEEEEEEEEEEEEEEEE MMMMMMMM           MMMMMMMM RRRRRRRRRRRRRRR    
E::::::::::::::::::E M:::::::M         M:::::::M R::::::::::::::R   
EE:::::EEEEEEEEE:::E M::::::::M       M::::::::M R:::::RRRRRR:::::R 
  E::::E       EEEEE M:::::::::M     M:::::::::M RR::::R      R::::R
  E::::E             M::::::M:::M   M:::M::::::M   R:::R      R::::R
  E:::::EEEEEEEEEE   M:::::M M:::M M:::M M:::::M   R:::RRRRRR:::::R 
  E::::::::::::::E   M:::::M  M:::M:::M  M:::::M   R:::::::::::RR   
  E:::::EEEEEEEEEE   M:::::M   M:::::M   M:::::M   R:::RRRRRR::::R  
  E::::E             M:::::M    M:::M    M:::::M   R:::R      R::::R
  E::::E       EEEEE M:::::M     MMM     M:::::M   R:::R      R::::R
EE:::::EEEEEEEE::::E M:::::M             M:::::M   R:::R      R::::R
E::::::::::::::::::E M:::::M             M:::::M RR::::R      R::::R
EEEEEEEEEEEEEEEEEEEE MMMMMMM             MMMMMMM RRRRRRR      RRRRRR
                                                                    
[hadoop@ip-10-192-10-241 ~]$

Subsequent logins

Exit the cluster to complete the setup process.

Download the SSH config file template.

Edit the file according to the instructions inside, and:

mv redbad-ssh-config-template redbad-ssh-config
chmod 600 redbad-ssh-config

From now on, use this command to login to the REDBAD master:

ssh -F redbad-ssh-config redbad

Only works if you replaced the template text with the right values.

Next, read about CommonCrawl and the REDBAD cluster for more details:

Unfortunately, we had to move the Zeppelin notebooks away from REDBAD - at least for now - and use the cluster only for running stand-alone applications.

Updated instructions in CommonCrawl on REDBAD.

See also

If you are on Windows, you either use WSL(2) and the same config file as above, or you setup PuTTY to forward ports 18080 (Spark History Server) and 8890 (Zeppelin).

Back to the cluster page.