Cluster REDBAD

Use Docker container rubigdata/redbad to issue commands on our brand new educational cluster REDBAD.


The cluster consists of 2 main nodes (redbad01 and redbad02) and 11 workers (rbdata01 - rbdata11). You need to connect to the VPN provided by the Science Faculty to access those machines (or, alternatively, run the container on a computer in the Huygens building; additional instructions specifically for Huygens are provided at the bottom of this page). Setting up VPN is preferable though, because then you can access all the Web UIs without setting up ssh tunnels for every different UI).

House Rules

The cluster has not been secured, so you are expected to behave properly.

Do not impersonate a different student!

We have used the github usernames of students who accepted the final project to create user directories on the cluster.

Yarn has been setup with three different queues. For initial trial runs, use the smaller default queue. When these run fine, you think the code is ready for larger scale data processing and you’d only need additional resources to succeed, you can submit jobs to the silver queue.

Please use gold only for code that has been tested thorougly, and is intended as a (semi-)final run that requires quite a few resources.

Getting Started

Set the environment variable GITHUB_USERNAME to the github accountname you used throughout the course, i.e., replace the XXXXX here:


Create and start a container to work on the cluster:

docker create --name redbad -e HADOOP_USER_NAME=${GITHUB_USERNAME} -it rubigdata/redbad
docker start redbad

Attach to the container for shell access to the cluster:

docker attach redbad

Using HDFS

When you issue HDFS commands in the rubigdata/redbad container, they are directed to the cluster. Take a look at the directories that already exist on the cluster:

hdfs dfs -ls /

HDFS has been configured in High Availability mode, where one of redbad01 and redbad02 is active, and the other one standby. Depending on the namenode that is the active one (which would switch upon failure), you would access the namenode’s WebUI at or (these links require Science VPN, setup an ssh tunnel to localhost otherwise).

Curious which namenode has which role? It’s listed in the Namenode’s Web UI, but you can also find it using the HDFS CLI:

hdfs haadmin -getAllServiceState

(Apart from running Hadoop in high availability mode, the cluster is also prepared for a so-called federated setup, with an educational part identified by gelre and a research part identified by frisia. However, only the educational cluster is operational at this point of time, so you may forget this immediately.)

Distributed Filesystem Configuration

File and directory names can be given with or without prefixing an explicit cluster-identifier (gelre for the educational cluster) or namenode (redbad01 or redbad02, depending upon which is active and which is standby); in other words, /, hdfs:///, hdfs://gelre/ and hdfs://redbad01:8020/ or hdfs://redbad02:8020/ all refer to the same root directory of the HDFS filesystem on REDBAD.

Let’s continue our discussion of the listing of the root directory at the cluster. /app-logs and /tmp are system directories that end users can safely ignore. The home directories are important though; these reside under /user, issuing hdfs dfs -ls or hdfs dfs -ls . lists the contents of your (empty) homedir, /user/XXXXX.

Upload the test data for the example standalone Spark program to your home directory:

hdfs dfs -put rubigdata-test.txt /user/${HADOOP_USER_NAME}

Check that it has been added correctly:

hdfs dfs -ls /user/${HADOOP_USER_NAME}

Crawl segment

Directory /cc-single-warc-segment/ is the main target in Part III. It contains a “small” part of one of the monthly crawls. When we ran our first commands to test the cluster setup, we copied this segment onto the cluster using the hadoop distcp command.

A single crawl segment is more than enough data to test your program created in part I & II on its scalability:

[hadoop@77316fead95b rubigdata]$ hdfs dfs -du -h -s hdfs:///single-warc-segment
727.4 G  1.1 T  hdfs:///single-warc-segment

The first number is the size of the data in HDFS, i.e., what you notice as a user when you read the data. The second number is the size on raw disk, including its replicas. (The second number is lower than 3x the first number, because we stored this data using the new Hadoop 3 feature of erasure coding, a different way to provide fault tolerance that trades off disk space consumption for CPU and network costs while reading the data.)


Directory /cc-index-subset contains the CC-INDEX files in Parquet format, that you can query to identify interesting subsets. You will find more information about the CC-INDEX in the Commoncrawl documentation, and we provide a few examples using the CC-INDEX to get you started using this resource (feel free to skip).


As soon as the cluster could execute map-reduce jobs (about a week ago), we have been running distributed copy jobs to transfer a copy of the CC-MAIN-2021-17 (april 2021) crawl to REDBAD. At the time of writing, the map-reduce task to copy the crawl has reached about 80% of completion; you can check if it is still running using yarn app -list or yarn top.

Copying the data is time-consuming, but so is processing all that data, especially since our cluster has not that many nodes. Let’s focus on a single segment for now, unless you have very good reasons to plough through even more than 0.8 TB of crawl data.

If you think you really do all that Crawl data, contact us (in the Matrix room) first.

Using Spark

Now that we revisited usage of HDFS (introduced in assignment two), let’s move on to running Spark programs on the cluster. Take a simple program that you know is correct, for example, compile and package the letter-counting Spark program that we started with in Part II.

Before you proceed: edit the example program to read test file rubigdata-test.txt from the correct location in HDFS, i.e., hdfs:///user/XXXXX/rubigdata-test.txt.

Then create the Fat Jar for submitting the code to Spark:

sbt assembly

(You will probably recall from part II that the first run of sbt takes much longer than subsequent runs… I found that this step benefits a lot from temporarily turning off VPN. Upon [success], do not forget to turn VPN back on, or your machine cannot find the REDBAD nodes.)

Finally, you are ready to submit code as a Spark job to the cluster!

Spark jobs are submitted using Yarn in deploy mode cluster. (Our tiny container is not really suited to act as a true node in the cluster, so we should not use Yarn client mode.) Apart from deploy mode, also specify the desired queue (one of default, silver or gold, refer to the House Rules before deviating from default); and, issue spark-submit:

spark-submit --deploy-mode cluster --queue default target/scala-2.12/RUBigDataApp-assembly-1.0.jar

If things work out fine, you will now see quite a few messages appear, first some basic log info. After these initial (ten plus) debug lines, Yarn will indicate that it uploads your Fat Jar to the cluster to be deployed as a Spark job.

As the following step, Yarn will indicate that it proceeds to submit the application and notifies you of the application identifier; you need that identifier for inspecting status and output. The application will start with state: ACCEPTED until Yarn can allocate the required resources to run the job, when its status will move to state: RUNNING.

The cluster has been setup with a so-called job history server that aggregates the logs of an application (i.e., it collects the logs from the different worker nodes and puts a merged copy on HDFS). We also configured the Spark history server, that provides user friendly access via a Web UI for running (incomplete) and finished (completed) applications.

You can inspect logs of applications using the Yarn CLI (use a docker exec -it redbad /bin/bash command in a different terminal while your other container is still busy reporting debug information), e.g., assuming application identifier application_1623272363921_0008 you would request:

yarn logs \
  -applicationId application_1623272363921_0008 \
  -log_files stdout

More conventiently than using these CLI commands, you can also find your application in the Spark History Server that runs on rbdata01. Its Web UI is found at Click on your application’s identifier to view Spark’s job information. Under the Executors tab, find the line corresponding to the driver node to locate the output of our println commands (these print on the stdout of the driver node), and you see something similar to this screenshot from my browser window:

Screenshot Logs

If all of this worked fine, you should copy the code for your project from parts I and II into the new redbad container, and try to process crawl data read from HDFS!

Using the Huygens machines

If you cannot get VPN to work from home, a solution is to start the docker container on machines in the Huygens building.

Things may actually also work correctly by using your own machine connected to internet via eduroam; I have not yet had the chance to test this.

Docker is available for our use on the machines in HG 03.761 hg761pcXX (replace XX by the machine number). You can ssh into machines in those rooms, as long as they were booted in Linux. Hereto, you have start an ssh client from the Linux Login server, Login on lilo via ssh; or, if you’re on Windows but did not install WSL(2), you’re probably better off using Putty instead.

Students in the course should be able to run Docker when logged in onto one of these machines, assuming you were formally enrolled when CNCZ updated the group earlier this semester; if docker ps gives errors and the output of groups does not include docker, let me know and I will add you (I need your science accountname to do so).

If you need help, do not hesitate to find us in the Matrix room for assistance.

Back to assignments overview / part I / part II