Game Jam

Game Jam

Game Jam

2020 has been a wild ride to say the least. January alone started with a million Hong Kong protesters on a New Year’s Day march, the Australian bush catching fire(ironic considering UN declared it the International Year of Plant Health), the assassination of an Iranian major general by a US drone, prompting speculations about a new war, and if that wasn’t enough, COVID-19 came along, as of now with 28.8 million infections, 900k+ dead, mass lockdowns, and the largest economic recession since the Great Depression. Honestly, so much bad stuff has happened this year that just writing it would probably trigger someone’s PTSD. Instead I leave you with this only slightly depressing recap & video. After all, images are worth a thousand words.

Maybe prompted by this confusion at the state of the world, I have been thinking a lot about philosophy these days. Out of the many people to write on the subject, there is one that feels somewhat fitting for our situation, Arthur Schopenhauer.

A notoriously pessimistic philosopher, Schopenhauer formed his system of thought from the teachings of Plato, Kant and the Indian Upanishads. He drew from the first two the idea that what we experience as the world is not actual reality and supplemented it with Indian concept of Maya, a veil that hides the true nature of the world from us. His idea of the world as will and representation is a metaphysical system that considers Plato’s “Ideas” and Kant’s “things-in-themselves” to be the same thing, an underlying structure to existence that we cannot see directly, but rather only their representations. But his idea of the will took it one step further. He said there was only one will, an undivided entity that drove all things in the universe through ceaseless striving towards its goal. The laws of physics, the movements of animals and the consciousness of human beings are all the same will manifesting itself in different gradations of development. It drives everything from gravitational to sexual attraction. You can see why this metaphysical system appealed to many thinkers that came after him, such as the philosophers Wittgenstein and Nietzsche, the scientists Schrodinger and Einstein, and the psycho-analysts Freud and Jung.

He was also strongly influenced by Buddhist thought, claiming that all human suffering came from attachment to desires. To Schopenhauer, desire was pain that persisted until it was satisfied. Satisfaction itself was simply the neutral lack of desire as opposed to a positive value such as joy. In essence, Schopenhauer thought that life spent in pursuit of desire was constant pain, with only short periods of boredom as reprieve, a negative-sum game. Another striking idea of his is that we live in the worst possible sustainable world. To Schopenhauer, the world we live in could be worse, but if that was the case, it would be so unstable it would fall into chaos. In the context of 2020, I think we can understand what he meant.

His ideas on aesthetics led him to be called the artist’s philosopher because they were a source of inspiration to many composers and authors. In Schopenhauer’s philosophy, the only tenable way to achieve happiness is through asceticism. Only by renouncing worldly desires and trying to understand the will itself can you transcend the cycle of pain and boredom. One of the ways he suggested it was through art. He considered art an effort to capture and express the true essence of its subject, bypassing representation and getting closer to the will. He even said music was will itself, drawing parallels between the strivings of desire and the motions of a melody.

In the end though, he was a bitter man, with strained relationships in his familial, social and professional circles. His relationship with his mother, who in his lifetime was a more successful writer and was usually in the company of admired contemporaries such as J. W. von Goethe, whom Schopenhauer considered a hero, was not the best. Even when he managed to gain the admiration of Goethe, he almost immediately ruined it by publishing a treatise on color theory, the same subject that they discussed with each other, that differed in many points to Goethe’s. That, along with his headstrong and self-assured attitude, led Goethe to distance himself from Schopenhauer. Even in academics, he wasn’t very popular. At one point, when he was appointed to an academic position in the University of Berlin, he scheduled his lectures to be at the same time as his intellectual nemesis, G. W. F. Hegel. Almost no one attended and he left the post two years later. His mother said it best in a letter she sent him

“You are unbearable and burdensome, and very hard to live with; all your good qualities are overshadowed by your conceit, and made useless to the world simply because you cannot restrain your propensity to pick holes in other people.”

As compelling as his pessimism is, especially in 2020, over the past couple of days I was more interested in his metaphysical system of will and representation, specifically how similar it sounded to Object Oriented Programming. OOP is a computer programming model that puts emphasis on data in a program, as opposed to operations. For example, let’s say you wanted to make a car racing game. If you were to make that game using OOP standards, you would create a “class” called Car. A class is essentially an abstract schematic of what you want every car to be. It would describe the data necessary to create any model of car, such as its manufacturer and horsepower, as well as its functions, such as acceleration, steering and braking. After defining a class, all you would need to do is say new Car(), and pass in the necessary information to get a Car object. This structure is useful to avoid redundancy and points of failure in a program, as well as improving readability. But it is also eerily similar to Schopenhauer-ian thought. The class could be the will and all the objects you create from it could be manifestations, or representations, of that will. In a more visual example, consider the two boxes below. The first is the world as representation. It could be a game, the visual rendering of the will, or your perception of reality. The second is the world as will. It is the code of the game or the will of the universe you live in. The circle in the center is the subject and you can “simulate” its will by dragging and releasing it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
class Circle {
    constructor(x,y) {
        this.position = [x,y]
        this.velocity = [0,0]
    }
    update() {
        this.position[0] += this.velocity[0]
        this.position[1] += this.velocity[1]
        if (this.hitsUniverseBoundary())
            this.bounceOff()
    }
}

var universe = []
universe.push(new Circle(200,200))

That circle must be filled with existential dread from being alone in such a small universe. Let’s give him a friend. Notice that they are both subject to the same “will” despite being individual objects with different starting positions. If we make the will dictate how they handle interactions with each other, we will see similar behavior from both.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
 
    class Circle{
        constructor(x,y) {
            this.position = [x,y]
            this.velocity = [0,0]
        }
        update() {
            this.position[0] += this.velocity[0]
            this.position[1] += this.velocity[1]
            if (this.hitsUniverseBoundary())
                this.bounceOff()
            // added this clause
            if (this.hitsOtherCircle()) {
                this.bounceOff()
                this.glow()
            }
        }
    }

    var universe = []
    universe.push(new Circle(100,200))
    universe.push(new Circle(300,200))

Another useful aspect of OOP is inheritance. Inheritance is an idea that since different objects can have many attributes and functions in common, we can create a class that contains them, and limit the classes of the objects themselves to describe only the specific differences, saving us from lots of redundancy. Again, eerily similar to the concept of the will coming in gradations. We can apply both to humans and animals. Humans, for all practical purposes, are animals. We eat, sleep, propagate and react to our environment in ways that let us do so for longer. But we also have the capacity for intelligent thought. In OOP terms, we are an Animal class with the added function of intelligence, and in Schopenhauer-ian thinking, we are a manifestation of the will one level above animal that overrides some basic strivings with more complex behavior that is nonetheless subservient to the will.

In the code below, we have created a hierarchy of will that starts with the class Circle from above, which Animal “inherits” or “extends” from, by adding the properties of lifespan and the function of reproduction, which in turn is extended by the Human class, with the property of “intelligence”, a somewhat weak aversion to being moved by irrationality, in this case dragging and releasing. Humans also have a desire to chase animals (using the same formula as gravitational force) and deal them damage on collision. A bit crude, but it does serve as a good illustration of how intelligence doesn’t necessarily mean defying the will or being above it. It is simply a higher level of the hierarchy. You can even control how humans move by dragging an animal around their vicinity. Don’t get them too close though; you might end up creating a gravitational singularity that sends the universe into utter chaos. Just saying.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
 
    class Animal extends Circle {
        constructor(x, y) {
            super(x,y)
            this.life = 50
        }
        update() {
            super.update()
            if(this.isAtHalfLife())
                this.reproduce()
        }
    }
    class Human extends Animal {
        constructor(x, y) {
            super(x,y)
            this.life = 100
        }
        update() {
            super.update()
            this.chaseAnimals()
            if(this.hitsAnimal())
                this.hurtAnimal()
        }
    }

    var universe = []
    universe.push(new Animal(100,200))
    universe.push(new Human(300,200))

You can see the “worst possible sustainable world” idea here as well. If animals couldn’t propagate, or were killed before they did, they would be wiped out in a minute and we’d be back to the cold, lonely and empty universe. But when we ensure there is always an animal alive in the code(it’s no Fibonacci, but it works), we allow the cycle to go on indefinitely, creating Schopenhauer’s universe.

There are a lot more comparisons I can make, from public and private variables to abstraction and polymorphism. I could probably write a whole essay on the analogues between death and garbage collection, but I digress. A more interesting take is that Schopenhauer isn’t necessarily similar to OOP, but simulation theory in general. Sure enough, a short google search of “Schopenhauer simulation” gives a short quote to the same effect in this really good article. I also found a whole paper called The World as Will and the Matrix as Representation which explores the same ideas through the characters and events of the classic 1999 movie. While it’s not a lot of sources, it’s fun to see other people have made the same connection.

So instead, let’s stick to philosophy and try to make sense of this world. The first and foremost problem is free will. How can an individual have it if will itself is an unchanging entity that lies on a cosmic scale? When you talk, are you saying what you think or what you are told to say? Probably the latter.

Even doing the opposite of what the will tells you is a weak freedom at best, because it is simply another form of subservience. Rejection of the will and existing independently of it are two completely different things. In the words of the great Albert Einstein:

“Man can do what he wills, but cannot will what he wills.”

Or can we? This is just random speculation on my part, but what if will isn’t constant? There is no sufficient reason to say it is. I mean, sure the universe we live in has probably been operating under the same rules since before the Big Bang. But maybe that’s because it hasn’t reached a level of complexity where the will begins to change itself. What if the more we understand the will, be it through art, spirituality or science, the closer we get to changing it? After all, if someone feels a desire to change the will and their desires are decided by that same will, doesn’t that imply a process of self-induced change?

If this theory sounds too idealistic, let’s look at it in programming terms. Can an object change its own source code, its class? Can we make a meta-program that can edit itself? Impossible, right? Well, maybe for some languages, but there are entire families out there with explicit support for it, like Lisp, one of the first programming languages used for artificial intelligence research. Coincidence? I think not.

We don’t even need to go to such an obscure language for meta-programming support. Right here in your browser, we have JavaScript. It is the same language that’s running the buttons and the simulations. And it has two properties which make it very easy to simulate meta-programming, the eval function and prototype object. eval lets you run any text you want as code and prototype is basically the class itself laid bare for you to mess around with. Using these two, we can let our humans change the entire universe.

First, let’s get rid of the animals; there has been enough suffering. Note that we leave the class in the code because it is a dependency for human existence. I will leave it up to you to think about the implications. Secondly, let’s simulate total ignorance on the part of humans, at least initially. They will bumble around the world like hapless idiots until they interact with each other. These interactions can be communication, meditation or research, at which point the human’s “mage level” will increase, unlocking secret runes and spells that can twist the world the way they see fit. These spells which will also be triggered randomly when mages bump into each other. In the box below, we have levels 0 to 4, with powers that go from the gory conjuring of blood to the phantasmagorical changing of the sky. Be warned, the arcane magic they discover could trigger epilepsy.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
 
    var spells = [
        "spewBlood()",
        "Mage.prototype.size = randomNumber(20,40)",
        "Mage.prototype.color = randomColor()",
        "Universe.prototype.gravity = !Universe.prototype.gravity",
        "Universe.prototype.background = randomColor()"
    ]
    class Mage extends Human {
        constructor(x,y) {
            super(x,y)
            this.mageLevel = 0
        }
        update() {
            if(this.hitMage()) {
                if(this.mageLevel == 4 and Math.random() > 0.3)
                    this.mageLevel++
                else if (Math.random() > 0.3)
                    eval(spells[floor(random(this.mageLevel))])
            }
        }
    }

Isn’t that amazing? This is where Schopenhauer and I part ways. Far from being a pessimist, I believe that we are destined for great things. We will will the will (say that three times fast) through our desire to better ourselves, and in doing so, change this negative-sum world into a positive one. If Schopenhauer bumped into you on the street, he would check his pockets, then try to dissuade you from suing. I would apologize and give you a wink.

How would you like to be a Level 5 Mage?

In the last article, Buzz showed us the basic ideas of calculus and derivatives. In this article, we will use those concepts to make a perceptron.

Before we go ahead and start making one, it’s useful to clarify what a perceptron (and a neural network in general) is. It is an object vaguely inspired by the biological neuron that takes in an input and maps it to a certain output, basically a function. In general, there are two types of perceptrons, ones that output continuous values (called linear regression models) and ones that output discrete values (logistic regression models). Let’s start with the former.

In a linear regression model, we try to optimize a perceptron into making accurate predictions based on a given dataset. The simplest and most common function this perceptron can be is a simple line, ( output = weight cdot input + bias ), weight and bias being the slope and y-intercept. Since we don’t know what weight and bias to use, we set them to random numbers. In the illustration below, you can see what a randomly initialized perceptron acts like and how close it comes to describing the real data, in this case a straight line. The blue lines show the amount of deviation from the correct answer. We call them the cost, loss or error of the perceptron. A convenient method to calculate cost is as the square of the difference between the output and the desired output. It’s convenient because the derivative is easy to calculate.

1
2
3
4
5
6
7
8
9
10
11
    var perceptron = [random(-1,1),random(-1,1)]
    function calculatePerceptron(x) {
        return perceptron[0]*x + perceptron[1]
    }
    function cost(x,y) {
        return (calculatePerceptron(x) - y)**2
    }
    function cost_deriv(x,y) {
        
        return 2 * (calculatePerceptron(x) - y)
    }

In the spirit of this article, let's suppose our data is a map of stars that Buzz Lightyear has to fly to on a mission. He knows the locations of the first few stars in his itinerary, but after that he's on his own. In this scenario, our data is the stars we already know of and the perceptron is meant to fit them as well as predict the location of the unknown stars. Again, since we don't know what weight and bias give us this behavior, we set them to random values.

Since a randomly initialized perceptron is useless, we need to find a way to change it into one with no loss. In other words, we want to find the weight and bias where the loss is at a minimum. If that sounds familiar, that is because it's gradient descent. If we find the derivative of the cost in terms of the weight and bias, we can optimize the perceptron. If the cost is \({((in \cdot w + b) - out)}^2\), the derivative is \(2 \cdot{((in \cdot w + b) - out)}\) in terms of the bias, same as the function above. But for the weight, we need to consider the chain rule. That means we multiply the previous derivative by the derivative of \(in \cdot w \), giving us \(2 \cdot{((in \cdot w + b) - out)} \cdot in\).

1
2
3
4
5
6
7
8
9
10
11
function update() {
    # variable to sum up the derivatives
    var adj = [0,0]
    for (var i = 0; i < points.length; i++) {
        var p = points[i]
        adj[0] -= cost_deriv(p.x, p.y)*p.x
        adj[1] -= cost_deriv(p.x, p.y)
    } 
    perceptron[0] += 0.05*adj[0]
    perceptron[1] += 0.007*adj[1]
}

Now that the perceptron has been optimized, it fits the data we have perfectly and and Buzz can find all the other stars he wants by just extending the line.

Note that since the perceptron is a straight line, it can only get a cost of zero when the data itself is in a straight line. If the data has slight deviations, the cost will settle at a number slightly above zero because that is the best possible fit. And if the data isn’t even a line, let’s say it goes up and down, then a single perceptron is basically worthless, since it can only model one direction. You can see the best fits of different types of data on a single perceptron below.

As you can see, a single perceptron is not expressive enough when it comes to complex data that doesn’t necessarily show linear behavior. In practice, we solve this by using perceptrons in chains and layers to create more complex models, which is what we are going to do in the next and final article. But here, in the spirit of experimentation, we can also change the type of perceptron so that it can fit the data. We can make it a quadratic, cubic or even sine function. And if we get the derivatives right, each one is a valid perceptron that can in some cases outperform a simple linear function.

We can see a lot of intuitive stuff when we mess around with the function and data pairs. One of the first things we notice is that the cubic function can fit easily to all the types of data. It makes sense, because a quadratic expression is just a cubic expression with a zero coefficient. And a linear expression is one with two. If that’s the case, you might be wondering why we don’t prefer them to linear expressions in neural networks. The correct answer is that they tend to overfit and have exploding gradients, both of which will be explained in the next article. The simplified answer is that cubic functions, and higher-polynomials in general, have very sensitive coefficients. A small change affects the graph drastically, making them too unstable for practical purposes. And even when we find a way to make them stable, it comes at the cost of speed and efficiency, which have a lot of importance when it comes to practical machine learning, since it’s not uncommon for a neural network to have thousands, millions or billions of perceptrons.

Another less obvious thing we notice is that the sine function sometimes has trouble fitting to quadratic data, remaining a horizontal line. This is because the perceptron has found a local minimum instead of the absolute minimum. What it means is that sometimes your perceptron finds the best weights if you initialize it a certain way and misses them in others, which also comes into play when you make large networks.

All of this applies to logistic regression as well. But while linear regression makes perceptrons predict data, logistic regression makes them classify it. For example, if the galaxy is either being invaded or experiencing a pandemic, Buzz doesn’t need to know where the spaceships or planets are, he needs to know what they are when he sees them. In this scenario, we have a dataset of green(safe) and orange(dangerous) points, the few places we know the status of, and Buzz wants to figure out which group a newly added point belongs to.

So how do we make a perceptron output discrete values? We can't. But we can do the next best thing, we can make it output probabilities on those discrete values. Linear perceptrons are not very useful to decide probabilities, because it makes very little sense to say 1000% green or -12% orange. We need it to output something from 0% to 100%. This is where the sigmoid activation function comes in. An activation function is a sort of wrapper we put over the perceptron to make its output convenient to use. Sigmoid uses the formula \( \dfrac{1}{1+e^{-x}} \) to squash a linear equation between 0 and 1. Now 1000 maps to 100% and -12 maps to 0%. Sigmoid also has a convenient derivative, \( sigmoid(x) \cdot (1-sigmoid(x)) \).

When we use logistic regression, it's common to change the cost function too. While we can technically still use the difference squared, it gives poor results because the maximum possible cost is only 1, making gradient descent work slowly. We use the negative log functions, \(-log({1 - output})\) and \(-log({output})\), depending on whether we want 0 or 1, to give us a more exaggerated cost, as you can see below.

Combining the activation function and cost function with a simple linear perceptron gives us a more complex derivative because of all the nesting. But if we follow the chain rule, it comes together easily enough. The cost function is \(log(sig(perceptron))\), so the derivative will be \(logDeriv(sig(perceptron)) \cdot sigDeriv(perceptron) \cdot perceptronDeriv \). Here it is in code form, finding a line to separate the green and orange points.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
function sigmoid(x) {
    return 1/(1+Math.exp(-x))
}
function sigmoid_deriv(x) {
    return sigmoid(x)*(1-sigmoid(x))
}
function calculatePerceptron(data) {
    return perceptron[0]*data.x + perceptron[1]*data.y + perceptron[2])
}
function cost(data) {
    var x = sigmoid(calculatePerceptron(data))

    if (data.class == 1) {
            return -Math.log(x)
    }
    return -Math.log(1-x)
}
function cost_deriv(data, y) {
    var x = sigmoid(calculatePerceptron(data))

    if (data.class == 1) {
            return -1/x
    }
    return 1/(1-x) 
    // not negative because the inner derivative cancels it out
}
function update() {
    // variable to sum up the derivatives
    var adj = [0,0,0]
    for (var i = 0; i < points.length; i++) {
        var p = points[i]
        adj[0] -= cost_deriv(p) * sigmoid_deriv(calculatePerceptron(p)) * p.x
        adj[1] -= cost_deriv(p) * sigmoid_deriv(calculatePerceptron(p)) * p.y
        adj[2] -= cost_deriv(p) * sigmoid_deriv(calculatePerceptron(p))
    } 
    perceptron[0] += 0.3*adj[0]
    perceptron[1] += 0.3*adj[1]
    perceptron[2] += 0.3*adj[2]
}

Now that we have the ideal perceptron, all we need to do to find the class of a data point is to check the ouptut of the sigmoid. If it’s 0.5, then the point lies right on the line and can’t be said to be one or the other. If it is less than 0.5, we say it is closer to the 0 class and if it is greater we say it is in the 1 class. We can visualize this by trying the perceptron on the whole plane and seeing how the output changes.

Now Buzz can tell whether any section of space is safe or dangerous based on the little data he has. Let’s see how it works on different datasets. You can change the datasets below to random clusters and nested rings to see how well the perceptron manages to separate them.

Like the linear regression example, sometimes a simple perceptron isn’t complex enough to model data with curves, especially closed curves like circles, which can never be modeled by a line. The cost never reaches a reasonable low either. Again, in practice this is remedied by using multiple layers of simple perceptrons, but for the sake of experimentation, we will see how more exotic perceptrons fare with the same data.

As you can see, more complex functions tend to fit more complex data, but even the most complex functions can’t perfectly separate data that is thoroughly mixed. This is a common problem you will see with neural networks that are given bad or patternless data, they will give back confusing information that contains a lot of false positives and negatives.

If you are wondering why the quadratic function can model circles and ellipses, it is because we are using the the general formula \( a \cdot x^2 + b \cdot x+c \cdot y^2+d \cdot y + e \), which can model those shapes as well as parabolic and hyperbolic curves. Plugging it into a sine function adds another layer of complexity, resulting in graphs like concentric rings and expanding stars.

And that is it for perceptrons. Read the next article to see how we can combine perceptrons to create much more powerful and expressive neural networks.

In the previous article, we learned how to make a single perceptron find a best-fit line, with an introduction to complex models and activation functions. In this article, we will expand those concepts into creating a fully-connected multi-layer neural network by chaining individual perceptrons. This is accomplished by combining gradient descent with backpropagation.

A multi-layer perceptron(MLP) is composed of layers, each of which contains a certain amount of perceptrons. When we input some data, each layer calculates an output based on its perceptrons’ weights and values, but instead of passing those outputs as the final answer, it passes them onto the next layer for more computation until it reaches the final layer. We call this computation of an output a forward pass. The benefit of all this convolution is that it has lots of connections, which means a lot of complexity. As we have already seen, complexity is very useful in a neural network, because it allows us to fit any type of data.

In the simulation below, you can see a visual representation of an MLP. The buttons allow you to choose the number of layers and perceptrons per layer. You can also press any perceptron in the simulation to see how the forward pass goes from there.

Note that the number of perceptrons can change from layer to layer. In fact, some very useful variants of neural network such as autoencoders and GANs depend on this property.

Because there are so many weights, biases, inputs and biases involved in a neural network, it becomes practically impossible to manipulate them as individual values. This is why machine learning libraries (and us, starting now) use arrays, more accurately matrices and tensors, to represent an MLP. For simplicity’s sake, every matrix used in this article will be one dimensional. It is also important to note that all matrix multiplication in the code will be element-wise, meaning

Each perceptron will be a matrix of weights, one for each input (output of the previous layer), and one bias. That way, we can find the output of a perceptron by multiplying in with a matrix of the input data and adding the bias, like so.

In the same vein, we can define a layer as an array of perceptrons and an MLP as an array of layers. By describing them in this way, we significantly reduce the number of variables we have to work with and simplify our manipulations on them (see the eval functions in the accompanying code). Now we can calculate perceptrons and layers with thousands of variables in just one function call. In the simulation below, you can hover over each perceptron below to see how many parameters(weights+bias) it has.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
    // a percptron is an array of weights
    class Perceptron extends Array{
        constructor (numInputs) {
            for (var i = 0; i < numInputs; i++) {
                this.push(random(-1,1))
            }
            this.bias = random(-1,1)
        }
        eval = function (d) {
            return sum(this * d) + this.bias
        }
    }

    // likewise, a layer is an array of perceptrons
    class Layer extends Array {
        constructor (numPerceptrons, numInputs) {
            super()
            for (var i = 0; i < numPerceptrons; i++) {
                this.push(new Perceptron(numInputs))
            }
        }
        eval = function (d) {
            var res = []
            this.lastInput = d
            for (var i < 0; i < this.length; i++) {
                res.push(this[i].eval(d))
            }
            this.lastOutput = d
            return res
        }
    }

    // finally, an MLP is an array of layers
    class MLP extends Array {
        constructor (schema) {
            super()
            for (var i = 1; i < schema.length; i++) {
                this.push(new Layer(schema[i], schema[i-1]))
            }
        }
    }

    // first element describes the incoming data
    // the rest describe the number of perceptrons per layer
    var schema = [1,1,2,3,4]
    var mlp = new MLP(schema)

Since we have put down our MLP components in code, we have all we need to implement the forward pass algorithm and see how the it maps out on a graph.

1
2
3
4
5
6
7
8
9
10
11
12
    MLP.forward = function (input) {

        // feed forward the output of current layer
        // as input of next layer
        for (var i = 0; i < this.length; i++)
            input = this[i].eval(input)

        return input
    }
    
    var schema = [1,1,2,3,4]
    var mlp = new MLP(schema)

You might be thinking “Huh? Isn’t the whole point of an MLP that it’s more complex than a line? What’s the difference between this and a single perceptron?”. That is 100% correct. Nesting perceptrons on their own is just applying transformations on the line, so it doesn’t change the shape of the graph. This is where ReLU comes in. ReLU is an activation function (like sigmoid) that takes a line and clips off its negative side. You can see what appling ReLU does to random lines below.

While it might seem like a trivial change, it is enough to make our MLP adopt shapes much more complex than quadratic and cubic functions, because each perceptron has potential to change the direction of the line. The derivative of a ReLU is also very convenient. It is 1 for the line segment, and 0 for the flattened segment.

1
2
3
4
    // we don't make the last layer a ReLU 
    // because we want the output to allow negative values
    var schema = [1, [1,'relu'], [2,'relu'], [3,'relu'], 4]
    var mlp = new MLP(schema)

Speaking of derivatives, I think this is a good place to refresh and go a little deeper into the calculus we have glossed over. If you already know calculus, feel free to skip this section. If you don't, let's go back and talk about how derivatives are the slope of a curve at a certain point. Since they are slopes, we can represent them as fractions. Like we say \( m = \dfrac{\Delta y}{\Delta x} \) for a line, we can say \( m = \dfrac{\partial y}{\partial x} \) for any graph in general (lines included). It is read as the derivative of "y in terms of x". In a normal perceptron, we know that our cost function is \( c(x) = {(x-y)}^2 \), its derivative in terms of \(x\) is \(2 \cdot (x-y) \) and our perceptron is \(p(x) = w \cdot input + b\). In a normal perceptron, the cost would be \(c(p(x))\) and we are supposed to find the derivative of \(c(p(x))\) in terms of w (i.e \(\dfrac{\partial c(p(x))}{\partial w}\)).

Since we only know it in terms of \(p(x)\) (i.e \(\dfrac{\partial c(p(x))}{\partial p(x)} = 2 \cdot (p(x)-y)\)), it looks like we are at a dead end. But this is where a convenient property of fractions comes into play. We can use the property \(\dfrac{y}{x}\cdot\dfrac{x}{z} = \dfrac{y}{z}\) to rephrase \(\dfrac{\partial c(p(x))}{\partial w}\) as \(\dfrac{\partial c(p(x))}{\partial p(x)} \cdot \dfrac{\partial p(x)}{\partial w}\) which simplifies into \(2 \cdot (p(x)-y) \cdot input \). Seem familiar? It's the chain rule. I showed it to you in this slightly more verbose way to explain how backpropagation works.

Backpropagation is an algorithm that uses the chan rule of calculus to calculate derivatives for perceptrons within perceptrons or, more intuitively, perceptrons that feed into other perceptrons. Let's say you have two perceptrons(\(p_1\) and \(p_2\)) joined end to end. That gives you a structure and equations that look like this.

To do gradient descent, we need to be able to find the derivative of the cost with respect to both perceptrons. Let's start with the easier \(p_2\).

How about \(p_1\)? We can use chain rule like so.

See the pattern? The derivative for one perceptron is depends on the one immediately next to it. Extending the chain to three perceptrons makes it even clearer.

Let’s analyze one last structure before we put this into code. What happens when a perceptron feeds into two instead of one?

Since P1 has both a presence in P2 and P3, we sum up the derivatives.

Don’t worry if it doesn’t sink in immediately. The important thing is to grasp that when this is scaled up, any perceptron’s derivative can be calculated by using the derivatives of the others it feeds into. It’s the forward pass in reverse.

Once we understand this, finding the gradients for all the perceptrons becomes simple, and we can run gradient descent on our MLP.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
    MLP.backward = function (desired) {
        // calculate cost derivatives for the first layer
        // multiply with activation function derivs
        var global_derivs = cost_deriv(this[this.length-1].lastOutput, desired) * this[this.length-1].activation_derivs()
        var prev_derivs;
        for (var i = this.length-1; i >= 0; i--) {
            var l = this[i]
            // clone of this layer to store derivatives
            var d = this.derivs[i]
            if (i != 0)
                // store derivs for previous layer if it exists
                prev_derivs = new Matrix(this[i-1].length, 0)

            for (var j = 0; j < d.length; j++) {
                var p = new Matrix(d[j].length, global_derivs[j]) * l.lastInput

                d[j] = d[j] + p
                d[j].bias += global_derivs[j]
                if (i == 0) continue
                for (var k = 0; k < prev_derivs.length; k++) {
                    prev_derivs[k] += global_derivs[j] * l[j][k]
                }
            }
            global_derivs = prev_derivs
        }
    }
    MLP.applyDerivs = function () {
        for (var i = 0 ; i < this.derivs.length; i++) {
            for (var j = 0 ; j < this.derivs[i].length; j++) {
                for (var k = 0; k < this.derivs[i][j].length; k++) {
                    this[i][j][k] -= WEIGHT_LRATE * this.derivs[i][j][k]
                }

                this[i][j].bias -= BIAS_LRATE * this.derivs[i][j].bias
            }
        }

        this.clearDerivs()
    }
    var schema = [1,1,2,3,4]
    var mlp = new MLP(schema)

And that’s it. We have made a fully operational MLP. We can even change it to a logistic regression network by just changing the last layer’s activation function and the cost, like below.

Note: You might want to stop it after running because it tends to eat up a lot of computing power and slow down the webpage.

1
2
3
    cost_deriv = neglog_cost_deriv
    var schema = [2,4,6,[8,'relu'],8,[1,'sigmoid']]
    var mlp = new MLP(schema)

We will finish this article by exploring some slightly more in-depth problems and solutions that come with an MLP. Since this article is hardly meant to be comprehensive, I recommend consulting more sources after it. There are great lectures by Andrew Ng, Andrej Karpathy and Geoffrey Hinton on Youtube, as well as a good practical series by sentdex.

Learning Rate

When we were implementing the gradient descent and perceptron algorithms, I glossed over some very important numbers in the code, the learning rates.

1
2
3
4
5
6
    // 0.01 is for speed purposes
    xs[i] -= 0.01 * derivative
    . . . 
    perceptron[0] += 0.3*adj[0]
    perceptron[1] += 0.3*adj[1]
    perceptron[2] += 0.3*adj[2]

It is important not to just subtract the gradient itself from the weight, as it can lead to jumps that are "too big" and miss the optimum solution. In the simulation below, you can see how a learning rate of 1 works on the graph of \(x^2\), compared to a slightly better 0.9, a reasonable 0.3 and an unreasonable 0.05. Click on the graph to trigger one pass.

As you can see, a badly picked learning rate like 1 can lead your algorithm to hover above the optimum and never reach it. A slightly less worse rate like 0.9 can keep missing the optimum value by overshooting and take a long time to converge. On the other hand, an unnecessarily small rate like 0.05 wastes time and energy by undershooting. In the end, only experience and educated guesses can get you a good or ideal learning rate, in this case 0.3.

It is useful to know that there are more sophisticated algorithms to manage learning rates, like RMSProp, AdaMax and Adam, which are used almost ubiquitously in practical applications, but are outside the scope of this article.

Exploding & Vanishing Gradients

In backpropagation, we have seen that a perceptron's derivative is calculated by multiplying the derivative of the first layer with weights from the others. In deep networks(networks with many layers), this can present a problem. Let's imagine our weights are all smaller than one, maybe an average of 0.5. In that case, for a perceptron \(x\) layers deep, we get the derivative multiplies by \(0.5^x\), which rapidly goes to zero. We call this a vanishing gradient. It is very harmful for an MLP because a zero derivative means zero gradient, which means zero gradient descent. Vanishing gradients make an MLP freeze and stop improving.

On the other hand, if the weights have an average value of 2, we get our derivative multiplied by \(2^x\), which goes to large values really fast. A ten layer network would have gradient of 1024. You can see why we call them exploding gradients. Just like having a big learning rate makes the MLP skip over optimal weights, exploding gradients make a network unstable and even lead "infinite" weights, which crash the network entirely.

A solution to both problems is to ensure that the weights are initialized so that the gradients get multiplied by a reasonable number. There are schemes such as normal distributions and Xavier initialization, which you can read about in this article.

Regularization

Another problem that plagues neural networks is their tendency to overfit. Overfitting is when the model becomes a little too good at predicting the dataset, at the expense of its predictive powers outside it. Think of it like this. When you want to predict a trend, you want a line that goes smoothly in that direction. If you have a lot of micro jumps and fluctuations in the prediction, it calls into judgment how effective it is. Someone saying Apple stock is going up tomorrow after seeing the general trend is more credible than someone who predicts the ups and downs it will have every minute of tomorrow by memorizing every minute it has been on the market so far.

A method commonly used for regularization is ensuring the weights are as small as possible, to avoid huge spikes or unstable behavior. We can do this easily by adding the magnitude of the weight to the cost. This article is very good at explaining it.

Batches

Using batches in a neural network means adding the derivatives of multiple pieces of data before updating the network’s weights. It gives a marked performance advantage because we can do one backward pass for many forward passes. It also makes sense intuitively because it seems counter-productive to change the entire network based on one piece of data. It sounds like the entire government changing because one person said they didn’t like it. But batches are more like democracy. Every voices their opinions and the most prevalent one is adopted.

Conclusion

I hope this article has been helpful in understanding and implementing your own neural networks from scratch. It is far from comprehensive, so I recommend you consult other sources. After all, the search for knowledge is never over.

To infinity and beyond.

Warning: This is not a comprehensive article on neural networks. While it does discuss the mathematical concepts behind them and expects at least some programming experience, there are many aspects of neural networks that are glossed over. The idea is to create one from scratch with only the bare essential knowledge, leaving the option open for further study on your own. It is meant for people who have tried reading the theory and want to see it simplified or people who want a quick intro which they can build upon later. More theory intensive resources I recommend are the Youtube series by Andrew Ng and sentdex.

Neural networks are a type of machine learning algorithm that are vaguely inspired by the human brain. They take a certain input and are trained to predict the desired outcome, which could be anything from house prices to pictures of dogs. They are constructed of very simple analogues to neurons, called perceptrons, which are basically linear functions that we will optimize. It’s okay if you don’t understand what all that means. That’s what this article is here for. All that you need to know is that a neural network takes in some numbers and tries to guess the best ones to spit out.

This article will be split into three parts:

• Part 1: Calculus An introduction to the prerequisite differential calculus that drives neural networks (with a bunch of interactive animations)
• Part 2: Perceptron Using calculus to code a perceptron, the smallest unit of a neural network (with a bunch of interactive animations)
• Part 3: Multi-Layer Perceptron (MLP) Stringing perceptrons end to end as well as stacking them on top of each other to make more powerful models (with a bunch of interactive animations, obviously)

So, without further ado, let’s get into it. Calculus, a terrifying branch of mathematics that’s up there with rocket science and neurosurgery when it comes to difficulty, or so they say. It’s actually very simple. A very good way to see differential calculus is as the slopes of curves. And I will show it to you in the best way to explain slopes, Buzz Lightyear.

We all know his catchphrase “To Infinity and Beyond”. While it’s very cool to say, mathematics tells us that he can never actually reach infinity, let alone go beyond it. Math is cruel like that sometimes. But we can (and do) say that Buzz is approaching infinity the more he goes forward and approaching negative infinity the more he goes backward. And we can measure how fast he is approaching infinity with how steeply he is ascending. In math, we call that the slope/gradient of the line.

In the simple equation ( y = x ), the gradient is 1. And we can set his inclination that way to get a natural looking flight path.

1
2
3
4
5
6
if (Math.abs(x) >= 2.5)
    speed *= -1
        
x += speed

drawBuzz(x, x, 1) // x, y, and inclination/slope

Now you know all the calculus you need to code a basic perceptron. How? By combining it with gradient descent. Gradient descent is the core algorithm behind neural networks. It uses the gradient of a function to find the location at which it has the lowest value. Since an increasing function has a positive gradient and a decreasing function has a negative gradient, in either case, going the opposite direction of the gradient takes you to the minimum of the function.

In the example below, there are 4 Buzz Lightyears scattered on random points in the function, and they all try to go to the minimum using gradient descent. You can use the reset button to make them go all over again. When you do that, you’ll notice that some get stuck on the upper “rung” of the function and think that’s the best they will do. In that case, we say they are caught in a “local minimum”, not the absolute one. You may also notice that some of them disappear over the left edge almost immediately because it keeps going down. Those Lightyears are even closer to the best minimum.

1
2
3
4
5
6
7
8
9
10
11
for (var i = 0; i < xs.length; i++) {
    var derivative = 3*cos(xs[i]*3)+1

    // subtracts the gradient
    // 0.01 is for speed purposes
    xs[i] -= 0.01 * derivative

    // recalculate derivative
    // because x changed
    derivative = 3*Math.cos(xs[i]*3)+1
}

There is also a variant of gradient descent called gradient ascent. It’s fundamentally the same, except it tries to find the maximum of the function by going along the gradient. Here it is with the same function. Notice that there are local maxima here instead of minima.

1
2
3
4
5
6
7
8
9
10
for (var i = 0; i < xs.length; i++) {
    var derivative = 3*cos(xs[i]*3)+1

    // only difference is the plus
    xs[i] += 0.01 * derivative

    // recalculate derivative
    // because x changed
    derivative = 3*Math.cos(xs[i]*3)+1
}

And that’s all there is to it. In the next part of this series, we’ll see how we can use gradient descent to make a perceptron, the smallest unit of a neural network, and make it find the best-fit line for arbitrary data.

The format of the CTF competition was 10 challenges ranging from easy to hard with 7 hours given to solve them. They tested our abilities in branches of cybersecurity such as General Information, Digital Forensics, Web Security, Cryptography and Malware Reverse Engineering. In this article, I will be showing how we solved the 8 questions we figured out during the competition, as well as one we did immediately after. Number 10 was way out of my league. Maybe I’ll write a follow up article when I get it. So without further ado, here are the questions.

Cracker (General Information)

A simple question, asking us the name of a popular Linux tool used as a “packet sniffer, WEP and WPA/WPA2-PSK cracker”.

The answer is obviously aircrack-ng.

Unprotected (Digital Forensics)

In this challenge, we were given Unprotected.pcap, a packet capture file, and told to find the flag inside it. By opening it in Wireshark, we can see it’s a combination of TCP and HTTP packets.

A simple filter data.data contains "flag" leaves just one TCP packet, which contains the flag flag{cl3ar_t3xt_15_alway35_5asy}.

Encrypted RSA (Cryptography)

In this challenge, we are given a RSA-encrypted file called secret, as well as the p, q and e used to encrypt it. All of this enough information to decrypt the file. So after a fast search on crypto.stackexchange.com, we made this script.

def egcd(a, b):
    x,y, u,v = 0,1, 1,0
    while a != 0:
        q, r = b//a, b%a
        m, n = x-u*q, y-v*q
        b,a, x,y, u,v = a,r, u,v, m,n
        gcd = b
    return gcd, x, y
 
def main():
 
    p = 11882546252751469607361356421348933496327112595288260315935663917400681403905188808476289112967043136936045873689827577396206505769293138372274271493958287
    q = 10374751834382966611285517450958269115435289482194774831009591093240922739864785750413607023913149510232252798244495377789107452564252835088008933746132847
    e = 65537
 
 
    cipher_text = int.from_bytes(open("secret","rb").read(), byteorder="big")
 
 
    # compute n
    n = p * q
 
    # Compute phi(n)
    phi = (p - 1) * (q - 1)
 
    # Compute modular inverse of e
    gcd, a, b = egcd(e, phi)
    d = a
 
    print( "n:  " + str(d));
 
    # Decrypt ciphertext
    pt = pow(cipher_text, d, n)
 
    print( "plain textt: " + str(int.to_bytes(pt, 128, byteorder="big")) )
 
main()

Running the script outputs a series of null bytes followed by Nice Job, flag is FLAG{Gr3at_J0b_F0r_Th3_D3crypti0n}.

Gu55y (Web Security)

In this challenge, we were given a url to exploit. Its functionality is to take your inputs and store them in your cookies as a serialized php list, so that it may display them again when you refresh the page.

Crossing out SQL injection as a vector, we inspected the the page source and found an HTML comment reading <!-- I love vim~ -->. Seeing that vim was involved, we tried to search for known vim file extensions such as .php~ and .php.un~. We succeeded in downloading .index.php.swp, which gave us this code.

#try to read fl4g.php
Class l33t{
    public function __toString()
    {
        return highlight_file($this->source,true);
    }
}

This code excerpt gave us two things, the location of the flag and the vector with which we could get it. We made a l33t object and set its source to fl4g.php. We then serialized it in a list, url-encoded it and put the value in our list cookie.

serialized: a:1:{i:0;O:4:"l33t":1:{s:6:"source";s:8:"fl4g.php";}}

urlencoded: a%3A1%3A%7Bi%3A0%3BO%3A4%3A%22l33t%22%3A1%3A%7Bs%3A6%3A%22source%22%3Bs%3A8%3A%22fl4g.php%22%3B%7D%7D

Refreshing the page gives us the flag flag{5w337_PHP_0bj3c7_!nj3c7!0n}

Habibamod (Digital Forensics)

In this challenge, we were given another packet capture file, Habibamod.pcap. In Wireshark, we can see it’s an HTTP session where a file is being uploaded.

The contents of the file are a JSON object with properties data and encoder.

{"data": ".!…!!..!!.!!…!!….!.!!..!!!.!!!!.!!.!.!.!…!..!!.!.!….!!.!.!.!…!…!!..!!!!..!..!!…..!!!.!.!.!!!..!..!…!…!!..!.!.!!…!!..!!…..!!..!…!!..!.!..!!…..!!..!!…!….!.!…….!!.!!!..!!..!….!.!!!..!..!..!.!!!..!!.!…….!!.!!.!.!!….!..!!.!!!.!!.!..!.!!.!!!..!!..!!!.!!!!!.!", "encoder": "ZGVmIG15X2VuY29kZXIoZGF0YSk6CiAgICBiaW5fcmVwID0gJycuam9pbihmb3JtYXQob3JkKGkpLCAnYicpIGZvciBpIGluIHgpCiAgICByZXR1cm4gYmluX3JlcC5yZXBsYWNlKCcwJywnLicpLnJlcGxhY2UoJzEnLCchJykgCg=="}

Decoding encoder with base64, we get a python function called my_encoder. After analyzing it, we made a python function of our own to reverse it. We figured out encoder was base64 because of the telltale == at the end.

 def my_encoder(data):
    bin_rep = ''.join(format(ord(i), 'b') for i in x)
    return bin_rep.replace('0','.').replace('1','!') 

 def our_decoder(data):
    # the previous function converts a string to a bitmap 
    # and then changes 0 and 1 to . and ! respectively

    # reverse the string replacement
    data = data.replace('.','0').replace('!',1)
    
    # change the bitmap to numbers
    data = [int(data[i:i+8], 2) for i in range(0, len(data), 8)]

    # change the numbers to a string
    data = ''.join([chr(i) for i in data])
    return data

Putting data into this function gives us the flag Flag{TMCTFy0urDec0de0f!@nd.Is@ma7ing}

GoldASM (Malware Reverse Engineering)

In this challenge, we are given an assembly file GoldASM.asm. It described a function that had many repetitions of this format.

    mov     rax, QWORD PTR [rbp-24]
    mov     eax, DWORD PTR [rax]
    cmp     eax, 70
    jne     .L2
    mov     rax, QWORD PTR [rbp-24]
    add     rax, 4
    mov     eax, DWORD PTR [rax]
    cmp     eax, 76
    jne     .L2

We realized that the code was going across a string and checking its value against hardcoded numbers. By removing the repeated parts of code and making the rest human readable, it becomes obvious.

    0:
    compare     eax, 70   # 'F' char
    4:
    compare     eax, 76   # 'L' char

There were certain spots though, where it wasn’t simply comparing. For example, one line doubled the number before comparing it and another subtracted from it.

    8:
    add eax, eax
    compare eax, 130     # meaning we want 65, or 'A'
    24:
    subtract eax, 75
    compare eax, 2       # meaning we want 77, or 'M'`

There were also some points which required a byte be less than a number instead of equal. After dealing with these edge cases, we got a list of numbers [70, 76, 65, 71, 123, 95, 75, 51, 101, 98, 95, 48, 110, 95, 83, 104, 49, 110, 105, 110, 103, 95, 125]. Converting them into characters gave us the flag FLAG{_K3eb_0n_Sh1ning_}.

Sadly, I don’t have any screenshots of the other challenges because they were web based and CyberTalents took them down when the hackathon ended.

All in all, this hackathon was lots of fun and and I’m interersted in doing it again if I can. I’ll make sure to document the whole thing too.