What are manifolds? I started down this rabbit hole because of Chris Colah’s excellent Neural Networks / Functional programming blog post:
At their crudest, types in computer science are a way of embedding some kind of data in n bits. Similarly, representations in deep learning are a way to embed a data manifold in n dimensions.
So the problem is that people talk about them even when you don’t think you need to care about topology, or even really know what topology means.
The following is a bit of a dive into examples of manifolds with the intention of better understanding what a manifold is.
Opening quote from Wikipedia:
In mathematics, a manifold is a topological space that locally resembles Euclidean space near each point.
Without knowing exactly what topological and Euclidean spaces are that’s hard to understand. Let’s try simple Wikipedia:
A manifold is a concept from mathematics. Making a manifold is like making a flat map of a sphere (the Earth).
Ok, kinda. Except this is nicely confused by further reading the Wikipedia page which has:
A ball (sphere plus interior) is a 3-manifold with boundary. Its boundary is a sphere, a 2-manifold.
So, to me, this makes no sense of creating a manifold (a flat map) from a manifold (a sphere is a 2-manifold). Why create a manifold if you’ve already got one? (My thinking here is wrong, but I’m trying to explain all my incorrect thinking as I go and clear it up at the end)
Some kind of useful quotes from the section on boundaries:
A piece of paper is a 2-manifold with a 1-manifold boundary (a line)
A ball is a 3-manifold with a 2-manifold boundary (a sphere)
Show me the money
‘Simple’ examples are usually the way for me out of confusion. The list of manifolds helps a lot:
- are all manifolds, so a line is a 1-manifold
- A x,y 2D plane is a 2-manifold
- All spheres are manifolds… stop there, I’m confused
This really gets me. Why do we need to create maps of the world if it’s already a manifold. The simple Wikipedia explanation makes sense, i.e. it’s obvious that we want to create a flat map of the world. It also makes sense that where as a triangle on the globe doesn’t add up to 180 degrees (bad) does nicely add up to 180 degrees on a flat map (good). But why, if the earth is already a manifold do we want to create another manifold from that. Isn’t being a manifold good enough? Some people are never satisfied.
A sphere is a 2-manifold, so although it is a 3D object, close to any one point it looks like a 2D grid.
The important part is the n-manifold. This is what cleared it up for me, you don’t care about creating one manifold from another.
An n-manifold resembles the nth dimension near one point. So a flat map is a 2-manifold and the earth is a 2-manifold because they both represent a 2D grid near one point.
In another post from Chris Colah, he talks explicitly about manifolds. But he talks about them assuming that you know what they are:
The manifold hypothesis is that natural data forms lower-dimensional manifolds in its embedding space.
Although John Nash is most famous for his Nash equilibrium, but it a lot of his most important work was to do with manifolds:
His famous work on the existence of smooth isometric embeddings of Riemannian manifolds into Euclidean space.
Interestingly differentiating a curve at a point on that curve gives you the slope of the flat line at that point. So there’s a connection between differentiation and manifolds.
There’s a Calculus on Manifolds book that looks interesting, taken from this talk on The simple essence of automatic differentiation. Then there is a link back to Chris Olah’s NN/FP post based on this Conal Elliot’s Automatic Differentiation paper.
Euclid forgive me
To be honest even Euclidean space gets me confused. That’s basically just which I can understand better. But is there a specific reason for using Euclidean? Is there some extra property of Euclidean space that isn’t inherent in ? There’s a fundamental principle of parallel lines not converging in Euclidean space, but then if a manifold is in Euclidean space, how can a sphere be a manifold? It doesn’t explain why we want to convert one manifold into another (as in the spheres section above), or even really know what topology means.
I keep thinking of Euclidean space as basically everything. But it’s not – it’s a n-dimensional grid (with infinite points as it’s the real number line in each direction) and it has straight edges. So a ball isn’t in Euclidean space, or “isn’t Euclidean”, I’m not sure which.
In image analysis applications, one can consider images as functions
on the Euclidean space (plane), sampled on a grid
Data manifolds (back to Chris)
Perhaps data manifolds are the structure of the data. For example as above an image data is a 2D Euclidean grid.
They get referred to in the MIT CS231n CNNs course when talking about ReLUs:
(-) Unfortunately, ReLU units can be fragile during training and can “die”. For example, a large gradient flowing through a ReLU neuron could cause the weights to update in such a way that the neuron will never activate on any datapoint again. If this happens, then the gradient flowing through the unit will forever be zero from that point on. That is, the ReLU units can irreversibly die during training since they can get knocked off the data manifold. For example, you may find that as much as 40% of your network can be “dead” (i.e. neurons that never activate across the entire training dataset) if the learning rate is set too high. With a proper setting of the learning rate this is less frequently an issue.
It’s as if data manifold is the useful/accessible part of the data.
What’s not Euclidean
For instance, in social networks,
the characteristics of users can be modeled as signals on the
vertices of the social graph . Sensor networks are graph
models of distributed interconnected sensors, whose readings
are modelled as time-dependent signals on the vertices.
In computer graphics and vision, 3D objects are
modeled as Riemannian manifolds (surfaces) endowed with
properties such as color texture.
Manifolds don’t have to Euclidean! But at one particular point on a manifold surface it is approximately a grid in the local area. So this holds if it is Euclidian, because locally on a sheet of paper it’s Euclidean as the sheet of paper is already Euclidean.
But certainly I like the explanation that 3D graphics are manifolds. So there’s nothing special that makes the earth a manifold it’s just one example of one. So I think this means that any 3D shape is a manifold.
So what’s not Euclidean and not a manifold?
What’s not a manifold?
That dear reader… is an exercise for you.
What did this get us?
- Euclidean space means a grid (duh)
- An image of pixels is a 2D grid – possibly the data manifold that Chris Olah was referring to
- A 3D graphic (or the earth) is a 2-manifold. These are not Euclidean but are 2D Euclidean (grid shaped) close to a point on their surface
- When your data is 3D graphics – that is your data manifold