𝛑
𝛑
Posts List
  1. FAQ
  2. Resources

CNN Study Notes

Jan 1, 2018 Felix Zhao 中文 587 words ~3 min read
This post was written in 2018. Tools and versions mentioned may be outdated, though the underlying ideas still hold.

FAQ

What does Maxpooling actually do

  1. Invariance — it cares more about whether a feature exists than exactly where it is. Think of it as a strong prior that forces learned features to tolerate some spatial variation.
  2. Shrinks the input size for the next layer (removes redundancy), cutting down computation and parameter count.
  3. Produces fixed-length output. (Useful for text classification where input length varies — pooling gives you a consistent output size.)
  4. Helps prevent overfitting, though it can also cause underfitting if overdone.

    Author: Zhihu user

How do you design a good neural network?

Honestly, no idea yet. But for parameter initialization you can just use what tensorflow gives you, for example:

W1 = tf.Variable(tf.random_normal([784,256]))
b1 = tf.Variable(tf.random_normal([256]))

W1 = tf.get_variable("W1", shape=[784, 256],
                     initializer=tf.contrib.layers.xavier_initializer())
b1 = tf.Variable(tf.random_normal([256]))

This gets you noticeably better accuracy. Good initial parameters make a real difference.

How many hidden layers should you use?

Trial and error.

What activation functions are there, what do they do, and how do you pick one?

Activation functions introduce non-linearity, so the network can handle non-linearly separable problems.
Visualizing 26 activation functions
What’s the point of activation functions in neural networks — Zhihu

What pooling methods are there?

  • maxpooling (slide a window, take the max value of features in the neighborhood)
  • meanpooling (take the average)

What do filters do?

At its core, a neural network is basically layers of filters stacked on top of each other — filtering out low-level features, selecting high-level ones.

How do you reduce overfitting in a backpropagation network?

What’s the difference between standard BP and accumulated BP?

Same as the difference between standard gradient descent and stochastic gradient descent.

How do you escape local minima?

  • Initialize multiple neural networks with different sets of parameters, train them all the standard way, pick the one with lowest error as your final model
  • Simulated annealing — at each step accept a worse solution with some probability
  • Use stochastic gradient descent

Epoch vs Batch — what’s the difference

One full pass through all the data is an Epoch. That full pass gets split into smaller chunks (batches) for each iteration.

When does a CNN require fixed-size input, and when doesn’t it?

If there’s a fully connected layer, input size must be fixed — because the FC layer connects every input pixel together, and once the parameter count is set it can’t change. So inputs have to be uniform. If you only have convolution and pooling layers, image size doesn’t matter — the filter just scans whatever it gets.

What if images are different sizes but you need fixed size?

  • Force resize, but this falls apart if you need fine-grained detection of objects — it distorts things.
  • Manual annotation — recommend labelImg

Resources