Choosing a neural network architecture: MLPs, CNNs, RNNs, and transformers

A neural network architecture is the arrangement of layers, connections, and operations that turns input data into an output. Choose it around the prediction you need, the structure of your data, and the limits of the system that will run it. An MLP, CNN, recurrent network, or transformer is a candidate to test, not a guarantee of better results.

For developers choosing a first model or replacing an existing one, the useful question is: which approach meets the quality target with acceptable latency, memory use, and maintenance? This guide compares four common model families and gives you a practical way to make that decision.

What a neural network architecture defines

The architecture specifies how data flows through the model: layer types, their sizes, their connections, and the operations between them. The learned weights and biases are the values fitted during training. Two models can share an architecture and produce different results because their weights, training data, or training procedure differ.

In a simple network, the input layer receives a feature vector, hidden layers transform it, and an output layer produces a prediction. Nonlinear activation functions let stacked layers represent relationships that a single linear mapping cannot. PyTorch’s neural-network tutorial shows how these components form a model.

Keep architecture choices separate from training settings. A loss function defines the training objective, backpropagation computes gradients, and an optimizer uses those gradients to update parameters. Learning rate, batch size, and training duration can change results without changing the network’s structure.

Start with the input and the required output

Write down one prediction task before choosing among types of neural networks. “Analyze customer data” leaves too much unspecified. “Predict next month’s demand from information available today” identifies an output and a time boundary.

• Tabular data: rows contain named numerical or categorical features. Check missing values, feature meaning, and which columns exist at prediction time.
• Image data: nearby pixels have spatial relationships. Decide whether the output is a class, an object location, or a segmentation mask.
• Sequential data: order matters. Define the context window, the forecast horizon, and whether future observations are available.
• Text: distinguish classification, retrieval, extraction, and text generation. They need different outputs and evaluation methods.

Also record who will use the result and what happens when the prediction is wrong. A small improvement in average accuracy may have little value if the model misses the cases that matter or cannot respond within the application’s deadline.

Four common neural network families

MLPs: feedforward neural networks for feature vectors

A multilayer perceptron passes information through fully connected layers with nonlinear activations. Its forward computation has no recurrent state linking successive inputs. The MLP chapter in Dive into Deep Learning explains this structure.

An MLP is a reasonable neural baseline for a fixed feature vector or a prediction head attached to another model. It does not build in the local spatial structure of an image. For structured data, compare it with linear models and tree ensembles before increasing depth or width.

CNNs: convolutional neural networks for local patterns

A convolutional neural network applies shared filters across local regions. This makes locality part of its design, rather than asking every input feature to connect independently to every hidden unit. See the D2L explanation of convolutions.

A CNN is a useful candidate for image classification and other computer vision tasks. Image resolution, receptive field, and the output head still matter. An image classifier does not become an object detector simply by receiving more training images. For your first comparison, use a suitable pretrained model and measure it on representative examples.

RNNs, LSTMs, and GRUs: recurrent neural networks for sequences

A recurrent neural network carries a hidden state from one step to the next. That gives it a way to incorporate earlier observations while processing a sequence; the D2L RNN chapter describes this recurrence.

LSTMs and GRUs use gates to control information flow. An LSTM also maintains a cell state, as shown in the PyTorch LSTM documentation. These designs address limitations of simple recurrent networks, but they do not guarantee accurate long-range predictions.

Consider a recurrent model for ordered sensor data or a streaming task, then test its state handling and latency. For time series forecasting, compare against a simple forecasting baseline as well. Sequence length alone does not establish which model will win.

Transformers: attention-based relationships between positions

A transformer combines attention with feedforward layers and other components. Self-attention relates positions within a sequence; positional information supplies order. The D2L transformer chapter explains the standard design.

Pretrained transformers are useful candidates for natural language processing and tasks where relationships across the input matter. Their suitability depends on the checkpoint, task, input length, and deployment budget. A large language model is not automatically the best classifier.

Establish a baseline before adding complexity

Start with a method that makes the comparison meaningful: the current rule, a constant forecast, logistic regression, or another straightforward model appropriate to the task. Google’s guidance on ML experiments recommends measuring a baseline, making controlled changes, and recording unsuccessful experiments too.

For tabular data, include a tree-based candidate. A 2022 NeurIPS benchmark found strong tree-model performance on its medium-sized tabular datasets. That result supports testing trees; it does not prove that they beat every neural network on every dataset.

Set a stopping rule. If a simpler model meets the agreed quality and operating requirements, a more complex architecture needs a concrete benefit to justify its cost. Inspect errors before adding layers: missing information or inconsistent labels may be the real limitation.

Decide between transfer learning and training from scratch

Architecture and initialization are separate decisions. A pretrained checkpoint already contains learned weights. You can freeze a backbone and train a new prediction head, or fine-tune some or all of its parameters. The PyTorch transfer-learning tutorial demonstrates both approaches for images.

Check whether the checkpoint’s inputs and training domain fit your task. Review its license, preprocessing requirements, and documented limitations. A small labeled dataset can make reuse attractive, but it cannot establish whether the pretrained model is reliable on your own cases.

For a language-model adaptation project, our LoRA fine-tuning guide covers one parameter-efficient method. LoRA changes how you adapt weights; it is not a replacement for choosing an appropriate model and evaluation set.

Protect the comparison from data leakage

Keep the test set out of architecture selection and tuning. Fit preprocessing on training data, then apply the fitted transformation to validation and test data. The scikit-learn pitfalls guide explains how leakage can make an unsuitable model look accurate.

Choose splits that reflect deployment. If multiple examples belong to the same customer, device, or document, consider a group split. For forecasting, evaluate on later observations rather than randomly mixing past and future. The cross-validation documentation covers grouped and time-aware approaches.

Give candidates a stated tuning budget and record the data version, preprocessing, seed, configuration, and selection metric. Repeat promising experiments when feasible to see whether small differences persist. Inspect errors by relevant category or operating condition, rather than relying on one aggregate score.

Compare training cost with inference requirements

Training time and serving latency answer different questions. A model that trains quickly may still be unsuitable for an interactive application. Conversely, an expensive one-time training run may be acceptable if the deployed model meets a repeated operational need.

Measure the full prediction path on the intended hardware: preprocessing, model execution, and output handling. Record input size, batch size, concurrency, software versions, and whether initialization is included. Separate cold-start measurements from steady-state requests. Our training versus inference hardware guide explains the broader distinction.

• Quality: use a metric tied to the task, and examine important failure cases.
• Latency: record the distribution, including a high percentile such as p95, rather than just the average.
• Throughput: measure how much work finishes at the required latency and concurrency.
• Memory: measure peak use for the intended input and batch sizes.
• Operating effort: include retraining, monitoring, runtime support, and deployment complexity.

Parameter count alone is not a complete hardware-sizing method. Intermediate tensors, precision, input dimensions, and runtime behavior also matter. Do not choose a GPU model from an architecture name alone; first establish what your actual workload requires.

Three examples of a useful first comparison

These are proposed experiment designs, not benchmark results.

• Predict demand from weekly sales records: compare a seasonal baseline and a tree model using valid lagged features before testing an MLP or sequence model. Evaluate future weeks using only information available at the forecast date.
• Classify product defects from photos: compare a pretrained compact CNN with another suitable vision model. Hold out representative production conditions and measure the effect of image resolution on missed defects and latency.
• Route support messages: compare a simple text classifier with a pretrained encoder. Add a generative model only if the task needs generation or the measured benefit justifies its extra operating demands.

The point is to define what evidence would change your choice. “Try a transformer” is an experiment only after you specify the task, baseline, data, metric, and constraints.

Neural network architecture selection checklist

1. Define the input, output, and information available at prediction time.
2. Choose a quality metric and concrete latency and memory limits.
3. Build a simple baseline and inspect its mistakes.
4. Select a small set of plausible architectures or pretrained checkpoints.
5. Use the same evaluation protocol and document the tuning budget.
6. Measure quality and operating behavior on representative data and hardware.
7. Keep the simplest candidate that meets the requirements, with the evidence and limitations recorded.

Common questions about neural network architectures

Are these four families a complete list?

No. Models can combine components, and other designs serve different tasks. These four families are a starting point for selection, not an exhaustive classification or four mutually exclusive categories.

Are recurrent neural networks outdated?

Do not reject or select one based on age. Compare an appropriate recurrent implementation with alternatives on the sequence task, including state management, predictive quality, memory, and serving latency.

Does a deeper neural network always perform better?

No. More layers can change capacity and optimization difficulty. Better training-set fit does not establish better performance on unseen data. Use validation results and the application’s constraints to justify extra complexity.