Difference between local vs. HPC processing

Running neural network training locally versus on High-Performance Computing (HPC) systems introduces key differences:

Scale differences: Local machines typically handle smaller models and datasets, while HPC environments can process massive datasets and more complex architectures.

Training time: What might take days on a local machine could be completed in hours or minutes on HPC systems due to parallel processing capabilities.

Resource allocation: Local setups often face memory constraints, while HPC environments provide managed resources that can be dynamically allocated.

Code adaptation: Minimal code changes are needed when moving to HPC, mainly related to data loading and distribution across nodes.

Setup complexity: Local environments require simple installation of packages like PyTorch, while HPC systems need job submission scripts and knowledge of workload managers.

For smaller neural networks like our Iris dataset example, local processing is sufficient. Larger, production-scale models benefit significantly from HPC resources.

Bristol HPC Resources

University of Bristol provides access to powerful HPC facilities:

  • BlueCrystal Phase 4: A general-purpose HPC cluster with GPU capabilities
  • BluePebble: Designed specifically for machine learning workloads with advanced GPU nodes

Registered students and researchers should visit the High Performance Computing or Bristol’s new supercomputer Isambard 3 and Isambard-AI website for access information and documentation.

Where to go next?

Learning outcomes

In this course we have:

  • Understand the basic theory of a feed forward multi layer perceptron.
  • Start to get the grips with PyTorch, tensors and writing classes in Python.
  • Understand how to pre-process data for training (including test train split)
  • Understand how to feed forward data and evaluate test loss for a neural network.
  • Have a basic intuition for what gradient descent and back propagation are.
  • Implement back propagation to update our weights and biases and reduce our test loss.

Where to go next?

Deep learning and neural networks are a huge field of active research that we cannot cover in 3 hours. This is an intro level class designed to cover the learning outcomes above and serve as a prerequisite to further topics in AI, neural networks and deep learning, such as:

  • Cross entropy loss and more advanced optimisers
  • Training neural networks using High Powered Computing (HPC) resources
  • Convolutional Neural Networks (CNNs) for image/video analysis
  • Recurrent Neural Networks (RNNs) for time series and natural language processing
  • Transformers and Large Language Models (LLMs)
  • Graphical Neural Networks (GNNs)
  • Reinforcement Learning (RL)

For moving forward in your deep learning journey, we recommend some of these external resources: