Initially a variant of LSTM known as AWD LSTM was pre trained (unsupervised pre training) for language modelling task using wikipedia articles. In the next step the output layer was turned into a classifier and was fine tuned using various datasets from IMDB, yelp etc. When the model was tested on unseen data, sate of the art results were obtained. The paper further went on to claim that if a model was built using 10,000 rows from scratch then fine tuning the above model (transfer learning) would give much better results with 100 rows only. The only thing to keep in mind is they did not used a transformer in their architecture. This was because both these concepts were researched parallely (transformers and transfer learning) so researchers on both the sides had no idea of what work the other was doing. Transformers paper came in 2017 and ULMFit paper (transfer learning) came in early 2018.

Now architecture wise we had state of the art architecture i.e. Transformers and training wise we have a very beautiful and elegant concept of Transfer Learning. LLMs were the outcome of the combination of these 2 ideas.

Training AI models today isn’t just about designing better architectures—it’s also about managing data efficiently. Modern models require vast datasets and need those datasets delivered quickly to GPUs and other accelerators. The problem? Traditional data loading systems often lag behind, slowing everything down. These older systems rely heavily on process-based methods that struggle to keep up with the demand, leading to GPU downtime, longer training sessions, and higher costs. This becomes even more frustrating when you’re trying to scale up or work with a mix of data types.

To tackle these issues, Meta AI has developed SPDL (Scalable and Performant Data Loading), a tool designed to improve how data is delivered during AI training. SPDL uses thread-based loading, which is a departure from the traditional process-based approach, to speed things up. It handles data from all sorts of sources—whether you’re pulling from the cloud or a local storage system—and integrates it seamlessly into your training workflow.

SPDL was built with scalability in mind. It works across distributed systems, so whether you’re training on a single GPU or a large cluster, SPDL has you covered. It’s also designed to work well with PyTorch, one of the most widely used AI frameworks, making it easier for teams to adopt. And since it’s open-source, anyone can take advantage of it or even contribute to its improvement.

The hallmark of a good AI model is its ability to work the same in different groups, settings, and situations. See how these NCI researchers used in-house and external images to test their prostate model’s generalizability.