Resources
Discover useful tools, tutorials, and helpful resources organized by category
Book Neural Networks and Deep Learning
Neural Networks and Deep Learning is a practical and approachable introduction to how modern neural networks work, written with a strong focus on intuition rather than heavy theory. The book walks readers through the core ideas behind deep learning — including perceptrons, backpropagation, gradient descent, overfitting, convolutional networks, and recurrent networks — while gradually building real implementations from scratch in Python. Instead of treating AI as a black box, it explains why neural networks learn, how training actually happens, and what design choices improve performance, making it especially valuable for programmers and self-learners who want to understand the foundations behind today’s AI systems without needing an advanced mathematics background.
Paper LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
The paper LeWorldModel introduces a new way to train AI “world models” directly from raw pixels in a simple and stable manner, addressing a key problem in prior approaches where models collapse or require complex tricks to work. It proposes a streamlined architecture using just two loss terms—one to predict future states in a latent space and another to keep those representations well-structured—eliminating the need for heavy tuning or pre-trained components. Despite being lightweight (about 15M parameters and trainable on a single GPU), the model achieves competitive performance across various control tasks while planning much faster than larger foundation-model-based systems, and it also learns meaningful physical structure, allowing it to detect unrealistic or physically implausible scenarios.
Book Language Models Interview Handbook
This book is a practical guide to understanding and preparing for large language model (LLM) engineering roles, focusing not just on theory but on how LLMs work as complete systems in real-world applications. It walks through the full stack—from tokenization, embeddings, and attention to retrieval (RAG), evaluation, deployment, and optimization—while emphasizing how these components interact to deliver reliable, scalable AI products. Rather than memorizing concepts, the book trains readers to think like engineers: framing problems, making trade-offs between cost, latency, and accuracy, and explaining design decisions clearly in interviews. It also covers common patterns like hybrid search, reranking, and context management, along with career guidance such as resume building and communicating impact, ultimately aiming to help readers bridge the gap between LLM theory and production-ready systems.
Paper Attention Residuals
The paper introduces Attention Residuals, a new approach to replacing the traditional residual connections used in Transformer models, which typically add layer outputs together in a fixed and uniform way. Instead of this rigid accumulation, the authors propose using an attention mechanism across layers, allowing each layer to selectively retrieve and combine information from earlier layers based on the input. This helps address issues like information dilution and inefficient scaling in deep networks, enabling models to better preserve useful representations as they grow deeper. To make the method practical, the paper also introduces a more efficient block-based variant, and experiments show that this approach can improve performance and training behavior with minimal overhead, making it a promising drop-in improvement for modern large language models.
Book Understanding Deep Learning
Understanding Deep Learning by Simon J. D. Prince is a comprehensive and accessible introduction to the core ideas behind modern deep learning. The book focuses on explaining why neural networks work rather than just how to implement them, gradually building from basic concepts like supervised learning and simple neural networks to more advanced topics such as deep architectures and generative models. With clear explanations and visual intuition, it aims to help readers develop a solid conceptual understanding of deep learning systems and the principles that power many of today’s AI applications.
Paper Multi-agent cooperation through in-context co-player inference
This paper explores how cooperation can emerge among self-interested agents in multi-agent reinforcement learning without hardcoded assumptions about how other agents learn; instead of using specialized meta-learning or explicit timescale separations, the authors show that training sequence-model agents against a diverse mix of co-players naturally leads them to infer and adapt to their partners’ strategies within an episode, producing in-context best-response behaviors that, under mutual pressure to shape each other’s learning dynamics, lead to cooperative outcomes, suggesting a scalable way to achieve cooperation using standard decentralized RL with sequence models.
Blog Zartbot
The zartbot/blog repository functions as a technical knowledge base and discussion forum primarily focused on cloud-native networking, high-performance computing (HPC), and system-level optimization. The issues section serves as a collection of long-form articles and engineering deep dives where the author, zartbot (a notable engineer in the networking and AI infrastructure space), explores advanced topics such as SD-WAN architecture, RDMA/RoCE virtualization, BGP and cloud-scale routing protocols, and GPU networking for AI training. By leveraging GitHub Issues as a blogging platform, the repository facilitates technical discourse on specialized subjects like NetDAM (Network Direct Attached Memory) and the trade-offs of the CAP theorem in modern distributed systems, catering to a professional audience of network architects and system programmers.
Paper Deep Delta Learning
The paper introduces Deep Delta Learning (DDL), a new neural network architecture that extends traditional deep residual connections by replacing the fixed identity shortcut with a learnable, data-dependent geometric transformation called the Delta Operator. By parameterizing this operator with a direction vector and a gating scalar, the model can smoothly interpolate between identity mapping, projection, and reflection behaviors, giving the network fine-grained control over how information is erased and written at each layer. This enables it to represent more complex, non-monotonic dynamics than standard residual networks while retaining stable training, effectively broadening the expressive power of deep models without losing the benefits of residual learning.
Paper mHC: Manifold-Constrained Hyper-Connections
The paper "mHC: Manifold-Constrained Hyper-Connections" by DeepSeek-AI introduces a novel architectural framework designed to overcome the training instabilities inherent in high-complexity "Hyper-Connections." While traditional residual streams rely on identity mapping to maintain signal stability, expanding these connections often leads to unbounded signal amplification that causes models to crash during scaling. To solve this, the authors propose mHC, which utilizes manifold projection and doubly stochastic matrices to ensure feature preservation across layers, effectively restoring stability without sacrificing topological diversity. Empirical tests on models up to 27B parameters demonstrate that mHC achieves superior performance on benchmarks like BigBench Hard with minimal computational overhead, offering a more robust path for scaling future foundational models.
Video Steve Yegge's Vibe Coding Manifesto
In this interview from the AI Engineer Summit, legendary developer Steve Yegge lays out his "Vibe Coding Manifesto," arguing that the industry is shifting from traditional manual coding to a "factory farming" model driven by AI agents. Yegge claims that senior engineers are often the most resistant to this change because their identities are tied to old workflows, even though early adopters are seeing productivity gains as high as 10x [03:09]. He envisions a future where the IDE is replaced by an "agent orchestration dashboard" [14:11], where developers act more like high-level product managers or "NASCAR drivers" who manage dozens of agents simultaneously rather than writing individual lines of code [10:11]. While acknowledging the current chaos and execution problems within major labs like OpenAI and Google [29:48], Yegge remains infectious about the future, asserting that while you don't need to know syntax anymore, you still need deep technical concepts to effectively guide the "village of agents" that will soon be building our software [34:47].
Video John Schulman on dead ends, scaling RL, and building research institutions
A conversation from Cursor CEO with Thinking Machines chief scientist John Schulman on the first year LLMs could have been useful, building research teams, and where RL goes from here.
Video Bill Dally - Trends in Deep Learning Hardware
In this talk, Bill Dally walks through the evolving landscape of hardware designed for deep learning, explaining how computational demands of neural networks drive innovation in processors and systems. He highlights the gap between traditional CPUs and what modern AI workloads require — massive parallelism, high throughput, efficient data movement — and shows how specialized hardware (like GPUs and newer AI accelerators) addresses those needs. He also outlines architectural trends and design considerations for future deep-learning hardware: energy efficiency, memory bandwidth, and optimized data flow being key. Overall, the video offers a clear, forward-looking perspective on why and how hardware must evolve to keep up with rapidly advancing AI.
Paper Detect Anything via Next Point Prediction
In this work, the authors propose a model called Rex‑Omni — a 3-billion-parameter multimodal large language model (MLLM) — which reframes a wide range of visual-perception tasks (object detection, spatial grounding, key-pointing, OCR) as a “next-point prediction” problem. Instead of the traditional regression-based coordinate prediction widely used in detectors, Rex-Omni quantises spatial coordinates into discrete tokens and asks the model to predict the next token (or point) in a sequence, allowing detection and localization to be treated in the same token-prediction paradigm as language modelling. The paper argues that this unification leads to both stronger language-vision alignment and improved localization performance. They support their claims with evaluations on standard benchmarks (such as COCO) and show that Rex-Omni outperforms prior methods when given the same or less supervision. Beyond just detection, the design opens up a path toward more general “detect-anything” models where a single architecture handles many tasks.
Paper Recurrent Neural Network Regularization
The paper Recurrent Neural Network Regularization by Zaremba, Sutskever, and Vinyals (2014) introduces a novel approach to applying dropout in Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Traditional dropout techniques, effective in feedforward networks, often fail in RNNs due to the recurrent connections amplifying noise, which can hinder learning. The authors propose a method to apply dropout selectively within LSTMs, demonstrating its efficacy in reducing overfitting across various tasks, including language modeling, speech recognition, image caption generation, and machine translation.
Paper DeepSeek-OCR: Contexts Optical Compression
The paper introduces DeepSeek-OCR, a vision-language model designed to go beyond traditional OCR by extracting text from images and understanding visual context such as layout, semantics, and multimodal relationships. It presents a mixed-expert architecture where only relevant sub-modules activate, enabling efficient processing of high-resolution document images and complex visuals. The authors describe their training system, datasets spanning screenshots, PDFs, and scene text, and evaluate the model on benchmark tasks where it demonstrates strong performance in text extraction, document understanding and layout reasoning. The paper concludes that the unified vision + language approach offers improved accuracy and flexibility for modern OCR applications.
Github MiniMind
MiniMind is an ultra-lightweight, open-source large language model project that boasts a version as small as 1/7,000 the size of GPT‑3 — making it feasible to train on a standard single consumer GPU. It includes not just the model architecture (built from scratch in native PyTorch, without relying on high-level libraries) but also full pipelines for mixed-expert modules (MoE), data cleaning, pre-training, supervised fine-tuning (SFT), LoRA tuning, direct preference optimization (DPO), and model distillation. With a vision toward both researchers and beginners, MiniMind also offers a vision-multimodal variant (MiniMind-V) and emphasizes hands-on learning and democratizing access to large-model creation.
Video Richard Sutton – Father of RL thinks LLMs are a dead end
In this particular episode, host Dwarkesh Patel interviews Richard Sutton, often dubbed the “father of reinforcement learning.” They discuss Sutton’s views on the current trajectory of large language models (LLMs), why he believes LLMs might be a “dead end” in AI research, and how reinforcement learning (RL) still holds promise for building more generalizable, decision-making systems. The conversation combines deep technical insights with philosophical arguments about AI’s future and the kinds of problems that remain unsolved.
Podcast Dwarkesh Podcast
Dwarkesh Podcast, hosted by Dwarkesh Patel, features deeply researched interviews and conversations on artificial intelligence, robotics, cognition, technology, and future thinking. The show often invites experts and thinkers—such as Andrej Karpathy and Richard Sutton—to discuss their perspectives on AGI, reinforcement learning, scaling, and paradigm shifts in AI. Episodes blend technical depth with broader vision and speculation, making it appealing to listeners who want to explore both practical and philosophical dimensions of the AI frontier.
Github Nanochat
nanochat is a minimalist, full-stack implementation of a ChatGPT-style large language model built to be accessible and hackable. The repository combines all phases—tokenization, pretraining, fine-tuning, evaluation, inference, and a simple web UI—into a clean, dependency-light codebase. Karpathy designed it so that with a single script (e.g. speedrun.sh), one can train a conversational model on an 8× H100 node for about four hours (costing ~$100) and immediately interact with it via the web.
Paper POISONING ATTACKS ON LLMS REQUIRE A NEAR-CONSTANT NUMBER OF POISON SAMPLES
In this paper, the authors challenge the prevailing assumption that poisoning large language models (LLMs) becomes harder as model size and training data scale up. Through large-scale experiments on models ranging from 600 million to 13 billion parameters, they show that injecting only a fixed, near-constant number of malicious training samples (e.g. ~250) can successfully implant backdoors regardless of how much clean data the model sees. They further demonstrate that this phenomenon holds both in pretraining and fine-tuning settings, suggesting that backdoor attacks could be more broadly feasible than previously thought. The findings raise serious implications about how we assess and defend against data poisoning in future, larger models.
Paper The Annotated Transformer
The Transformer uses self-attention and feed-forward layers in stacked encoder and decoder blocks to process sequences, unlike previous models relying on RNNs or convolutions, achieving parallel processing and reducing computational complexity.
Paper Video models are zero-shot learners and reasoners
The authors explore how a modern generative video model, Veo 3, demonstrates a surprising level of zero-shot capability across many visual tasks it never explicitly trained for. Tasks include segmentation, edge detection, image editing, understanding physical dynamics, affordance recognition, and even reasoning puzzles like maze solving and symmetry completion. They argue that these emergent abilities suggest video models are evolving toward generalist vision foundation models, much like how LLMs transformed language understanding. Their experiments support a hierarchy of visual competence — from perception to modeling to manipulation to reasoning — showing that Veo 3 can “see, model, act, and reason” with visual content in ways that hint at future unified vision systems.
Blog Lil’Log
Lil’Log (lilianweng.github.io) is the personal blog of Lilian Weng, where she documents her explorations in AI, machine learning, and deep learning research. She publishes well-researched, thoughtful essays on topics like large language models, diffusion models, generative AI, hallucination in LLMs, reinforcement learning, model inference, and more. The posts often include background theory, recent progress, tradeoffs, and references to academic papers — making it a bridge between cutting-edge research and accessible discussion for technical audiences.
Paper DeepSeek-R1: incentivizes reasoning in LLMs through pure reinforcement learning
DeepSeek-R1 demonstrates that large language models can be trained to develop advanced reasoning abilities using reinforcement learning (RL) rather than depending on human-annotated reasoning traces. By building on DeepSeek-V3 Base and using a method called Group Relative Policy Optimization (GRPO), the authors use only final prediction correctness as the reward signal—without instructing the specific reasoning path. The result is that the model learns emergent behaviors like self-reflection, verification steps, and dynamic adaptation of strategy. DeepSeek-R1 outperforms models trained with supervised learning using human demonstrations in tasks involving mathematics, coding, and STEM reasoning. The supplementary materials (ESM1) provide detailed experimental setups, dataset details (including how they handle decontamination of evaluation data), and additional evaluation metrics that support the main claims.
Blog Shunyu Yao’s site
Shunyu Yao’s site showcases his work as a researcher at OpenAI, focusing on “agents” — systems that perceive, reason, and take actions in digital environments. The site features his selected publications, including papers such as ReAct, Tree of Thoughts, Deep Research, and SWE-agent, along with links to repo, talks, benchmarks, and blog posts. It reflects an overarching interest in how language models can be structured not just to predict text, but to act, interact, evaluate, and generalize — moving toward agents that integrate reasoning and action. The writing and work are forward-looking, engaging questions about what constitutes evaluation, how digital environments should be designed, and what the “second half” of AI might look like.
Course Prompt Engineering Guide
Prompting Guide is a comprehensive online resource created by DAIR.AI that gathers up-to-date theory and best practices for designing effective prompts when working with large language models (LLMs). It covers the basics of prompting—such as instructions, context, input data, and output formats—while providing more advanced techniques like function calling, context caching, synthetic dataset generation, and handling model risk (bias, factuality, etc.). Users can find example prompts (for tasks like summarization, reasoning, QA, coding), model-specific guidance, tools, notebooks, and research readings, all intended to help developers, researchers, and AI practitioners get better results and understand LLM capabilities more deeply.
Video Stanford CS229 I Machine Learning I Building Large Language Models (LLMs)
In this lecture from Stanford’s CS229 course, the speaker offers a high-level walkthrough of how large language models are built, touching on more than just model architecture. Key topics include pre-training vs post-training, the importance of data quality and filtering, evaluation metrics (such as perplexity and human benchmarks), and system implementation (e.g. resource constraints like compute and infrastructure). The talk emphasizes that while novel architecture attracts attention, what often makes the biggest difference in practice are data, evaluation, and systems engineering.
Book Build a Large Language Model (From Scratch)
In Build a Large Language Model (From Scratch), Sebastian Raschka leads readers on a hands-on journey through designing, coding, training, and fine-tuning an entire large language model using publicly available tools. Beginning with planning and preparing your data, the book walks you through implementing the model architecture, loading pretrained weights, and adapting the model for tasks like text classification or instruction-following. Even if you’re working on a laptop, it shows how you can build a GPT-style LLM, while also discussing scaling options. With clear explanations, diagrams, code examples, and practical exercises, the book aims to demystify what goes on “inside” generative AI and help you really understand what makes these models work.
Blog Anthropic Engineering Blog
At Anthropic, engineering centers on building AI systems that are reliable, interpretable, and steerable—especially through things like agents and tool integrations. Their team publishes about how they enable Claude (their AI model) to use tools well, optimize performance, and evaluate tools rigorously. They also talk about internal research efforts like scaling interpretability, making features behaving predictably, and improving developer workflows. Their engineering blog shows not just what they build, but how they think about safety, feedback loops, and tooling to support clean, robust, agent-based AI systems.
Blog Thinking Machines Blog
Thinking Machines Lab is an AI research & product company founded in 2025 by Mira Murati. Their blog—titled “Connectionism”—serves as a public window into their work, promising technical posts, research, and code releases. Their mission is to make frontier AI systems easier to understand, customize, and reliably determinable (e.g. tackling nondeterminism in inference), while emphasizing human-AI collaboration, model intelligence, infrastructure quality, and safety.
Podcast a16z Podcast
The a16z Podcast, produced by the venture capital firm Andreessen Horowitz (a16z), explores the intersection of technology, culture, and the future, with a recurring theme that “software eats the world.” Featuring conversations with industry experts, business leaders, and innovative thinkers, the show offers rich insights into emerging trends, startup dynamics, and the evolving digital landscape. Multiple episodes drop each week, making it a timely resource for anyone curious about where tech is headed.
Paper Mixtral of Experts
Mixtral 8x7B introduces a Sparse Mixture of Experts (SMoE) language model built on the Mistral 7B architecture, but enhanced with eight feedforward “experts” per layer. A lightweight router activates just two of these experts per token during inference, giving each token access to a massive 47 billion parameters while only actively using 13 billion—striking an effective balance between capacity and efficiency (47B vs 13B). Trained with a context size of 32K tokens, Mixtral outperforms or matches the performance of Llama 2 70B and GPT-3.5 across benchmarks—especially excelling in math, code generation, and multilingual tasks. Its instruction-tuned variant, Mixtral 8x7B – Instruct, surpasses leading models like GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B-chat on human evaluation tests. Both models are released under an Apache 2.0 license and accompanied by open-source tooling integrations.
Paper Scaling Laws for Neural Language Models
The paper empirically uncovers robust power-law relationships between a language model’s performance (measured via cross-entropy loss) and three core factors: model size, dataset size, and compute budget. Remarkably, these trends hold across more than seven orders of magnitude, while architectural variations like layer depth or width exert minimal impact. The authors further derive simple formulas that predict overfitting behavior and optimal compute allocation—revealing that larger models are significantly more sample-efficient, and that achieving compute-efficient training often means using very large models trained on modest data volumes and stopping well before full convergence.
Paper Attention Is All You Need
The paper introduces the Transformer, an innovative neural network architecture that relies entirely on attention mechanisms—discarding both recurrence and convolution—delivering faster training and superior performance. Tested on machine translation tasks, the model achieves a remarkable 28.4 BLEU score on English-to-German and sets a new state-of-the-art 41.8 BLEU on English-to-French translation, all while training with significantly less compute time. The authors also demonstrate the model’s versatility by applying it successfully to constituency parsing, showcasing its efficiency, parallelism, and adaptability across diverse tasks.
Video A Visual Guide to Quantization
In this visually rich post, Maarten Grootendorst offers an intuitive and deeply illustrated introduction to quantization in the context of large language models, guiding readers through fundamentals like numerical representation (FP32, FP16, BF16, INT8), symmetric vs. asymmetric quantization, and advanced techniques such as dynamic and static post-training quantization. With over 50 custom visuals, he breaks down key concepts like scale and zero-point calculations, quantization error, clipping strategies, and post-training tweaks (e.g., GPTQ, GGUF) to make compression techniques accessible and engaging—even for those new to the subject.
Course Neural Networks by 3Blue1Brown
Grant Sanderson’s (3Blue1Brown) Neural Networks playlist offers an elegant and visual exploration of how neural networks learn—from the basics of node-to-node processing and gradient descent to backpropagation and feature learning—using rich animations and intuitive storytelling that make complex concepts feel both beautiful and approachable.
Course Stanford CS231N Deep Learning for Computer Vision
This free YouTube playlist compiles the latest edition of Stanford’s legendary CS 231N: Deep Learning for Computer Vision course, presenting a thorough, lecture-by-lecture exploration of how neural networks interpret and generate images. From core concepts like convolutional layers and optimization techniques to modern techniques in object detection, generative modeling, and explainability, the course is tailored for both students and professionals looking to deepen their understanding of cutting-edge computer vision methods.
Blog Simon Willison’s Weblog
Simon Willison—co-creator of the Django web framework and creator of the open-source tool Datasette—has been blogging since 2002 about web development, data journalism, AI tooling, and LLMs. His site features fast, insightful entries—including project demos like EmbeddingGemma and semantic tools like the “Semantic Galaxy”—plus thoughtful reflections on topics like prompt engineering and AI agents. Willison also publishes both free and paid newsletters, maintains all his posts on a custom Django-powered site, and continues contributing to open-source and tech communities.
Blog Andrej Karpathy
Andrej Karpathy’s Blog is the personal website of Andrej Karpathy, an AI researcher and educator known for his work in deep learning and large language models. The blog features a mix of technical content, personal reflections, and educational resources. Notably, Karpathy offers a self-hosted, minimalist blog experience, emphasizing simplicity and directness in his writing. His posts cover topics such as the philosophy of technology, personal anecdotes, and insights into AI development. Additionally, Karpathy has launched Eureka Labs, an AI+Education company, and shares educational videos on his YouTube channel, focusing on deep learning and large language models.
Video What is ChatGPT doing...and why does it work?
In this engaging AMA-style presentation, Stephen Wolfram breaks down the core magic behind ChatGPT by illustrating how it constructs language “one word at a time.” He walks viewers through the foundational mechanics—how ChatGPT predicts the next most probable word based on patterns seen in massive text corpora—and explains the surprisingly simple yet powerful processes that allow it to generate human-like responses. Using intuitive analogies and step-by-step reasoning, the video demystifies how ChatGPT models language, why its responses can feel so coherent, and what “makes it work” beyond rote memorization.
Video Deep Dive into LLMs like ChatGPT
In this three-and-a-half-hour lecture, Andrej Karpathy—former Tesla AI lead and OpenAI founding member—walks a general audience through the inner workings of large language models like ChatGPT. He delves into the full training stack: from preprocessing massive internet text corpora and tokenization to model structure, inference dynamics, fine-tuning techniques, hallucinations, reinforcement learning, memory systems, and model alignment strategies. With clarity and insight, Karpathy shares mental models to understand how LLMs think, their limitations, and how to use them most effectively—making complex AI concepts accessible and actionable.
Video Andrej Karpathy
Andrej Karpathy’s YouTube channel is a treasure trove for anyone fascinated by AI and deep learning. As a former AI leader at Tesla and founding member of OpenAI, Karpathy brings unparalleled expertise to his content—ranging from foundational tutorials like building GPT models from scratch to deep dives into how large language models actually work. His videos are insightful, technical yet accessible, and feature a unique mix of theory, code walkthroughs, and real-world AI application, making complex concepts feel intuitive and exciting.
Course Practical Deep Learning
A free course designed for people with some coding experience, who want to learn how to apply deep learning and machine learning to practical problems.
Course DeepLearning.ai
DeepLearning.AI is a pioneering education technology company founded by AI innovator Andrew Ng in 2017. Its mission is to equip a global audience—from aspiring AI professionals to non-technical learners—with high-quality, hands-on training in artificial intelligence. Through widely respected programs on Coursera like the Deep Learning Specialization and AI for Everyone, DeepLearning.AI has helped millions build tangible machine learning skills, supported by a collaborative community and a growing ecosystem of AI tools and partnerships.
