TinyTextEngine is a compact, from-scratch implementation of a Transformer-based language model. It includes custom attention, tokenization, and training pipelines to demonstrate how core Large Language Model (LLM) concepts work at a smaller scale.
The Small Language Model project is designed as a learning-oriented implementation of a language model inspired by modern LLM architectures, but at a much smaller and more approachable scale.
It walks through the entire lifecycle of a language model:
The project is notebook-driven, making it ideal for students, researchers, and engineers who want to understand how language models work under the hood.
Transformer-style Architecture
Implements core ideas such as embeddings, attention mechanisms, and sequence modeling.
From-Scratch Learning Approach
Avoids heavy abstractions to keep the logic transparent and easy to follow.
Jupyter Notebook Workflow
Step-by-step notebooks guide you through every stage of the pipeline.
Evaluation & Visualization
Includes performance metrics and visual reports for inference and efficiency analysis.
Lightweight & Reproducible
Designed to run on modest hardware with a simple dependency setup.