Many IT professionals now work with chatbots every day: to write scripts, explain a log or draft documentation. But if we don’t understand what goes on behind these tools, we either trust them too much or fail to use their capabilities properly. This tutorial explains how large language models work, without complicated math.
Tokens and next-token prediction
A language model does not see text word by word; it breaks it into small pieces called tokens. A token can be a whole word, part of a word or a punctuation mark. In many models, Persian text is split into more tokens than English text of the same length; this affects speed, the cost of paid services, and how much text fits into a single request.
The core of an LLM sounds simple: given the text so far, what is the most likely next token? The model generates one token, adds it to the text and predicts again. These models are built on an architecture called the transformer, which uses an “attention” mechanism to weigh how much each part of the text relates to the others. A setting called temperature controls how random the choices are: lower values give more predictable answers, higher values more varied ones.
How is a model trained?
- Pre-training: the model is trained on an enormous amount of text and learns language patterns and general knowledge statistically. This stage requires large numbers of GPUs and very high cost.
- Instruction tuning: the model is trained on question-and-answer examples so that it follows instructions.
- Alignment with human feedback: methods based on human evaluation make the answers more helpful and safer.
Context window and memory
Every model has a context window: the maximum number of tokens it can see at once in a conversation, including your instructions, attached files and earlier replies. On its own, the model remembers nothing from previous conversations unless the service provides a separate memory feature. To answer questions based on a company’s own documents, a technique called RAG is commonly used: the system first searches for the relevant parts of the documents and passes them to the model along with the question.
Key limitations
- Hallucination: because the model generates “likely” text, not necessarily “correct” text, it may confidently invent a command, a source or a number.
- Knowledge cutoff: the model’s knowledge is limited to its training data, and without access to search it knows nothing about recent changes.
- Weakness in precise calculation: it can make mistakes in arithmetic, counting and multi-step logic unless it uses a helper tool.
- Bias: one-sided patterns in the training data can be repeated in its answers.
- Prompt injection: text the model reads, such as an email or a web page, may contain malicious instructions that change the model’s behavior; this is a serious risk for systems that connect a model to email or internal tools.
Always check a command or setting suggested by the model, such as a firewall rule, against the official documentation, and run it in a test environment first.
Cloud model or local model?
You can use models through cloud services, or run some open-weight models on your company’s own server. A cloud service is usually more capable and less hassle, but your data leaves the company, and access from Iran may be limited. A local model gives you more control over your data, but it needs suitable hardware (usually a GPU with enough memory) and technical maintenance. Check the license of each model separately as well.
Checklist
- Treat model output as a draft and verify important information against the original source.
- Enter confidential information only into a service whose data retention terms you have reviewed.
- If you connect a model to email, files or internal tools, limit its access to the minimum required.
- For questions about company documents, consider RAG instead of relying on the model’s own knowledge.
- Before running a local model, estimate the hardware, licensing and maintenance needs.
If you plan to run a language model on an internal server or virtual machine, ITPlus724 engineers can design and implement its server, network, access security and backup.