Softmax

Softmax is a small mathematical function that turns a list of raw scores into percentages that add up to 100. If a word's attention scores for three neighbors are something like 5, 2, and 1, softmax converts them into proportions such as 70 percent, 20 percent, 10 percent.

It shows up wherever a model needs to turn "how much of each" into clean weights, most notably in self-attention (how much to attend to each word) and in producing the final probabilities over possible next words. You can think of softmax as the step that says "given these scores, here is how to split your attention or your bet."