Hyperparameter
Hyperparameter
What is a hyperparameter?
- A setting people choose before training
- Learning rate, number of layers, etc.
- Has a major effect on performance
Hyperparameters must already be fixed the moment you press the training button, so in practice they shape the entire experiment design, not just the model code. Without touching a single line of the architecture, raising the learning rate 10x makes the loss diverge and halts training, while lowering it 10x turns a three-day run into a three-week schedule. In other words, your team’s GPU hours, cloud costs, and delivery dates are decided here first.
Values fall into three broad groups: learning rate, batch size, and number of epochs, which set how the model learns; number of layers, hidden units, and LoRA rank, which set the model architecture; and dropout rate and weight decay, which curb overfitting. Learning rate is usually swept on a log scale between 1e-3 and 1e-5, and once combinations pass a few dozen, teams switch from grid search to random search or Bayesian optimization. The basic rule is to draw the line this way: values updated automatically through backpropagation are parameters, and values that people fix are hyperparameters. LLM temperature and top-p are inference-time settings, not training settings, so they are managed separately.
Tuning is not something to run indefinitely; it has to stop within a budget. Split the data 6:2:2 into training, validation, and test sets and compare using validation performance only. If the dataset is small, in the low thousands of records, confirm with the average of 5-fold cross-validation. Cap trials at 20–30 and document a stopping rule in advance, for example dropping a direction if the validation metric fails to improve by even 0.5 percentage points over ten consecutive trials. That way who achieved what with which settings can be reproduced later.
The most common mistake in practice is tuning on test data. If you compare dozens of combinations on the same data without separating validation and test sets, that number is a score earned by checking against the answers, and performance collapses the moment the model goes into production. “We used the values straight from the paper” is another common misconception: with a different data size and batch size, the same learning rate behaves completely differently. If you miss that doubling the batch size means the learning rate needs adjusting too, and go looking for the cause in the model architecture, you can lose weeks outright.
Sungil KimA KAIST and Samsung Electronics alumnus and global pitching expert who coached the winning teams at the UN Tourism and Select USA IR pitch competitions
Jaryong KooA marketing PhD and leading authority on AI, data analytics, and personal brandingThe term is now used for values that people set, rather than the model learning them, when designing and training a machine learning model.
Consider a hypothetical case. Company A, an auto-parts maker with 240 employees and annual revenue of around KRW 90 (≈ US$1) billion, had spent three months on an anomaly-detection model for its press equipment. The report showed 92% validation accuracy, but on the floor there were more than 30 false alarms a day, and shift supervisors were working with the alerts switched off entirely.
- When Kim, a manager on the data analytics team, opened the 47 experiment logs from six weeks of work, every combination had been compared only on the same test-set score, and no validation set had been split off at all.
- An outside advisory consultant pointed out, “This 92% is a score earned by checking against the answers,” and the team began by re-splitting the 80,000 equipment log records into training, validation, and test sets at 6:2:2.
- The production engineering lead set a hard limit: “Finish in two weeks on a single GPU.” So the team capped the search at 24 combinations and used random search to scan only learning rates from 1e-3 to 1e-5 and batch sizes of 32, 64, and 128.
- When loss diverged in the combination with the batch size raised to 128, Kim re-ran the experiment after cutting the learning rate to a quarter at the same time, and raising dropout from 0.2 to 0.4 also brought overfitting under control.
- The model retrained with the final settings reached 89% test accuracy, a 3-point drop in the reported figure, but on-site false alarms fell from 30 a day to 6, and within two weeks the shift supervisors turned the alerts back on.