CTRL K

Model Metrics

When training machine learning models, hyperparameters such as learning rate, epochs, and batch size, along with evaluation metrics like precision and recall, are fundamental components that significantly influence model performance. These elements play a critical role in optimizing the training process and ensuring robust outcomes.
Our training configuration framework is designed to seamlessly handle these essential parameters, This approach enables users to focus on their primary machine learning objectives.
Additionally, the integration of MLflow provides a structured and user-friendly logging interface. The training logs are presented in a clear, intuitive format, allowing users to effortlessly monitor and evaluate model performance throughout the training process.

Matrix :

A detailed explanation of model evaluation metrics across different model types, including Object Detection, Segmentation, and LLM models. This includes an overview of the key metrics available in the View Experiment section and their significance in assessing model performance.

Object Detection :

Loss_cls (Classification Loss) – Measures how well the model classifies objects into the correct categories.
Loss_bbox (Bounding Box Regression Loss) – Evaluates the accuracy of predicted bounding boxes by comparing them to ground truth boxes.
Loss_bbox (Bounding Box Regression Loss) – The total loss during training, combining all individual losses.
coco/bbox_mAP (mean Average Precision): This is the mean of the Average Precision (AP) across all object categories in the COCO dataset for bounding box predictions. It gives an overall performance metric for how well the model detects objects, considering both precision and recall.
coco/bbox_mAP_50 (mAP at IoU threshold of 0.5): This measures the mean Average Precision when the Intersection over Union (IoU) threshold is set to 0.5. It evaluates the model’s ability to correctly localize objects with at least 50% overlap with the ground truth bounding boxes.
coco/bbox_mAP_75 (mAP at IoU threshold of 0.75): This metric measures mean Average Precision at a higher IoU threshold of 0.75, providing a stricter evaluation of the model’s bounding box prediction accuracy, focusing on more precise localizations.
coco/bbox_mAP_s (mAP for small objects): This measures the mean Average Precision for small objects (based on their area or size). It evaluates how well the model detects smaller objects in the image.
coco/bbox_mAP_m (mAP for medium objects): These measures mean Average Precision for medium-sized objects, which fall between small and large object categories. It evaluates the model’s ability to detect objects of moderate size.
coco/bbox_mAP_l (mAP for large objects): These measures mean Average Precision for large objects, focusing on how well the model detects objects that occupy a larger portion of the image.

Segmentation :

train_loss (Total Training Loss): The total training loss is the sum of all the individual loss components (cross-entropy loss, dice loss, etc.) during training. It reflects how well the model is learning the segmentation task.
Train_decode_loss_ce (Cross-Entropy Loss during Training): This is the cross-entropy loss computed during training for the segmentation task. It measures the difference between the predicted class probabilities and the ground truth labels, penalizing incorrect class predictions for each pixel.
Train_decode_acc_seg (Segmentation Accuracy during Training): This metric represents the pixel-wise accuracy for the segmentation predictions during training. It calculates the percentage of pixels in the predicted segmentation map that match the ground truth.
Val_aAcc (Overall Accuracy during Validation): Overall accuracy measures the proportion of correctly classified pixels across all classes during validation. It’s an aggregate metric indicating how well the model performs across the entire validation dataset.
Val_mIoU (mean Intersection over Union during Validation): mean IoU measures the average intersection-over-union across all classes during validation. It evaluates the accuracy of predicted pixel regions for each class by comparing the predicted segmentation mask to the ground truth.
val_mAcc (mean Accuracy during Validation): mean accuracy calculates the average pixel accuracy per class during validation, focusing on how well the model detects each individual class.

LLM :

train_loss (Training Loss): This represents the loss calculated during training, for next-token prediction or masked token prediction. A lower train_loss indicates that the model is learning effectively, but excessively low values may suggest overfitting.
eval_loss (Evaluation Loss / Validation Loss): This is the loss computed on a separate validation dataset, used to assess how well the model generalizes to unseen data. A lower eval_loss compared to train_loss suggests good generalization, while a significantly higher eval_loss may indicate overfitting.

Model Metrics

When training machine learning models, hyperparameters such as learning rate, epochs, and batch size, along with evaluation metrics like precision and recall, are fundamental components that significantly influence model performance. These elements play a critical role in optimizing the training process and ensuring robust outcomes.

Our training configuration framework is designed to seamlessly handle these essential parameters, This approach enables users to focus on their primary machine learning objectives.

Additionally, the integration of MLflow provides a structured and user-friendly logging interface. The training logs are presented in a clear, intuitive format, allowing users to effortlessly monitor and evaluate model performance throughout the training process.

Matrix :

A detailed explanation of model evaluation metrics across different model types, including Object Detection, Segmentation, and LLM models. This includes an overview of the key metrics available in the View Experiment section and their significance in assessing model performance.

Object Detection :

Loss_cls (Classification Loss) – Measures how well the model classifies objects into the correct categories.

Loss_bbox (Bounding Box Regression Loss) – Evaluates the accuracy of predicted bounding boxes by comparing them to ground truth boxes.

Loss_bbox (Bounding Box Regression Loss) – The total loss during training, combining all individual losses.

coco/bbox_mAP (mean Average Precision): This is the mean of the Average Precision (AP) across all object categories in the COCO dataset for bounding box predictions. It gives an overall performance metric for how well the model detects objects, considering both precision and recall.

coco/bbox_mAP_50 (mAP at IoU threshold of 0.5): This measures the mean Average Precision when the Intersection over Union (IoU) threshold is set to 0.5. It evaluates the model’s ability to correctly localize objects with at least 50% overlap with the ground truth bounding boxes.

coco/bbox_mAP_75 (mAP at IoU threshold of 0.75): This metric measures mean Average Precision at a higher IoU threshold of 0.75, providing a stricter evaluation of the model’s bounding box prediction accuracy, focusing on more precise localizations.

coco/bbox_mAP_s (mAP for small objects): This measures the mean Average Precision for small objects (based on their area or size). It evaluates how well the model detects smaller objects in the image.

coco/bbox_mAP_m (mAP for medium objects): These measures mean Average Precision for medium-sized objects, which fall between small and large object categories. It evaluates the model’s ability to detect objects of moderate size.

coco/bbox_mAP_l (mAP for large objects): These measures mean Average Precision for large objects, focusing on how well the model detects objects that occupy a larger portion of the image.

Segmentation :

train_loss (Total Training Loss): The total training loss is the sum of all the individual loss components (cross-entropy loss, dice loss, etc.) during training. It reflects how well the model is learning the segmentation task.

Train_decode_loss_ce (Cross-Entropy Loss during Training): This is the cross-entropy loss computed during training for the segmentation task. It measures the difference between the predicted class probabilities and the ground truth labels, penalizing incorrect class predictions for each pixel.

Train_decode_acc_seg (Segmentation Accuracy during Training): This metric represents the pixel-wise accuracy for the segmentation predictions during training. It calculates the percentage of pixels in the predicted segmentation map that match the ground truth.

Val_aAcc (Overall Accuracy during Validation): Overall accuracy measures the proportion of correctly classified pixels across all classes during validation. It’s an aggregate metric indicating how well the model performs across the entire validation dataset.

Val_mIoU (mean Intersection over Union during Validation): mean IoU measures the average intersection-over-union across all classes during validation. It evaluates the accuracy of predicted pixel regions for each class by comparing the predicted segmentation mask to the ground truth.

val_mAcc (mean Accuracy during Validation): mean accuracy calculates the average pixel accuracy per class during validation, focusing on how well the model detects each individual class.

LLM :

train_loss (Training Loss): This represents the loss calculated during training, for next-token prediction or masked token prediction. A lower train_loss indicates that the model is learning effectively, but excessively low values may suggest overfitting.

eval_loss (Evaluation Loss / Validation Loss): This is the loss computed on a separate validation dataset, used to assess how well the model generalizes to unseen data. A lower eval_loss compared to train_loss suggests good generalization, while a significantly higher eval_loss may indicate overfitting.