Your plan is good as a starting point, but I would change several things before calling it “best.” For microbiology + 20–150 objects per image + accurate masks, the biggest risks are over-augmentation, small validation set, dense-object NMS/settings, and mask resolution.

Ultralytics docs confirm that YOLO11-seg supports instance segmentation and that yolo11l-seg.pt is a valid pretrained model; YOLO11l-seg has much more compute than m/s models, so it is reasonable for accuracy if your 3060 can handle it. (Ultralytics Docs)

Main problems in your config

1. Too much augmentation

For microscopy, your objects are already dense. Heavy mosaic=1.0, copy_paste=0.5, mixup=0.15, scale=0.7, hsv_s=0.8, hsv_v=0.5 can create unrealistic cells and hurt mask boundary quality.

I would reduce:

python
mosaic=0.3       # not 1.0
copy_paste=0.15 # not 0.5
mixup=0.0       # or max 0.05
scale=0.4
hsv_s=0.3
hsv_v=0.25

Ultralytics describes mosaic as combining four images, mixup as blending images/labels, and copy-paste as increasing segmentation object instances; these are useful, but need tuning per dataset. (Ultralytics Docs)

2. multi_scale=True is suspicious

Current Ultralytics docs describe multi_scale as a float, where for example 0.25 means training randomly from 0.75x to 1.25x image size. (Ultralytics Docs)

So do not use:

python
multi_scale=True

Use either:

python
multi_scale=0.0

or a small value:

python
multi_scale=0.15

For small dense objects, I would start with multi_scale=0.0 or 0.10–0.15, not aggressive multi-scale.

3. dropout=0.1 is not useful here

Ultralytics lists dropout as a regularization setting for classification tasks, not normal segmentation training. (Ultralytics Docs)
Remove it.

4. erasing=0.4 is also not for segmentation

Current docs list erasing as a classification augmentation. (Ultralytics Docs)
Remove it for YOLO-seg.

5. mask_ratio=2 is good for your case

Default is mask_ratio=4; lower means higher-resolution masks during training, which is better for small/precise microbiology masks but uses more memory. (Ultralytics Docs)

For your goal “draw them well,” I would keep:

python
mask_ratio=2

If VRAM allows and objects are very tiny, you can test mask_ratio=1, but it may be too heavy on a 3060.

6. overlap_mask=True should be tested

Ultralytics says overlap_mask=True merges masks for training, with smaller masks overlaid on larger masks when overlap exists. (Ultralytics Docs)

For touching/overlapping cells, test both:

python
overlap_mask=True

and

python
overlap_mask=False

Do a short 80–120 epoch comparison. If cells overlap a lot and mask separation matters, False may give cleaner instance boundaries.

---

Better training config

I would start with this:

python
from ultralytics import YOLO

model = YOLO("yolo11l-seg.pt")

model.train(
    data="custom_dataset.yaml",

    # Resolution
    imgsz=1280,

    # Hardware: make sure this is the 3060, not the 2060
    device=0,
    batch=4,
    workers=4,
    amp=True,
    cache="ram",

    # Duration
    epochs=350,
    patience=70,
    close_mosaic=30,

    # Optimizer
    optimizer="AdamW",
    lr0=0.001,
    lrf=0.01,
    warmup_epochs=5,
    cos_lr=True,
    weight_decay=0.0005,

    # Augmentation - safer for microscopy
    mosaic=0.3,
    copy_paste=0.15,
    copy_paste_mode="flip",
    mixup=0.0,
    degrees=45.0,
    flipud=0.5,
    fliplr=0.5,
    scale=0.4,
    translate=0.05,
    hsv_h=0.01,
    hsv_s=0.3,
    hsv_v=0.25,

    # Mask quality
    overlap_mask=True,   # also test False
    mask_ratio=2,
    multi_scale=0.0,     # or 0.10/0.15 after baseline

    # Dense objects
    max_det=300,

    # Saving
    save=True,
    save_period=50,
    plots=True,
    project="runs/cells",
    name="main_train_v2",
)

Then fine-tune with augmentation mostly off:

python
from ultralytics import YOLO

model = YOLO("runs/cells/main_train_v2/weights/best.pt")

model.train(
    data="custom_dataset.yaml",
    imgsz=1280,
    device=0,
    batch=4,
    workers=4,
    amp=True,
    cache="ram",

    epochs=80,
    patience=30,

    optimizer="AdamW",
    lr0=0.0002,
    lrf=0.01,
    warmup_epochs=2,
    cos_lr=True,
    weight_decay=0.0005,

    mosaic=0.0,
    copy_paste=0.0,
    mixup=0.0,
    degrees=15.0,
    flipud=0.5,
    fliplr=0.5,
    scale=0.2,
    translate=0.03,
    hsv_h=0.005,
    hsv_s=0.15,
    hsv_v=0.15,

    overlap_mask=True,
    mask_ratio=2,
    multi_scale=0.0,
    max_det=300,

    save=True,
    plots=True,
    project="runs/cells",
    name="finetune_clean_masks",
)

This second phase often improves clean mask edges because the model stops seeing artificial mosaic/copy-paste compositions near the end.

---

About your GPUs: 2060 + 3060

Use the 3060 only for this training unless you are sure both cards can handle the per-GPU batch. Ultralytics supports multi-GPU training with device=[0, 1], but it distributes work across GPUs; it does not combine VRAM into one 18 GB GPU. (Ultralytics Docs)

Also, 2060 + 3060 is mismatched. The 2060 may slow training or cause OOM first. Best practical setup:

python
device=0  # only if GPU 0 is the 3060

Check with:

bash
nvidia-smi

If GPU 0 is the 2060 and GPU 1 is the 3060, use:

python
device=1

---

Validation split: improve this if possible

900 train / 100 val is acceptable, but for microscopy I would prefer:

text
800 train / 100 val / 100 test

or:

text
850 train / 150 val

Most important: split by slide / sample / experiment / day, not random crops from the same slide. If train and val contain very similar fields from the same source image, your validation score will look good but real performance may be worse.

---

Prediction settings matter for “detect all”

After training, do not use default prediction settings blindly. Default conf=0.25 may miss weak/small objects; Ultralytics docs say conf filters low-confidence objects and max_det limits detections per image. (Ultralytics Docs)

For dense cells, try:

python
model = YOLO("runs/cells/finetune_clean_masks/weights/best.pt")

results = model.predict(
    source="test_images",
    imgsz=1280,
    conf=0.05,
    iou=0.6,
    max_det=300,
    retina_masks=True,
    save=True,
    save_txt=True,
)

Use retina_masks=True for high-resolution masks matching original image size. (Ultralytics Docs)

Then tune:

text
More missed cells     -> lower conf: 0.03–0.10
Too many duplicates   -> lower iou: 0.45–0.60
Merged close cells    -> higher iou: 0.65–0.80 sometimes helps keep close detections
Too few detections    -> increase max_det

---

My recommendation

Your current setup is not optimal. I would improve it like this:

1. Keep imgsz=1280.
2. Keep mask_ratio=2.
3. Remove dropout and erasing.
4. Reduce mosaic/copy-paste/mixup strongly.
5. Do a clean fine-tuning phase with mosaic/copy-paste off.
6. Use the 3060 only.
7. Test overlap_mask=True vs False.
8. Use retina_masks=True, low conf, and max_det=300 during prediction.

For best results, run three experiments:

text
A: yolo11m-seg, moderate aug, mask_ratio=2
B: yolo11l-seg, moderate aug, mask_ratio=2
C: best of A/B, fine-tune clean masks

Pick based on seg mAP, recall, and visual mask quality, not box mAP only.