脸部LoRA训练项目：素材流水线 + Gradio控制台 + RunPod云端续炼方案（v7续炼完成）
This commit is contained in:
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,305 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
merge_fixed.py — 原图 + AI修改图 无缝融合工具
|
||||
============================================
|
||||
背景：ComfyUI inpainting 移除合影中的其他人物时，即使指定"非修改区域不得像素漂移"，
|
||||
全图清晰度仍不可避免损失。本工具把：
|
||||
- 原图的"未修改区域"（清晰度完好）
|
||||
- 修改图的"修改区域"（人物被干净移除）
|
||||
自动合并，过渡自然无痕迹。
|
||||
|
||||
用法：
|
||||
python merge_fixed.py <原图目录> <修改图目录> <输出目录> [--feather 25] [--threshold 20]
|
||||
|
||||
匹配规则：
|
||||
- 原图 IMG_0463.jpg（任意后缀 jpg/png/webp/jpeg）
|
||||
- 修改图 IMG_0463*.png（以原图文件名开头、.png 结尾、中间任意字符）
|
||||
- 一张原图可有多张修改图，每张都独立合并
|
||||
- 输出：{原图stem}_{修改图中间部分}.png
|
||||
|
||||
算法：
|
||||
1. 修改图 resize 到原图尺寸
|
||||
2. Lab 色彩空间差异图 → 高斯模糊去噪 → 阈值 → 闭运算+膨胀 = 修改区 mask
|
||||
3. 羽化混合：result = 原图*(1-alpha) + 修改图*alpha（alpha 由 mask 高斯模糊得到）
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
||||
|
||||
# 复用 face_checker 的人脸检测（目标人物面部保护）
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import face_checker as fc
|
||||
|
||||
IMG_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".bmp"}
|
||||
_fc_det = None
|
||||
|
||||
|
||||
def get_face_detector():
|
||||
global _fc_det
|
||||
if _fc_det is None:
|
||||
_fc_det = fc.FaceDetector()
|
||||
return _fc_det
|
||||
|
||||
|
||||
def protect_faces(alpha, mod_rgb, expand=1.35, margin=40):
|
||||
"""
|
||||
面部强制保护（保险）：检测修改图中残留的人脸（=目标人物，被移除者已被 inpaint 掉），
|
||||
把这些人脸区域的 alpha 置 0（100% 用原图像素，零改变）。
|
||||
通用规则（有限修改→原图）已覆盖大部分，此项兜底确保面部绝对不变。
|
||||
"""
|
||||
det = get_face_detector()
|
||||
faces = fc._detect_faces_fast(mod_rgb, det)
|
||||
if not faces:
|
||||
return alpha
|
||||
for f in faces:
|
||||
x, y, w, h = f[0], f[1], f[2], f[3]
|
||||
cw, ch = w * expand, h * expand
|
||||
x0 = max(0, int(x - (cw - w) / 2) - margin)
|
||||
y0 = max(0, int(y - (ch - h) / 2) - margin)
|
||||
x1 = min(alpha.shape[1], int(x + w + (cw - w) / 2) + margin)
|
||||
y1 = min(alpha.shape[0], int(y + h + (ch - h) / 2) + margin)
|
||||
alpha[y0:y1, x0:x1] = 0.0
|
||||
return alpha
|
||||
|
||||
|
||||
def build_alpha_from_mask(mask_rgb, feather=20):
|
||||
"""
|
||||
从 ComfyUI inpaint 蒙版生成 alpha：白色区域 = 被修改区（100% 修改图），黑色 = 原图。
|
||||
蒙版是最准确的"修改区"标记（用户在 ComfyUI 画的就是要移除的人物），
|
||||
比 diff 检测可靠 100 倍——紧贴人物的残影完美解决。
|
||||
feather: 蒙版边缘羽化像素
|
||||
"""
|
||||
gray = cv2.cvtColor(mask_rgb, cv2.COLOR_RGB2GRAY).astype(np.float32) / 255.0
|
||||
# 蒙版白色→1（修改图），黑色→0（原图）
|
||||
alpha = gray
|
||||
if feather > 0:
|
||||
k = feather * 2 + 1
|
||||
alpha = cv2.GaussianBlur(alpha, (k, k), 0)
|
||||
return alpha
|
||||
|
||||
|
||||
def find_mask(orig_path, mask_dir):
|
||||
"""按文件名匹配蒙版：{stem}*mask*.png 或 {stem}*.png（取 mask 关键字优先）"""
|
||||
stem = Path(orig_path).stem
|
||||
cands = sorted(p for p in mask_dir.iterdir()
|
||||
if p.suffix.lower() == ".png" and p.name.startswith(stem))
|
||||
# 优先带 mask 关键字的
|
||||
for p in cands:
|
||||
if "mask" in p.name.lower():
|
||||
return p
|
||||
return cands[0] if cands else None
|
||||
|
||||
|
||||
def build_alpha(orig_rgb, mod_rgb, blur_kernel=15, threshold=4.0, min_area_ratio=0.005,
|
||||
softness=1.5, dilate=60, body_extend=2.2):
|
||||
"""
|
||||
融合权重 alpha（0=原图像素, 1=修改图像素）——双图人脸差集定位被移除人物：
|
||||
1. 原图人脸 - 修改图人脸 = 被移除人物（合影中被清掉的人）
|
||||
2. 被移除人物区域（脸框 + 向下 body_extend 倍覆盖身体）= 修改区
|
||||
3. 目标人物（修改图残留的最大人脸）紧贴框 = 保留区（强制原图）
|
||||
4. diff 高值连通块（排除保留区）兜底其他修改
|
||||
5. 膨胀覆盖边缘残影
|
||||
紧贴合影场景：被移除人物与目标紧贴时，靠"人脸差集"精确定位，不靠 diff 幅度。
|
||||
"""
|
||||
det = get_face_detector()
|
||||
orig_faces = fc._detect_faces_fast(orig_rgb, det)
|
||||
mod_faces = fc._detect_faces_fast(mod_rgb, det)
|
||||
h, w = orig_rgb.shape[:2]
|
||||
|
||||
def matched(of, mfaces):
|
||||
ox, oy = of[0] + of[2] / 2, of[1] + of[3] / 2
|
||||
return any(abs(ox - (mf[0] + mf[2] / 2)) < of[2] * 0.8 and abs(oy - (mf[1] + mf[3] / 2)) < of[3] * 0.8
|
||||
for mf in mfaces)
|
||||
|
||||
removed_faces = [f for f in orig_faces if not matched(f, mod_faces)]
|
||||
# 过滤误检小脸（背景物体/花纹被当脸）：只保留足够大的人脸
|
||||
# 真实人物脸 ≥ 图面积 0.05% 或短边 ≥ 40px
|
||||
min_face_area = max(1, int(h * w * 0.0005))
|
||||
removed_faces = [f for f in removed_faces
|
||||
if f[2] * f[3] >= min_face_area and min(f[2], f[3]) >= 40]
|
||||
|
||||
# 被移除人物区（脸 + 身体向下扩展）
|
||||
mod_mask = np.zeros((h, w), np.uint8)
|
||||
for f in removed_faces:
|
||||
x, y, fw, fh = [int(v) for v in f[:4]]
|
||||
x0 = max(0, int(x - fw * 0.4)); x1 = min(w, int(x + fw * 1.4))
|
||||
y0 = max(0, int(y - fh * 0.3)); y1 = min(h, int(y + fh * body_extend))
|
||||
mod_mask[y0:y1, x0:x1] = 1
|
||||
|
||||
# 保留区：目标人物最大脸，紧贴框（左右不扩，防侵入紧贴的被移除人物）
|
||||
keep_mask = np.zeros((h, w), np.uint8)
|
||||
if mod_faces:
|
||||
big = max(mod_faces, key=lambda f: f[2] * f[3])
|
||||
x, y, fw, fh = [int(v) for v in big[:4]]
|
||||
x0, x1 = max(0, x), min(w, x + fw)
|
||||
y0, y1 = max(0, int(y - fh * 0.35)), min(h, int(y + fh * 1.2))
|
||||
keep_mask[y0:y1, x0:x1] = 1
|
||||
|
||||
# diff 兜底连通块（排除保留区）。无被移除人物时用高阈值（保守，防降质误判）
|
||||
o_lab = cv2.cvtColor(orig_rgb, cv2.COLOR_RGB2LAB).astype(np.float32)
|
||||
m_lab = cv2.cvtColor(mod_rgb, cv2.COLOR_RGB2LAB).astype(np.float32)
|
||||
diff_s = cv2.GaussianBlur(np.abs(o_lab - m_lab).mean(axis=2), (blur_kernel, blur_kernel), 0)
|
||||
eff_threshold = threshold if removed_faces else max(threshold, 15.0)
|
||||
base = ((diff_s > eff_threshold) & ~keep_mask.astype(bool)).astype(np.uint8)
|
||||
num, labels, stats, _ = cv2.connectedComponentsWithStats(base, 8)
|
||||
min_area = max(1, int(min_area_ratio * h * w))
|
||||
filtered = np.zeros_like(base)
|
||||
for i in range(1, num):
|
||||
if stats[i, cv2.CC_STAT_AREA] >= min_area:
|
||||
filtered[labels == i] = 1
|
||||
|
||||
core = np.maximum(filtered, mod_mask)
|
||||
if dilate > 0:
|
||||
core = cv2.dilate(core, np.ones((dilate, dilate), np.uint8))
|
||||
alpha = core.astype(np.float32)
|
||||
if softness > 0:
|
||||
k = int(softness * 6) * 2 + 1
|
||||
alpha = cv2.GaussianBlur(alpha, (k, k), 0)
|
||||
return alpha
|
||||
"""
|
||||
融合权重 alpha（0=原图像素, 1=修改图像素）：
|
||||
- 固定阈值判定明显修改区（人物移除 diff 高，有限降质 diff 低，区分度 ~20x）
|
||||
- diff > 阈值 → alpha≈1（100% 修改图，修改区内部零残影）
|
||||
- diff < 阈值 → alpha≈0（100% 原图，目标人物/有限修改区零改变）
|
||||
- 修改区膨胀 dilate 像素（默认 75）：覆盖人物边缘 10-40px 的浅残影带
|
||||
（残影带 diff 接近未修改区，无法用阈值检测，必须靠膨胀）
|
||||
- 陡 sigmoid 边缘过渡防硬边
|
||||
"""
|
||||
o = cv2.cvtColor(orig_rgb, cv2.COLOR_RGB2LAB).astype(np.float32)
|
||||
m = cv2.cvtColor(mod_rgb, cv2.COLOR_RGB2LAB).astype(np.float32)
|
||||
diff = np.abs(o - m).mean(axis=2)
|
||||
diff_s = cv2.GaussianBlur(diff, (blur_kernel, blur_kernel), 0)
|
||||
|
||||
alpha = 1.0 / (1.0 + np.exp(-(diff_s - threshold) / softness))
|
||||
|
||||
# 修改区膨胀：覆盖人物边缘残影带
|
||||
if dilate > 0:
|
||||
kernel = np.ones((dilate, dilate), np.uint8)
|
||||
core = (alpha > 0.5).astype(np.uint8)
|
||||
core = cv2.dilate(core, kernel)
|
||||
alpha = np.maximum(alpha, core.astype(np.float32))
|
||||
return alpha
|
||||
|
||||
|
||||
def seamless_merge(orig_rgb, mod_rgb, alpha):
|
||||
"""
|
||||
连续 alpha 混合：result = orig*(1-alpha) + mod*alpha。
|
||||
alpha 已是平滑的浮点图（sigmoid 过渡），无需额外羽化。
|
||||
"""
|
||||
a = alpha[..., None].astype(np.float32)
|
||||
result = orig_rgb.astype(np.float32) * (1.0 - a) + mod_rgb.astype(np.float32) * a
|
||||
return np.clip(result, 0, 255).astype(np.uint8)
|
||||
|
||||
|
||||
def find_modified(orig_path, mod_dir):
|
||||
"""按文件名匹配修改图：{stem}*.png"""
|
||||
stem = Path(orig_path).stem
|
||||
return sorted(p for p in mod_dir.iterdir()
|
||||
if p.suffix.lower() == ".png" and p.name.startswith(stem))
|
||||
|
||||
|
||||
def process(orig_dir, mod_dir, out_dir, alpha_threshold=8.0, softness=1.5, protect=True,
|
||||
mask_dir=None, feather=20, verbose=True):
|
||||
"""
|
||||
alpha_threshold: 无蒙版时的 diff 阈值（默认 8；误判时调）
|
||||
softness: 过渡带宽度
|
||||
protect: 面部强制保护（无蒙版 fallback 时的保险）
|
||||
mask_dir: ComfyUI inpaint 蒙版目录（可选，有则优先用，蒙版白色=被移除区，最准确）
|
||||
feather: 蒙版边缘羽化
|
||||
"""
|
||||
orig_dir, mod_dir, out_dir = Path(orig_dir), Path(mod_dir), Path(out_dir)
|
||||
if mask_dir:
|
||||
mask_dir = Path(mask_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
originals = sorted(p for p in orig_dir.iterdir()
|
||||
if p.suffix.lower() in IMG_EXTS and not p.name.startswith("."))
|
||||
|
||||
if not originals:
|
||||
print(f"[WARN] 原图目录没有图片: {orig_dir}")
|
||||
return
|
||||
|
||||
total = 0
|
||||
for op in originals:
|
||||
mods = find_modified(op, mod_dir)
|
||||
if not mods:
|
||||
if verbose:
|
||||
print(f"[跳过] {op.name}: 无匹配修改图")
|
||||
continue
|
||||
|
||||
from PIL import Image
|
||||
orig_img = Image.open(op).convert("RGB")
|
||||
ow, oh = orig_img.size
|
||||
orig_rgb = np.array(orig_img)
|
||||
|
||||
# 尝试匹配蒙版
|
||||
mask_path = find_mask(op, mask_dir) if mask_dir else None
|
||||
|
||||
for mp in mods:
|
||||
try:
|
||||
mod_img = Image.open(mp).convert("RGB")
|
||||
if mod_img.size != (ow, oh):
|
||||
mod_img = mod_img.resize((ow, oh), Image.LANCZOS)
|
||||
mod_rgb = np.array(mod_img)
|
||||
|
||||
if mask_path and mask_path.exists():
|
||||
# 蒙版模式：最准确
|
||||
mask_img = Image.open(mask_path).convert("RGB")
|
||||
if mask_img.size != (ow, oh):
|
||||
mask_img = mask_img.resize((ow, oh), Image.LANCZOS)
|
||||
mask_rgb = np.array(mask_img)
|
||||
alpha = build_alpha_from_mask(mask_rgb, feather=feather)
|
||||
mode = f"蒙版模式({mask_path.name})"
|
||||
else:
|
||||
# 双图人脸差集模式（自动定位被移除人物 + 保留目标脸）
|
||||
alpha = build_alpha(orig_rgb, mod_rgb, threshold=alpha_threshold, softness=softness)
|
||||
mode = "差集模式"
|
||||
|
||||
area_ratio = float((alpha > 0.5).mean())
|
||||
if area_ratio < 0.0005:
|
||||
if verbose:
|
||||
print(f"[警告] {op.name} <- {mp.name}: 明显修改区仅 {area_ratio*100:.2f}%%，可能未检测到修改")
|
||||
mask_area_note = f"明显修改区 {area_ratio*100:.2f}% (偏小?)"
|
||||
else:
|
||||
mask_area_note = f"明显修改区 {area_ratio*100:.1f}%"
|
||||
|
||||
result = seamless_merge(orig_rgb, mod_rgb, alpha)
|
||||
mid = mp.stem[len(Path(op).stem):] or "_mod"
|
||||
out_name = f"{Path(op).stem}{mid}.png"
|
||||
Image.fromarray(result).save(out_dir / out_name)
|
||||
total += 1
|
||||
if verbose:
|
||||
print(f"[OK] {out_name} | {mode} | {mask_area_note}")
|
||||
except Exception as e:
|
||||
if verbose:
|
||||
print(f"[失败] {op.name} <- {mp.name}: {e}")
|
||||
|
||||
print(f"\n完成：共输出 {total} 张融合图 -> {out_dir}")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="原图 + AI修改图 无缝融合工具（支持 ComfyUI 蒙版）")
|
||||
ap.add_argument("orig_dir", help="原图目录")
|
||||
ap.add_argument("mod_dir", help="修改图目录（文件名以原图名开头、.png 结尾）")
|
||||
ap.add_argument("out_dir", help="输出目录")
|
||||
ap.add_argument("--mask-dir", default=None,
|
||||
help="ComfyUI inpaint 蒙版目录（可选，有则优先用；蒙版白色=被移除区，最准确解决紧贴残影）")
|
||||
ap.add_argument("--alpha-threshold", type=float, default=8.0,
|
||||
help="无蒙版时的 diff 阈值（默认 8）")
|
||||
ap.add_argument("--softness", type=float, default=1.5, help="过渡带宽度（默认 1.5）")
|
||||
ap.add_argument("--feather", type=int, default=20, help="蒙版边缘羽化（默认 20）")
|
||||
ap.add_argument("--no-protect-face", action="store_true", help="关闭面部强制保护（无蒙版时的保险）")
|
||||
args = ap.parse_args()
|
||||
|
||||
process(args.orig_dir, args.mod_dir, args.out_dir,
|
||||
alpha_threshold=args.alpha_threshold, softness=args.softness,
|
||||
protect=not args.no_protect_face, mask_dir=args.mask_dir, feather=args.feather)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Binary file not shown.
@@ -0,0 +1,236 @@
|
||||
# FaceLoRA v3 training monitor (scheduled task, fully autonomous)
|
||||
# ============================================================
|
||||
# 用途：云端训练全自动监控。Windows 计划任务每 5 分钟调用一次。
|
||||
# 功能：
|
||||
# 1. 增量下载 checkpoint（HTTP HEAD 探测 + aria2c）——任何时刻断线已下载的都是安全的
|
||||
# 2. 完成判定：e120 + final 都下载 -> 删 pod 停计费
|
||||
# 3. 崩溃检测：最后 checkpoint 45min 无更新 -> 抢救 + 删 pod
|
||||
# 4. 超时止损：超过预算小时数 -> 删 pod
|
||||
# 5. 状态持久化到 monitor_state.txt（纯文本，PS5.1 下 .json 写入不可靠）
|
||||
# 输出（全在 ASCII 目录 C:\Users\hmo\AppData\Local\Temp\opencode\face_lora\）：
|
||||
# monitor_v3.log - 详细运行日志
|
||||
# train_events.log - 关键事件时间线（下载/完成/止损/删pod）
|
||||
# monitor_state.txt - 持久化状态（pod_start/downloaded/done/started）
|
||||
# checkpoints_v3/ - 已下载的 checkpoint
|
||||
#
|
||||
# !! PS 5.1 铁律（血泪教训，2026-08-09）：
|
||||
# - 变量名不区分大小写！$STATE(路径) 会被 $state(hashtable) 覆盖 → 路径变量必须用独特名（$STATEFILE）
|
||||
# - 中文路径下 Add-Content 写新 .json 文件不可靠 → 输出目录用纯 ASCII，状态用纯文本
|
||||
# - train_env.json 读取失败必须退出，绝不用空值执行删 pod（曾因此差点误删）
|
||||
# ============================================================
|
||||
$ErrorActionPreference = "Continue"
|
||||
|
||||
# ---- 项目路径（注意：脚本必须存 GBK 编码，PowerShell 5.1 用 ANSI 读）----
|
||||
$PROJ = "D:\F\NewI\opencode\daily-workspace\projects\脸部LoRA训练-Qwen-Image"
|
||||
# 输出目录用纯 ASCII（避开中文路径 + PS5.1 编码 bug）
|
||||
$OUTDIR = "C:\Users\hmo\AppData\Local\Temp\opencode\face_lora"
|
||||
|
||||
# ---- 读取训练环境配置（train_env.json：pod_id/dl_url/ssh/max_hours/total_steps）----
|
||||
$CFG = Join-Path $PROJ "temp\train_env.json"
|
||||
if (-not (Test-Path $CFG)) {
|
||||
Write-Host "missing train_env.json"
|
||||
exit 1
|
||||
}
|
||||
$env_cfg = Get-Content $CFG -Raw | ConvertFrom-Json
|
||||
$PODID = $env_cfg.pod_id
|
||||
$DLURL = $env_cfg.dl_url
|
||||
$SSHHOST = $env_cfg.ssh_host
|
||||
$SSHPORT = $env_cfg.ssh_port
|
||||
$MAX_HOURS = [double]$env_cfg.max_hours
|
||||
$TOTAL_STEPS = [int]$env_cfg.total_steps
|
||||
# 安全阀：任何关键字段为空 → 立即退出，绝不用空值执行删 pod 等破坏性操作
|
||||
if (-not $PODID -or -not $DLURL -or $MAX_HOURS -le 0) {
|
||||
Write-Host ("train_env invalid: pod=" + $PODID + " max_hours=" + $MAX_HOURS)
|
||||
exit 1
|
||||
}
|
||||
|
||||
$MLOG = Join-Path $OUTDIR "monitor_v3.log"
|
||||
$EVENT = Join-Path $OUTDIR "train_events.log"
|
||||
$CDIR = Join-Path $OUTDIR "checkpoints_v3"
|
||||
$STATEFILE = Join-Path $OUTDIR "monitor_state.txt"
|
||||
$RPKEY = (Get-Content (Join-Path $PROJ ".runpod_api_key") -Raw).Trim()
|
||||
New-Item -ItemType Directory -Force $CDIR, (Split-Path $MLOG) | Out-Null
|
||||
|
||||
# ---- 单例守卫（防计划任务重复触发）----
|
||||
$LOCK = Join-Path $OUTDIR "monitor_v3.lock"
|
||||
if (Test-Path $LOCK) {
|
||||
$oldPid = [int](Get-Content $LOCK)
|
||||
if (Get-Process -Id $oldPid -ErrorAction SilentlyContinue) {
|
||||
exit 0
|
||||
}
|
||||
}
|
||||
Set-Content $LOCK $PID
|
||||
|
||||
function Log($msg) {
|
||||
$ts = Get-Date -Format "MM-dd HH:mm:ss"
|
||||
Add-Content -Path $MLOG -Value ("[" + $ts + "] " + $msg) -Encoding UTF8
|
||||
}
|
||||
function Event($msg) {
|
||||
$ts = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
|
||||
Add-Content -Path $EVENT -Value ("[" + $ts + "] " + $msg) -Encoding UTF8
|
||||
}
|
||||
|
||||
try {
|
||||
# ---- 读持久化状态（纯文本）----
|
||||
$state = @{ downloaded = @(); pod_start = ""; done = $false; started = $false }
|
||||
if (Test-Path $STATEFILE) {
|
||||
try {
|
||||
foreach ($ln in Get-Content $STATEFILE) {
|
||||
if ($ln -like "downloaded=*") { $state.downloaded = @($ln.Substring(11).Split(";") | Where-Object { $_ }) }
|
||||
elseif ($ln -like "pod_start=*") { $state.pod_start = $ln.Substring(10) }
|
||||
elseif ($ln -eq "done=true") { $state.done = $true }
|
||||
elseif ($ln -eq "started=true") { $state.started = $true }
|
||||
}
|
||||
} catch {}
|
||||
}
|
||||
if (-not $state.pod_start) {
|
||||
$state.pod_start = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
|
||||
Event ("monitor start: pod=" + $PODID + " dl_url=" + $DLURL + " max_hours=" + $MAX_HOURS)
|
||||
}
|
||||
|
||||
if ($state.done) {
|
||||
Log "done, skip"
|
||||
exit 0
|
||||
}
|
||||
|
||||
# ---- 0. 探测训练是否已开始（train.log 有 steps）----
|
||||
if (-not $state.started) {
|
||||
try {
|
||||
$tail = & ssh -i "$env:USERPROFILE\.ssh\id_rsa" -p $SSHPORT -o BatchMode=yes -o ConnectTimeout=10 $SSHHOST "tail -c 2000 /workspace/train.log 2>/dev/null | tr '\r' '\n' | tail -2" 2>$null
|
||||
if ($tail -match ("(\d+)/" + $TOTAL_STEPS)) {
|
||||
$state.started = $true
|
||||
Event ("training started: step " + $Matches[1] + "/" + $TOTAL_STEPS)
|
||||
}
|
||||
} catch {}
|
||||
}
|
||||
|
||||
# ---- 1. 探测云端 checkpoint（HTTP HEAD）----
|
||||
$epochs = @(10,20,30,40,50,60,70,80,90,100,110,120)
|
||||
$newFiles = @()
|
||||
foreach ($e in $epochs) {
|
||||
$fn = "myface_lora-{0:D6}.safetensors" -f $e
|
||||
if ($fn -in $state.downloaded) { continue }
|
||||
try {
|
||||
$resp = Invoke-WebRequest -Uri ($DLURL + "/" + $fn) -Method Head -TimeoutSec 15 -UseBasicParsing
|
||||
if ($resp.StatusCode -eq 200) { $newFiles += $fn }
|
||||
} catch {}
|
||||
}
|
||||
try {
|
||||
$resp = Invoke-WebRequest -Uri ($DLURL + "/myface_lora.safetensors") -Method Head -TimeoutSec 15 -UseBasicParsing
|
||||
if ($resp.StatusCode -eq 200) { $newFiles += "myface_lora.safetensors" }
|
||||
} catch {}
|
||||
|
||||
# ---- 2. 增量下载 ----
|
||||
foreach ($f in $newFiles) {
|
||||
$fp = Join-Path $CDIR $f
|
||||
if (Test-Path $fp) {
|
||||
$sz = [math]::Round((Get-Item $fp).Length/1MB,0)
|
||||
if ($sz -gt 100) {
|
||||
Log ("already have: " + $f + " (" + $sz + " MB), skip")
|
||||
$state.downloaded += $f
|
||||
continue
|
||||
}
|
||||
Remove-Item $fp -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
Log ("new checkpoint: " + $f + " downloading")
|
||||
& aria2c -x16 -s16 -k1M --continue -d "$CDIR" -o $f ($DLURL + "/" + $f) 2>&1 | Out-Null
|
||||
if (Test-Path $fp) {
|
||||
$sz = [math]::Round((Get-Item $fp).Length/1MB,0)
|
||||
if ($sz -gt 100) {
|
||||
Log ("download OK: " + $f + " (" + $sz + " MB)")
|
||||
Event ("CHECKPOINT DOWNLOADED: " + $f + " (" + $sz + " MB)")
|
||||
$state.downloaded += $f
|
||||
} else {
|
||||
Remove-Item $fp -Force -ErrorAction SilentlyContinue
|
||||
Log ("incomplete (" + $sz + " MB), retry next round: " + $f)
|
||||
}
|
||||
} else {
|
||||
Log ("download FAIL: " + $f)
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
# ---- 3. 完成判定 ----
|
||||
$hasFinal = "myface_lora.safetensors" -in $state.downloaded
|
||||
if ($hasFinal) {
|
||||
Log "=== TRAINING COMPLETE: final checkpoint downloaded ==="
|
||||
Event ("=== TRAINING COMPLETE: all checkpoints in checkpoints_v3\ ===")
|
||||
try {
|
||||
Invoke-RestMethod -Uri ("https://rest.runpod.io/v1/pods/" + $PODID) -Method Delete -Headers @{Authorization = "Bearer $RPKEY"} -TimeoutSec 30 | Out-Null
|
||||
Log ("pod " + $PODID + " deleted (complete)")
|
||||
Event ("POD DELETED: " + $PODID + " (complete, billing stopped)")
|
||||
} catch { Log ("pod delete fail: " + $_.Exception.Message) }
|
||||
$state.done = $true
|
||||
}
|
||||
|
||||
# ---- 4. 崩溃检测 ----
|
||||
if ($state.downloaded.Count -gt 0 -and -not $state.done) {
|
||||
$lastFile = $state.downloaded[-1]
|
||||
$lastPath = Join-Path $CDIR $lastFile
|
||||
if (Test-Path $lastPath) {
|
||||
$ageMin = [int]((Get-Date) - (Get-Item $lastPath).LastWriteTime).TotalMinutes
|
||||
if ($ageMin -gt 45) {
|
||||
Log ("CRASH: last ckpt (" + $lastFile + ") " + $ageMin + "min stale -> rescue + delete pod")
|
||||
Event ("CRASH: last ckpt " + $lastFile + " " + $ageMin + "min stale -> rescue + delete pod")
|
||||
foreach ($e in $epochs) {
|
||||
$fn = "myface_lora-{0:D6}.safetensors" -f $e
|
||||
if ($fn -in $state.downloaded) { continue }
|
||||
try {
|
||||
$r = Invoke-WebRequest -Uri ($DLURL + "/" + $fn) -Method Head -TimeoutSec 15 -UseBasicParsing
|
||||
if ($r.StatusCode -eq 200) {
|
||||
& aria2c -x16 -s16 -k1M -d "$CDIR" -o $fn ($DLURL + "/" + $fn) 2>&1 | Out-Null
|
||||
if (Test-Path (Join-Path $CDIR $fn)) { $state.downloaded += $fn; Event ("RESCUED: " + $fn) }
|
||||
}
|
||||
} catch {}
|
||||
}
|
||||
try {
|
||||
Invoke-RestMethod -Uri ("https://rest.runpod.io/v1/pods/" + $PODID) -Method Delete -Headers @{Authorization = "Bearer $RPKEY"} -TimeoutSec 30 | Out-Null
|
||||
Log ("pod " + $PODID + " deleted (crash)")
|
||||
Event ("POD DELETED: " + $PODID + " (crash stop)")
|
||||
} catch { Log ("pod delete fail: " + $_.Exception.Message) }
|
||||
$state.done = $true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# ---- 5. 超时止损 ----
|
||||
if (-not $state.done) {
|
||||
try {
|
||||
$start = [datetime]::ParseExact($state.pod_start, "yyyy-MM-dd HH:mm:ss", $null)
|
||||
$runHours = ((Get-Date) - $start).TotalHours
|
||||
if ($runHours -gt $MAX_HOURS) {
|
||||
Log ("TIMEOUT: " + [math]::Round($runHours,1) + "h > budget " + $MAX_HOURS + "h")
|
||||
Event ("TIMEOUT: " + [math]::Round($runHours,1) + "h > budget " + $MAX_HOURS + "h -> delete pod")
|
||||
try {
|
||||
Invoke-RestMethod -Uri ("https://rest.runpod.io/v1/pods/" + $PODID) -Method Delete -Headers @{Authorization = "Bearer $RPKEY"} -TimeoutSec 30 | Out-Null
|
||||
Log ("pod " + $PODID + " deleted (timeout, downloaded " + $state.downloaded.Count + ")")
|
||||
Event ("POD DELETED: " + $PODID + " (timeout, downloaded " + $state.downloaded.Count + ")")
|
||||
} catch { Log ("pod delete fail: " + $_.Exception.Message) }
|
||||
$state.done = $true
|
||||
} else {
|
||||
Log ("running: " + [math]::Round($runHours,1) + "/" + $MAX_HOURS + "h, downloaded " + $state.downloaded.Count)
|
||||
}
|
||||
} catch {
|
||||
Log ("time parse fail: " + $_.Exception.Message)
|
||||
}
|
||||
}
|
||||
|
||||
# ---- 持久化状态（纯文本行，PS5.1 可靠）----
|
||||
try {
|
||||
$lines = @()
|
||||
$lines += "pod_start=" + $state.pod_start
|
||||
$lines += "downloaded=" + ($state.downloaded -join ";")
|
||||
$lines += ("done=" + $state.done)
|
||||
$lines += ("started=" + $state.started)
|
||||
if (Test-Path $STATEFILE) { Remove-Item $STATEFILE -Force }
|
||||
Add-Content -Path $STATEFILE -Value $lines -Encoding UTF8
|
||||
} catch {
|
||||
Log ("state write fail: " + $_.Exception.Message)
|
||||
}
|
||||
$state.downloaded | ForEach-Object { Event ("state: downloaded " + $_) }
|
||||
if ($state.done) { Event ("=== MONITOR DONE: " + $state.downloaded.Count + " checkpoints ===") }
|
||||
else { Event ("state: running, downloaded=" + $state.downloaded.Count) }
|
||||
if ($state.done) { Log "=== monitor done ===" }
|
||||
} finally {
|
||||
Remove-Item $LOCK -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
LoRA checkpoint 验证工具
|
||||
========================
|
||||
加载 edit-2511 底模 + 训练好的 LoRA，用多组提示词生成对比图，
|
||||
验证 LoRA 是否生效、效果如何。
|
||||
|
||||
用法:
|
||||
python validate_lora.py # 验证最新 checkpoint
|
||||
python validate_lora.py --ckpt <路径> # 验证指定 checkpoint
|
||||
python validate_lora.py --all # 验证 output/checkpoints 下所有
|
||||
"""
|
||||
import glob
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
||||
|
||||
VENV = r"D:\AI\sd\musubi-tuner\.venv\Scripts"
|
||||
PY = os.path.join(VENV, "python.exe")
|
||||
SRC = r"D:\AI\sd\musubi-tuner\src\musubi_tuner"
|
||||
MODELS = r"D:\AI\sd\models\qwen-edit-2511"
|
||||
PROJ = r"D:\F\NewI\opencode\daily-workspace\projects\脸部LoRA训练-Qwen-Image"
|
||||
|
||||
TRIGGER = "lm_face_v1"
|
||||
CKPT_DIR = os.path.join(PROJ, "output", "checkpoints")
|
||||
OUT_DIR = os.path.join(PROJ, "output", "验证")
|
||||
|
||||
# 验证提示词：同一触发词 + 不同场景，看脸是否稳定一致
|
||||
PROMPTS = [
|
||||
("证件照", f"{TRIGGER}, professional headshot, business attire, neutral expression, studio lighting, plain white background"),
|
||||
("日常", f"{TRIGGER}, natural lifestyle portrait, by a window with soft diffused sunlight, warm tones"),
|
||||
("户外全身", f"{TRIGGER}, full body shot, standing in a park, natural daylight, casual clothing"),
|
||||
("半侧脸", f"{TRIGGER}, three-quarter view portrait, soft golden hour light, shallow depth of field"),
|
||||
]
|
||||
|
||||
BASE_CMD = [
|
||||
PY, os.path.join(SRC, "qwen_image_generate_image.py"),
|
||||
"--dit", os.path.join(MODELS, "transformer", "diffusion_pytorch_model-00001-of-00005.safetensors"),
|
||||
"--vae", os.path.join(MODELS, "diffusion_pytorch_model.safetensors"),
|
||||
"--text_encoder", os.path.join(MODELS, "text_encoder", "model-00001-of-00004.safetensors"),
|
||||
"--model_version", "edit-2511",
|
||||
"--fp8", "--fp8_scaled", "--blocks_to_swap", "24",
|
||||
"--infer_steps", "25",
|
||||
"--image_size", "1024", "1024",
|
||||
"--seed", "42",
|
||||
]
|
||||
|
||||
|
||||
def pick_ckpts(args):
|
||||
if args.ckpt:
|
||||
return [args.ckpt]
|
||||
if args.all and os.path.isdir(CKPT_DIR):
|
||||
return sorted(glob.glob(os.path.join(CKPT_DIR, "*.safetensors")))
|
||||
ckpts = sorted(glob.glob(os.path.join(CKPT_DIR, "*.safetensors")))
|
||||
return [ckpts[-1]] if ckpts else []
|
||||
|
||||
|
||||
def main():
|
||||
import argparse
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--ckpt", default=None, help="指定 checkpoint 路径")
|
||||
ap.add_argument("--all", action="store_true", help="验证所有 checkpoint")
|
||||
args = ap.parse_args()
|
||||
|
||||
ckpts = pick_ckpts(args)
|
||||
if not ckpts:
|
||||
print(f"[WARN] 没有找到 checkpoint: {CKPT_DIR}")
|
||||
sys.exit(1)
|
||||
|
||||
os.makedirs(OUT_DIR, exist_ok=True)
|
||||
print(f"找到 {len(ckpts)} 个 checkpoint，开始验证（每个 4 张图）...")
|
||||
|
||||
for ckpt in ckpts:
|
||||
name = os.path.splitext(os.path.basename(ckpt))[0]
|
||||
print(f"\n===== {name} =====")
|
||||
for tag, prompt in PROMPTS:
|
||||
out = os.path.join(OUT_DIR, f"{name}_{tag}.png")
|
||||
cmd = BASE_CMD + ["--lora_weight", ckpt, "--lora_multiplier", "0.8",
|
||||
"--prompt", prompt, "--save_path", out]
|
||||
print(f" 生成 [{tag}] ...")
|
||||
r = subprocess.run(cmd)
|
||||
if r.returncode == 0:
|
||||
print(f" OK -> {out}")
|
||||
else:
|
||||
print(f" FAIL exit={r.returncode}")
|
||||
|
||||
print(f"\n验证完成，图片在 {OUT_DIR}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,16 @@
|
||||
@echo off
|
||||
chcp 65001 >nul
|
||||
rem ============================================
|
||||
rem face_checker 一键审核：把照片放进 photos 文件夹，双击本文件
|
||||
rem ============================================
|
||||
set TOOL=D:\F\NewI\opencode\daily-workspace\projects\脸部LoRA训练-Qwen-Image\tools\face_checker.py
|
||||
set PY=D:\AI\sd\musubi-tuner\.venv\Scripts\python.exe
|
||||
set PHOTOS=D:\F\NewI\opencode\daily-workspace\projects\脸部LoRA训练-Qwen-Image\photos
|
||||
|
||||
if not exist "%PHOTOS%" mkdir "%PHOTOS%"
|
||||
echo 正在审核 %PHOTOS% ...
|
||||
"%PY%" "%TOOL%" check "%PHOTOS%"
|
||||
echo.
|
||||
echo 报告已生成: %PHOTOS%\素材审核报告.html
|
||||
start "" "%PHOTOS%\素材审核报告.html"
|
||||
pause
|
||||
@@ -0,0 +1,16 @@
|
||||
@echo off
|
||||
chcp 65001 >nul
|
||||
rem ============================================
|
||||
rem face_checker 智能选图：从 photos 里挑出最多样化的 20 张
|
||||
rem ============================================
|
||||
set TOOL=D:\F\NewI\opencode\daily-workspace\projects\脸部LoRA训练-Qwen-Image\tools\face_checker.py
|
||||
set PY=D:\AI\sd\musubi-tuner\.venv\Scripts\python.exe
|
||||
set PHOTOS=D:\F\NewI\opencode\daily-workspace\projects\脸部LoRA训练-Qwen-Image\photos
|
||||
|
||||
if not exist "%PHOTOS%" mkdir "%PHOTOS%"
|
||||
echo 正在智能选图（推荐 20 张）...
|
||||
"%PY%" "%TOOL%" pick "%PHOTOS%" --count 20
|
||||
echo.
|
||||
echo 推荐报告已生成: %PHOTOS%\智能选图推荐_20张.html
|
||||
start "" "%PHOTOS%\智能选图推荐_20张.html"
|
||||
pause
|
||||
@@ -0,0 +1,73 @@
|
||||
# face_checker 使用说明（素材自动审核 + 智能选图工具）
|
||||
|
||||
## 这是什么
|
||||
|
||||
训练脸部 LoRA 前的**素材质检员 + 选图助手**：
|
||||
- 自动检查每张照片：分辨率、清晰度、人脸数量、人脸大小、正侧脸角度、遮挡、重复图
|
||||
- 红黄绿三档判定：✅合格 / 🟡警告 / ❌不合格，附具体原因
|
||||
- **智能选图**：从一堆照片里自动挑出多样化组合（覆盖特写/半身/全身、不同角度、不同风格），你只需要删掉重复的
|
||||
|
||||
## 快速开始
|
||||
|
||||
用哪个 Python：musubi-tuner 的虚拟环境
|
||||
|
||||
```powershell
|
||||
# 场景一：审核照片，生成报告
|
||||
& "D:\AI\sd\musubi-tuner\.venv\Scripts\python.exe" "D:\F\NewI\opencode\daily-workspace\projects\脸部LoRA训练-Qwen-Image\tools\face_checker.py" check "照片目录"
|
||||
|
||||
# 场景二：智能选图（推荐 20 张）
|
||||
& "D:\AI\sd\musubi-tuner\.venv\Scripts\python.exe" "...\tools\face_checker.py" pick "照片目录" --count 20
|
||||
|
||||
# 场景三：审核 + 把合格照片整理成训练集
|
||||
& "...\face_checker.py" batch "照片目录" --out "训练集目录"
|
||||
```
|
||||
|
||||
> 提示：可以把工具路径存成 PowerShell 变量或做个小 .bat，避免每次打长路径（我可以帮你做）。
|
||||
|
||||
## 输出
|
||||
|
||||
| 命令 | 产出 |
|
||||
|---|---|
|
||||
| `check` | `素材审核报告.html`（浏览器打开看图片墙+判定）+ `素材审核结果.json` |
|
||||
| `pick` | `智能选图推荐_N张.html`（推荐清单+构图标签）+ `智能选图结果.json` |
|
||||
| `batch` | 报告 + 训练集文件夹（img_001.jpg 重命名、EXIF 修正、统一 JPG） |
|
||||
|
||||
## 判定规则（快查）
|
||||
|
||||
| 检查项 | 合格 | 警告 | 不合格 |
|
||||
|---|---|---|---|
|
||||
| 分辨率 | 短边 ≥1024 | - | 短边 <1024（微信压缩图） |
|
||||
| 清晰度 | 人脸区域锐利 | 偏糊 | 明显模糊 |
|
||||
| 人脸 | 1 张 | 多张脸 / 占比小 / 侧脸 / 疑似遮挡 | 没检测到脸 |
|
||||
| 重复 | - | 与另一张高度相似 | - |
|
||||
|
||||
## 老莫的操作流程
|
||||
|
||||
1. 把候选照片**全部**丢进一个文件夹（几十张也没关系，不用自己先筛）
|
||||
2. 跑 `pick --count 20` → 打开推荐报告看一眼，删掉不喜欢的
|
||||
3. 跑 `batch --out 训练集目录` → 自动整理成 `img_001.jpg...`
|
||||
4. 把训练集目录交给我，我负责打标 caption 和训练
|
||||
|
||||
## 注意事项
|
||||
|
||||
- **iPhone 的 HEIC 照片**：工具不支持，先转成 JPG（微信发一遍自己或格式转换工具）
|
||||
- 照片**别用微信压缩过的**（分辨率不够会被判不合格）
|
||||
- 想多选就 `--count 25`，想少选就 `--count 15`（推荐 15-25 张）
|
||||
- 重复判定基于视觉相似度，AI 判断可能有漏网，报告里的人工确认最重要
|
||||
|
||||
## 参数速查
|
||||
|
||||
```
|
||||
check <目录> [--out 报告目录]
|
||||
pick <目录> [--count N] [--out 报告目录]
|
||||
prepare <目录> --out <训练集目录>
|
||||
batch <目录> --out <训练集目录>
|
||||
crop <目录> --out <裁切目录> [--scale 1.8] ← 合影处理：按最大人脸裁切单人
|
||||
```
|
||||
|
||||
## 合影与背景处理（老莫须知）
|
||||
|
||||
- **合影**：用 `crop` 命令或 GUI 前先裁切——按最大人脸裁出单人（1.8 倍放大）。裁完短边 <1024 的放弃
|
||||
- **背景**：不要处理！保持自然多样（室内/户外/墙/街景），caption 写清场景即可。**不要抠图换纯色背景**（会导致 LoRA 把纯色背景和脸绑定，出图全是纯背景）
|
||||
- **禁止**：磨皮美颜、加水印边框文字
|
||||
- face_checker 的 prepare/整理会自动处理 EXIF 方向 + RGB 转换
|
||||
Reference in New Issue
Block a user