Commit Graph
1405 Commits
Author SHA1 Message Date
Jaret Burkett 682b27c6ee Reworked freeing memory manager for removing text encoder completly when not needed. 2026-08-09 07:08:24 -06:00
Jaret Burkett 3a28c4b1b7 Keep alive cron server to prevent failed connections 2026-08-08 20:27:01 -06:00
Jaret Burkett 8c1a4082fd Allow images to work with auto frame count, and include images in video datasets if they exist. 2026-08-08 20:26:28 -06:00
Jaret Burkett 6d8afa5684 Version bump 2026-08-08 18:58:11 -06:00
Jaret Burkett (Ostris) c596d4ab27 Merge pull request #1002 from whatsthisaithing/codex/fix-convrot-offload-stream-lifetime
Fix offload buffer stream lifetime
2026-08-08 18:55:18 -06:00
Fitzy d184c6c622 Fix offload buffer stream lifetime 2026-08-08 19:59:23 -04:00
Jaret Burkett f4e9130547 Fix race condition that can corrupt grads under certain conditions. 2026-08-07 14:52:47 -06:00
Jaret Burkett 817f3dcbcb Fix finite check on inverted masked prior 2026-08-07 09:49:12 -06:00
Jaret Burkett 685ce37a8d Adjust sample timestep sigmas to be model evals for h3 for 1 extra step 2026-08-06 19:54:04 -06:00
Jaret Burkett 9171d5ec1d Update training adapter path 2026-08-06 11:21:28 -06:00
Jaret Burkett 71625d1207 Add the alpha version of the MiniMax H3 trianing adapter and set it as the new default training method. 2026-08-06 11:11:31 -06:00
Jaret Burkett b904b99705 varsion bump 2026-08-06 09:25:35 -06:00
Jaret Burkett b811636ae4 Check for finite vs isnan on loss before backpropigating. to catch infinity overflows 2026-08-06 09:18:56 -06:00
Jaret Burkett edacd406b3 Handle minimax loading of non pruned model 2026-08-06 09:18:00 -06:00
Jaret Burkett 7309db4d74 Fix layer offloaded adaln projection layer move 2026-08-06 08:10:20 -06:00
Jaret Burkett 139a38f5bd Upcast adaln pruned layers to fp32 on h3 to prevent overflow. 2026-08-06 07:51:15 -06:00
Jaret Burkett 1e1418b22c Apply sigma to contrastive guidance to balance loss better. Prevent noise grads on images/non audio datasets. Prep for training adapters on MiniMax H3 2026-08-05 13:27:12 -06:00
Jaret Burkett 9065951da3 Version bump 2026-08-04 22:14:54 -06:00
Jaret Burkett 3afa270ab5 Fix audio losses for DOP and other preservation losses 2026-08-04 22:08:48 -06:00
Jaret Burkett 0f9094db95 Fix audio loss when doing do_guidance_loss 2026-08-04 21:43:31 -06:00
Jaret Burkett d870e9b68a Fix backwards compatability for older versions of triton 2026-08-04 21:07:57 -06:00
Jaret Burkett a8d67ecd90 Version Bump 2026-08-04 16:28:04 -06:00
Jaret Burkett 9fc1f208df Remove adaln_proj from the network modules for minimax_h3 2026-08-04 16:19:46 -06:00
Jaret Burkett 183433ae8e Make MiniMax H3 default to using contrastive guidance loss to prevent distillation breakdown 2026-08-04 16:13:45 -06:00
Jaret Burkett 00a93e3830 Add dataset flag to cache the raw tensors 2026-08-04 15:32:48 -06:00
Jaret Burkett 8a0bcf1ffe Fix issue loading H3 test encoder on older GPUS 2026-08-04 15:30:29 -06:00
Jaret Burkett dc29ae1187 Version Bump 2026-08-04 09:27:46 -06:00
Jaret Burkett d20a17c10e Limit max tokens to 512. Allow override with model kwargs. 2026-08-04 07:55:21 -06:00
Jaret Burkett 602306da77 Add gradient checkpointing to vae 2026-08-04 07:54:51 -06:00
Jaret Burkett 18f5810d6c Adjust default alpha for h3 2026-08-03 20:15:28 -06:00
Jaret Burkett a9a04547e9 Dont move encoders on and off device when caching until the first instance of needing to process a cache item. 2026-08-03 15:54:30 -06:00
Jaret Burkett 41676bb258 Queue up videos with multiple threads when caching latents so the VAE is not waiting on videos to process 2026-08-03 15:41:36 -06:00
Jaret Burkett 546eb7daff Look for existing models in folders recursivly 2026-08-03 15:39:55 -06:00
Jaret Burkett d3a3f70a2a Speed up quantization processing on H3 2026-08-03 15:21:39 -06:00
Jaret Burkett 88ac27fc8f Handle images with MiniMax H3. 2026-08-03 11:58:44 -06:00
Jaret Burkett bf739ff966 Fix issue with layer offloading with MinMax H3 2026-08-03 11:32:36 -06:00
Jaret Burkett 9d614a51fb Hard fail if a step in the docker build fails. 2026-08-03 10:38:31 -06:00
Jaret Burkett 8502a845b1 Add support for MiniMax H3 T2V and I2V training 2026-08-03 10:17:39 -06:00
Jaret Burkett 73cab2acf5 Reworked merge in out of loras with convrot weights for better roundtrip accuracy. 2026-08-02 08:29:17 -06:00
Jaret Burkett a6f6b6b896 Fixed experiment opt name match 2026-08-01 17:43:36 -06:00
Jaret Burkett 6b95282097 Recover from issue when a video model first fram may not have been cached properly 2026-08-01 12:00:33 -06:00
Jaret Burkett 5baa495585 DFE optimizations 2026-08-01 10:06:16 -06:00
Jaret Burkett fc78b07332 Attach the ema to the base model so it can be used on specific models 2026-08-01 10:05:42 -06:00
Jaret Burkett c68e58083f Add code for automagic experiment 2026-08-01 10:04:46 -06:00
Jaret Burkett 497014bf5d Set i2v to default to false when omitted from the config 2026-07-31 19:11:58 -06:00
Jaret Burkett 038f24e8c3 Revert torchao back to older version 2026-07-31 16:34:06 -06:00
Jaret Burkett 6e7bc81241 Allow random noise shift with video latents. 2026-07-31 05:58:08 -06:00
Jaret Burkett ddc69745fe Update docker build image with newer dependencies 2026-07-30 14:20:31 -06:00
Jaret Burkett (Ostris) 2cab330392 Merge pull request #986 from ostris/dev
Add AI Toolkit Manager script that auto installs/runs/and updates AI Toolkit.
2026-07-30 11:55:13 -06:00
Jaret Burkett c8636478f9 Version Bump 2026-07-30 11:48:01 -06:00