evan/ai-toolkit
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,169 Commits 2 Branches 0 Tags
a1ac6e8b01336b6cf12999a70a34572ab941c92f
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Jaret Burkett a1ac6e8b01 Reworked automagic v3 again. Seems more stable. Still testing. 2026-06-08 22:03:54 -06:00
Jaret Burkett 5d6887fd98 Major updates to automagic3 optimizer. Seems to be functioning more ideally and naturally decays, as it should. 2026-06-08 14:09:02 -06:00
Jaret Burkett 0d018db689 Rework the toggles so we can hide the trend line from the loss graph. No need for a raw toggle either. 2026-06-08 13:56:37 -06:00
Jaret Burkett cac3815b2c Make DOP run in a single backward pass, should be faster and more stable. Show dop loss on ui 2026-06-08 13:55:47 -06:00
Jaret Burkett 687def6f7a Add better smoothing with ema rounding of the ends of the loss graph for first and last instance do not dominate the smoothing factors. 2026-06-08 10:43:46 -06:00
Jaret Burkett d7f8887bbf Show all logged metrics on the loss graph 2026-06-08 09:43:03 -06:00
Jaret Burkett e281df70dd Allow automagic3 to run unfused. Add some clipping. 2026-06-08 08:59:06 -06:00
Jaret Burkett c9cdbb5bb7 Version bump 2026-06-07 16:08:11 -06:00
Jaret Burkett c78b1404e3 Deepen offload prefetch pipeline with per-slot events 
Replace the 2-slot ping-pong + single global "compute-started" event
with a depth-N ring buffer where each transfer waits only on the slot
it's reusing (D layers back) instead of the most-recent compute. Applies
to forward and backward, Linear and Conv. Depth is tunable via
AI_TOOLKIT_OFFLOAD_DEPTH (default 4).

Bit-exact vs non-offload (output, grad_input, weight grads). No speedup
on a bandwidth-bound PCIe link (already saturated at depth 2), but the
cleaner per-slot design removes the fragile shared-event serialization
and lets deeper prefetch help on faster buses.
 2026-06-07 16:07:13 -06:00
Jaret Burkett cdff6e36aa Pin inner stores of torachao to speed up layer offloading for quantized models around 25% 2026-06-07 15:52:01 -06:00
Jaret Burkett 75781fb5a5 Fix float8 weights not offloading to CPU in layer offloading 2026-06-07 15:36:17 -06:00
Jaret Burkett 35588726de Fixed issue where a buffer was stuck on cpu when offloading ideogram4 2026-06-07 13:31:03 -06:00
Jaret Burkett 1dc9a797cf Added Automagic v3 2026-06-07 12:06:42 -06:00
Jaret Burkett 82190b41e6 Fix bug where EMA was not being initialized even when setup in the config. EMA will now properly be setup and used. 2026-06-07 11:11:49 -06:00
Jaret Burkett 8968e41234 Set ideogram to use new loke saving format 2026-06-07 09:49:57 -06:00
Jaret Burkett fa0dca288d Version bump 2026-06-06 10:24:31 -06:00
Jaret Burkett 41157b460c Added ability to set the caption extention in dataset viewer, captioner, and trainer so one dataset can have multiple caption styles in different files with different extensions. Added dataset caption template for a blank ideogram 4 formatted template. 2026-06-06 08:32:24 -06:00
Jaret Burkett 10cdeb394e Version bump 2026-06-05 14:09:46 -06:00
Jaret Burkett 21a6beb194 Allow editing the boxes visually on the sample section for ideogram 2026-06-05 14:09:18 -06:00
Jaret Burkett 4441080c05 Rework the ui of ideogram dataset image caption editor 2026-06-05 13:20:58 -06:00
Jaret Burkett b70083a74f Fix issue with default sampels. 2026-06-05 12:33:42 -06:00
Jaret Burkett c994398850 Rework prompts and captioning systems for ideogram to more strictly match the format provided by ideogram. 2026-06-05 11:14:36 -06:00
Jaret Burkett 6fd2253932 Imporved editing boxed on the dataset viewer 2026-06-05 10:30:06 -06:00
Jaret Burkett ef12260b80 Add a prompt upsample ui for upsampling prompts to ideogram format prompts. 2026-06-05 09:56:02 -06:00
Jaret Burkett 90a2084f70 Allow adjusting, adding, and deleting bounding boxes. 2026-06-04 20:29:31 -06:00
Jaret Burkett bb60f6d1d1 Show bounding boxes on the sample image card if we have them in the prompt 2026-06-04 15:59:02 -06:00
Jaret Burkett edcc7415d1 Added an autocaptioner for Ideogram 4 captions 2026-06-04 13:04:23 -06:00
Jaret Burkett 6a8d9333b6 Improved the prompt handeling of ideogram4 model. Now used advanced prompts class to store them smaller and allow longer prompts 2026-06-04 13:00:23 -06:00
Jaret Burkett 2ddc2e1318 Update the default Ideogram 4 prompts to work significantly better. 2026-06-04 10:32:05 -06:00
Jaret Burkett 63b3181262 Added experimental support for Ideogram 4 2026-06-04 09:03:00 -06:00
Jaret Burkett b5f21ae695 Fixed issue where the training type dropdown in the ui was not showing fully 2026-06-03 10:14:15 -06:00
Jaret Burkett d9f26c2f87 Add gradient checkpointing to tipsv2 heads 2026-06-01 05:00:15 -06:00
Jaret Burkett bd468727a6 Fixed issue where some image cards in samples and datasets would show gray instead of the image until you scroll. 2026-06-01 03:36:32 -06:00
Jaret Burkett f5446c0d5f Version bump 2026-06-01 02:16:47 -06:00
Jaret Burkett e5439509b5 Added pure lpips dfe 2026-05-31 11:52:03 -06:00
Jaret Burkett 212cfe998a Add ability to download or delete the optimizer state from the ui 2026-05-29 08:56:22 -06:00
Jaret Burkett 5e84bf0d0b Fixed issue with hidream-01 that could cause a weird nan state. Took forever to track down as it was 1 in 10 starts. 2026-05-28 12:48:24 -06:00
Jaret Burkett 87bac27513 Fixed issue with new bucket scaler 2026-05-28 11:34:16 -06:00
Jaret Burkett 30886b8f92 Added a wallet indicator for Ostris Cloud 2026-05-28 11:09:51 -06:00
Jaret Burkett 3e86d81fc6 Adjust bucket sizes to achieve maximum pixels without going over. 2026-05-28 09:36:17 -06:00
Jaret Burkett ef57c1077c Change z image divisibility 2026-05-28 09:09:36 -06:00
Jaret Burkett 15082cfb8a Round buckets for divisibility instead of always rounding down. 2026-05-28 09:08:21 -06:00
Jaret Burkett c9264bdd0b Reworked the bucketing system to precisly match model specific divisibility. The old SDXL bucket sizes needed to go. 2026-05-28 08:35:58 -06:00
Jaret Burkett 68e9b38220 Add ability to trigger a save from the ui which whill make the trainer save on the next step 2026-05-26 08:38:49 -06:00
Jaret Burkett 2aa60e4ca5 Update default agreement threshold for automagic v2 to be 0.5 2026-05-26 07:57:54 -06:00
Jaret Burkett 76c99da4e4 Add version to the ui 2026-05-26 07:30:09 -06:00
Jaret Burkett 266956068a Add a way to delete checkpoints on the ui 2026-05-26 07:09:21 -06:00
Jaret Burkett 954c5efec8 Show job info on sidebar active job in ui 2026-05-25 11:07:29 -06:00
Jaret Burkett 083236a2a7 Drastically improve the loading speed of images in the ui by using a custom loader and abort controller to abort when images leave the view. 2026-05-25 10:04:25 -06:00
Jaret Burkett 7354def271 Added decode latent method to qwen image model 2026-05-25 10:02:09 -06:00
Jaret Burkett 3d836ac371 Show active jobs in the sidebar of the ui 2026-05-25 08:21:42 -06:00
Jaret Burkett a798e06dd2 Updated the support button to look more like a button 2026-05-25 07:48:04 -06:00
Jaret Burkett 8042cbe9d2 Added virtulization for sample images to handle huge number of samples more efficientyly. 2026-05-25 07:22:24 -06:00
Jaret Burkett fbac1cb7f5 Dont force flash attention on hidream 01. Causes random issues and is slower. 2026-05-24 16:05:22 -06:00
Jaret Burkett 307ff11bc5 Drastically improved the performance of the dataset viewer on large datasets by switching to virtualization. Fixed issue with images loading when they have two periods in a row .. 2026-05-24 15:13:55 -06:00
Jaret Burkett c6a7e81a70 Added a dataset image viewer. Fixed an issue where captions would show as not saved when they were saved. 2026-05-24 14:31:06 -06:00
Jaret Burkett 12304e170f Added some experimental loss targets 2026-05-24 14:13:23 -06:00
Jaret Burkett 644a6f9246 Fix device casting for zimage in some instances 2026-05-23 10:37:33 -06:00
Jaret Burkett c6ecc03ccd Fixed saving full model of z_image l2p for finetuning 2026-05-23 07:36:24 -06:00
Jaret Burkett 6102370df9 Add support for ZImage L2P 2026-05-22 14:58:28 -06:00
Jaret Burkett 6fc08a8928 Fix issue with saving images from new sample viewer 2026-05-22 11:58:32 -06:00
Jaret Burkett 5579837c3f Add hidream o1 to the readme 2026-05-21 07:56:54 -06:00
Jaret Burkett aecd554128 Add sapiens2 matting as a mask generator. Begin transition to model paths and model folders. 2026-05-20 08:56:16 -06:00
Jaret Burkett 15d4fb89ff Fixed overflow issue on modal 2026-05-19 10:12:43 -06:00
Jaret Burkett df851b3497 Made the UI mobile friendly, finally... 2026-05-19 09:27:36 -06:00
Jaret Burkett 6ecaf679dc Add ability to run small scripts from the ui and added a merge lora script 2026-05-18 14:33:47 -06:00
Jaret Burkett ec58dcde92 Add better gradient checkpointing to dfes 2026-05-18 09:25:23 -06:00
Jaret Burkett b42acb988f Remove future steps from loss log if resuming from an earlier step 2026-05-18 09:23:55 -06:00
Jaret Burkett e03c6e4dc9 Fix potential inconsistency with different attention mentods in hidream01 2026-05-13 09:08:11 -06:00
Jaret Burkett 4bfe944792 Scale dfe 7 with velocity_equiv_weight 2026-05-13 09:06:59 -06:00
Jaret Burkett (Ostris) fc4d6ebf39 Add support for fine-tuning Hidream O1 (#831) 
* Initial support for hidream. Lora keys likely need work

* Fix saving for hidream-o1

* Remove dependence on flash attention for hidream o1

* Fix gradient checkpointing for hidream o1

* A lot of fixes for hidream. Handle loading and saving as comfy model.

* Omit layers not used in comfy. Fix issue with lora loading keys in comfy

* Version bumpo
 2026-05-12 11:15:16 -06:00
Jaret Burkett f38de2a2fe Add tipsv2 locally and fix gradient checkpointing for it 2026-05-10 14:47:44 -06:00
Jaret Burkett d144cb5ea6 Switch to uplot for loss graph and rework performance of graph. It is significantly more perfromant now. 2026-05-07 07:38:00 -06:00
Jaret Burkett a12ddd72a1 Change the velocity weight cap on dfe 9 2026-05-07 07:37:05 -06:00
Jaret Burkett 6bb8acbffc Add agreement_threshold default of 0.6 to automagic 2 2026-05-05 19:13:00 -06:00
Jaret Burkett 963a9f42b2 Add decode latent to wan 2.1 models. Add gradinet checkpointing to wan vae. 2026-05-05 11:30:16 -06:00
Jaret Burkett 4260a3c5b6 Add optimizer test suite and make minor speed adjustments to Automagicv2 2026-05-05 10:02:30 -06:00
Jaret Burkett aeca7fe404 Add Automagic v2 optimizer. It uses significantly less vram and is much more efficient. 2026-05-05 09:09:07 -06:00
Jaret Burkett 0d91fcee9e Allow edit of captioning job 2026-04-30 16:00:16 -06:00
Jaret Burkett eadc9a58af Version bump 2026-04-30 06:00:58 -06:00
Jaret Burkett e9ab387dfd Fixed issue with qwen image edit models when using multiple control images when not caching text embeddings. 2026-04-30 11:59:12 +00:00
Jaret Burkett deb409085a Made it possible to use the flux 2 small decoder VAE when setting the vae_path manually for flux2 models 2026-04-30 05:42:31 -06:00
Jaret Burkett 7ccec8ec2c Add checkpointing and a proper decode for flux 2 VAEs so they can be used with DFE 2026-04-30 04:27:13 -06:00
Jaret Burkett b4f0efb025 Version Bump 2026-04-28 13:42:00 -06:00
Jaret Burkett af6458d1b5 Enable caching of ACE step latents. 2026-04-28 13:39:20 -06:00
Jaret Burkett 77b8765939 Fix issue with hover styling on select components. 2026-04-28 10:55:20 -06:00
Jaret Burkett 43989cc19e Add advanced config section to the captioner 2026-04-28 10:49:46 -06:00
Jaret Burkett f972b750e6 Performance improvements to captioner. Add ability to set a default caption for ace captioner to avoid needing to caption the audio so we only transcribe 2026-04-28 10:15:11 -06:00
Jaret Burkett acc6a36214 Scale DFE 9 to a velocity equiv weight to match flow matching gradient strength. Probably need to rework all DFEs to do this as the math checks out. 2026-04-28 09:10:02 -06:00
Jaret Burkett 1fc4ad3979 Add sapiens2 as a diffusion feature extractor 2026-04-27 15:59:03 -06:00
Jaret Burkett 67d67f8c1d Ignore hidden files when captioning 2026-04-25 17:38:31 -06:00
Jaret Burkett 998a02f30e Yet another pseudo_huber fix. Still drinking coffee. 2026-04-19 10:05:40 -06:00
Jaret Burkett fc85410c9a Fix issue with precision on pseudo_huber loss 2026-04-19 10:02:27 -06:00
Jaret Burkett 20a99258b8 Fix issue is is else on pseudo_huber loss 2026-04-19 09:59:18 -06:00
Jaret Burkett f4445cd78c Added psuedo_huber loss 2026-04-19 09:51:46 -06:00
Jaret Burkett 488878f354 Use hidden layers in the loss for DFE 7 and 8 2026-04-18 13:07:38 -06:00
Jaret Burkett beb40ae29b Add DFE8 with partial step 2026-04-17 17:40:16 -06:00
Jaret Burkett 7c4f18ce51 Fix ernie unpatchify 2026-04-17 12:03:39 -06:00
Jaret Burkett 8cb9649382 Add decode latent to ernie pipe 2026-04-17 12:01:09 -06:00
Jaret Burkett 67048df9f9 Merge branch 'main' of github.com:ostris/ai-toolkit 2026-04-17 06:05:40 -06:00