evan/ai-toolkit
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,166 Commits 2 Branches 0 Tags
cac3815b2c4a1e6751ea5c41b152d5ddab0773f0
Commit Graph
1166 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Jaret Burkett cac3815b2c Make DOP run in a single backward pass, should be faster and more stable. Show dop loss on ui 2026-06-08 13:55:47 -06:00
Jaret Burkett 687def6f7a Add better smoothing with ema rounding of the ends of the loss graph for first and last instance do not dominate the smoothing factors. 2026-06-08 10:43:46 -06:00
Jaret Burkett d7f8887bbf Show all logged metrics on the loss graph 2026-06-08 09:43:03 -06:00
Jaret Burkett e281df70dd Allow automagic3 to run unfused. Add some clipping. 2026-06-08 08:59:06 -06:00
Jaret Burkett c9cdbb5bb7 Version bump 2026-06-07 16:08:11 -06:00
Jaret Burkett c78b1404e3 Deepen offload prefetch pipeline with per-slot events 
Replace the 2-slot ping-pong + single global "compute-started" event
with a depth-N ring buffer where each transfer waits only on the slot
it's reusing (D layers back) instead of the most-recent compute. Applies
to forward and backward, Linear and Conv. Depth is tunable via
AI_TOOLKIT_OFFLOAD_DEPTH (default 4).

Bit-exact vs non-offload (output, grad_input, weight grads). No speedup
on a bandwidth-bound PCIe link (already saturated at depth 2), but the
cleaner per-slot design removes the fragile shared-event serialization
and lets deeper prefetch help on faster buses.
 2026-06-07 16:07:13 -06:00
Jaret Burkett cdff6e36aa Pin inner stores of torachao to speed up layer offloading for quantized models around 25% 2026-06-07 15:52:01 -06:00
Jaret Burkett 75781fb5a5 Fix float8 weights not offloading to CPU in layer offloading 2026-06-07 15:36:17 -06:00
PlagueKind 7c1a76f336 Fix text encoder offload bug when caching embeddings (#868) 2026-06-07 14:09:32 -06:00
Jaret Burkett 35588726de Fixed issue where a buffer was stuck on cpu when offloading ideogram4 2026-06-07 13:31:03 -06:00
Jaret Burkett 1dc9a797cf Added Automagic v3 2026-06-07 12:06:42 -06:00
Jaret Burkett 82190b41e6 Fix bug where EMA was not being initialized even when setup in the config. EMA will now properly be setup and used. 2026-06-07 11:11:49 -06:00
Jaret Burkett 8968e41234 Set ideogram to use new loke saving format 2026-06-07 09:49:57 -06:00
Jaret Burkett fa0dca288d Version bump 2026-06-06 10:24:31 -06:00
Jaret Burkett 41157b460c Added ability to set the caption extention in dataset viewer, captioner, and trainer so one dataset can have multiple caption styles in different files with different extensions. Added dataset caption template for a blank ideogram 4 formatted template. 2026-06-06 08:32:24 -06:00
Jaret Burkett 10cdeb394e Version bump 2026-06-05 14:09:46 -06:00
Jaret Burkett 21a6beb194 Allow editing the boxes visually on the sample section for ideogram 2026-06-05 14:09:18 -06:00
Jaret Burkett 4441080c05 Rework the ui of ideogram dataset image caption editor 2026-06-05 13:20:58 -06:00
Jaret Burkett b70083a74f Fix issue with default sampels. 2026-06-05 12:33:42 -06:00
Jaret Burkett c994398850 Rework prompts and captioning systems for ideogram to more strictly match the format provided by ideogram. 2026-06-05 11:14:36 -06:00
Jaret Burkett 6fd2253932 Imporved editing boxed on the dataset viewer 2026-06-05 10:30:06 -06:00
Jaret Burkett ef12260b80 Add a prompt upsample ui for upsampling prompts to ideogram format prompts. 2026-06-05 09:56:02 -06:00
Jaret Burkett 90a2084f70 Allow adjusting, adding, and deleting bounding boxes. 2026-06-04 20:29:31 -06:00
Jaret Burkett bb60f6d1d1 Show bounding boxes on the sample image card if we have them in the prompt 2026-06-04 15:59:02 -06:00
Jaret Burkett edcc7415d1 Added an autocaptioner for Ideogram 4 captions 2026-06-04 13:04:23 -06:00
Jaret Burkett 6a8d9333b6 Improved the prompt handeling of ideogram4 model. Now used advanced prompts class to store them smaller and allow longer prompts 2026-06-04 13:00:23 -06:00
Jaret Burkett 2ddc2e1318 Update the default Ideogram 4 prompts to work significantly better. 2026-06-04 10:32:05 -06:00
Jaret Burkett 63b3181262 Added experimental support for Ideogram 4 2026-06-04 09:03:00 -06:00
Jaret Burkett b5f21ae695 Fixed issue where the training type dropdown in the ui was not showing fully 2026-06-03 10:14:15 -06:00
Jaret Burkett d9f26c2f87 Add gradient checkpointing to tipsv2 heads 2026-06-01 05:00:15 -06:00
Jaret Burkett bd468727a6 Fixed issue where some image cards in samples and datasets would show gray instead of the image until you scroll. 2026-06-01 03:36:32 -06:00
Jaret Burkett f5446c0d5f Version bump 2026-06-01 02:16:47 -06:00
Jaret Burkett e5439509b5 Added pure lpips dfe 2026-05-31 11:52:03 -06:00
Jaret Burkett 212cfe998a Add ability to download or delete the optimizer state from the ui 2026-05-29 08:56:22 -06:00
Jaret Burkett 5e84bf0d0b Fixed issue with hidream-01 that could cause a weird nan state. Took forever to track down as it was 1 in 10 starts. 2026-05-28 12:48:24 -06:00
Jaret Burkett 87bac27513 Fixed issue with new bucket scaler 2026-05-28 11:34:16 -06:00
Jaret Burkett 30886b8f92 Added a wallet indicator for Ostris Cloud 2026-05-28 11:09:51 -06:00
Jaret Burkett 3e86d81fc6 Adjust bucket sizes to achieve maximum pixels without going over. 2026-05-28 09:36:17 -06:00
Jaret Burkett ef57c1077c Change z image divisibility 2026-05-28 09:09:36 -06:00
Jaret Burkett 15082cfb8a Round buckets for divisibility instead of always rounding down. 2026-05-28 09:08:21 -06:00
Jaret Burkett c9264bdd0b Reworked the bucketing system to precisly match model specific divisibility. The old SDXL bucket sizes needed to go. 2026-05-28 08:35:58 -06:00
Jaret Burkett 68e9b38220 Add ability to trigger a save from the ui which whill make the trainer save on the next step 2026-05-26 08:38:49 -06:00
Jaret Burkett 2aa60e4ca5 Update default agreement threshold for automagic v2 to be 0.5 2026-05-26 07:57:54 -06:00
Jaret Burkett 76c99da4e4 Add version to the ui 2026-05-26 07:30:09 -06:00
Jaret Burkett 266956068a Add a way to delete checkpoints on the ui 2026-05-26 07:09:21 -06:00
Jaret Burkett 954c5efec8 Show job info on sidebar active job in ui 2026-05-25 11:07:29 -06:00
Jaret Burkett 083236a2a7 Drastically improve the loading speed of images in the ui by using a custom loader and abort controller to abort when images leave the view. 2026-05-25 10:04:25 -06:00
Jaret Burkett 7354def271 Added decode latent method to qwen image model 2026-05-25 10:02:09 -06:00
Jaret Burkett 3d836ac371 Show active jobs in the sidebar of the ui 2026-05-25 08:21:42 -06:00
Jaret Burkett a798e06dd2 Updated the support button to look more like a button 2026-05-25 07:48:04 -06:00