n0p/mlx-video
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
135 Commits 1 Branch 0 Tags
main
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Prince Canuma 9ab4826d20 Merge pull request #22 from Blaizzy/pc/unify-apis 
Unify APIs and add LTX-2.3
 2026-03-18 22:04:21 +01:00
Prince Canuma 996a542011 Remove Wan2 model files, including configuration, attention mechanisms, and utility functions, to streamline the codebase and eliminate unused components. This cleanup enhances maintainability and focuses on the core functionality of the Wan2 module. 2026-03-18 17:59:43 +01:00
Prince Canuma b029668cd2 Refactor Wan model structure by renaming and relocating model imports from model.py to wan2.py, enhancing code organization and clarity across the Wan2 module. 2026-03-18 17:57:29 +01:00
Prince Canuma 6c63163671 Refactor Wan model imports and update script paths in pyproject.toml; transition from wan to wan2 module structure for improved organization and clarity. 2026-03-18 17:52:30 +01:00
Prince Canuma 17397da70c format 2026-03-18 17:40:05 +01:00
Prince Canuma 78bcfba31b Update README.md to reflect changes in command usage for Wan2.1 and Wan2.2 models, consolidating generation commands under the new mlx_video.wan2 module. 2026-03-18 17:38:49 +01:00
Prince Canuma 3e33172c12 Refactor and remove Wan2.1/2.2 model files; update README.md to include new model features and usage instructions for LTX-2 and Wan2 models. 2026-03-18 17:34:57 +01:00
Prince Canuma 95d7c81b20 Remove deprecated stubs for video conversion and generation; introduce new weight conversion and generation scripts for Wan2.2 models in MLX. 2026-03-18 17:20:36 +01:00
Prince Canuma 7b9d0a5e44 Merge branch 'main' into pc/unify-apis 2026-03-18 17:14:17 +01:00
Prince Canuma fea0f87df9 Fix token handling in LTX-2 text encoder by directly appending response tokens to the generated tokens list, improving clarity and consistency in token generation. 2026-03-18 13:50:33 +01:00
Prince Canuma f5e311a77c Update default values for STG and modality scales in LTX-2 video generation; enhance help descriptions for command-line arguments 2026-03-18 12:17:47 +01:00
Prince Canuma f8e371e9ce Enhance upsampler weight detection logic in LTX-2 model; improve clarity in comments and streamline spatial scale determination for x1.5 and x2 formats 2026-03-17 15:14:57 +01:00
Prince Canuma 57f66bcae2 Add custom spatial upscaling support to LTX-2 video generation; introduce spatial_upscaler parameter and enhance resolution handling for two-stage pipelines 2026-03-17 02:23:47 +01:00
Prince Canuma cc302d79b0 Refactor comments and optimize key skipping logic in LTX-2 model conversion; improve clarity in code documentation 2026-03-17 00:39:52 +01:00
Prince Canuma 643f250195 Update README.md with installation instructions, supported models, and usage examples; add new LTX-2 model documentation for pipelines and features. 2026-03-16 23:03:05 +01:00
Prince Canuma f9880a0683 Add audio encoder sanitization and configuration inference to LTX-2 model conversion process; update conversion print statements for new encoder step 2026-03-16 22:35:27 +01:00
Prince Canuma 7a576bfbf4 Refactor weight loading and utility functions for LTX-2 model; remove deprecated weight loading file and update imports accordingly 2026-03-16 22:25:22 +01:00
Prince Canuma dd573d53d2 Refactor audio VAE directory structure and update related paths in conversion and loading functions 2026-03-16 21:53:37 +01:00
Prince Canuma a6a6bb2166 Move weight loading functions to a new file for better organization and maintainability 2026-03-16 17:28:06 +01:00
Prince Canuma 3a0da19adb Refactor LTX-2 model structure 2026-03-16 14:50:01 +01:00
Prince Canuma decb3eb9e5 Add librosa dependency and enhance A2V documentation with additional pipeline options 2026-03-16 02:02:13 +01:00
Prince Canuma 6f6105b715 Add audio to video conditioning 2026-03-16 01:42:11 +01:00
Prince Canuma f53b9e0807 Add Dev Two-Stage HQ pipeline mode 2026-03-16 00:34:13 +01:00
Prince Canuma df81bc852f fix save tensors 2026-03-15 23:08:12 +01:00
Prince Canuma 38d46a6eda Implement regression tests for RoPE position precision using NumPy float64 reference. Add a new function to compute reference values and validate that float32 results closely match expected outputs, addressing high-frequency amplification issues. Update imports to include LTXModelConfig for enhanced configuration management. 2026-03-15 23:00:38 +01:00
Prince Canuma cecd68197c fix tiling, rope precision and weights 2026-03-15 22:58:55 +01:00
Prince Canuma ebcd5dd4e4 optimize memory usage by batching weight updates 2026-03-15 03:12:47 +01:00
Prince Canuma 53bae534e7 fix LTX-2.3 audio 2026-03-15 02:06:35 +01:00
Prince Canuma eb0d1355e4 Fix LTX-2.3 decoder grainy bug 2026-03-14 21:56:03 +01:00
Prince Canuma 5644492f7d Update generate.py to enhance denoising functionality with optional Spatiotemporal Guidance (STG) support. Modify DEFAULT_NEGATIVE_PROMPT for improved clarity and detail. Implement auto-detection of STG blocks based on transformer configuration. Refactor denoise_dev function to incorporate STG parameters, allowing for more flexible audio-visual integration during video generation. 2026-03-14 20:02:42 +01:00
Prince Canuma ffe271699a Refactor LoRA loading for v2.3 in generate.py to prioritize distilled-lora files over full model weights, enhancing model flexibility. Update key sanitization logic to utilize a replacement list for improved readability and maintainability. Modify denoise_dev_av function to include sigma parameters for audio and video modalities, ensuring consistent handling of latent variables during processing. Adjust Vocoder weight loading to allow for non-strict loading, accommodating additional keys in model weights. 2026-03-14 15:24:50 +01:00
Prince Canuma 9cba2ea7cd Enhance README.md with new usage examples for STG and modality scale parameters in video generation. Update generate.py to support STG and modality guidance in the denoising process, allowing for improved audio-visual integration. Refactor attention mechanisms in the transformer to include options for skipping self-attention, facilitating STG perturbation and modality isolation. Update LTXModel and transformer block processing to accommodate new parameters for enhanced flexibility in model configurations. 2026-03-14 10:26:12 +01:00
Prince Canuma f346e09de4 Refactor audio handling in generate_video function to preserve stage 1 audio latents during stage 2 processing. Remove redundant audio re-denoising steps, ensuring audio integrity while refining video output. Update comments for clarity on audio processing logic. 2026-03-13 16:09:07 +01:00
Prince Canuma 387d4fc301 improve dev color and quality 2026-03-13 09:51:24 +01:00
Prince Canuma 835ba33202 Enhance README.md with detailed descriptions of LTX-2 features, pipeline options, and usage examples for text-to-video, image-to-video, and audio-video generation. Update generate.py to improve LoRA loading functionality, allowing for local files, directories, or HuggingFace repos. This update improves flexibility in model configurations and enhances user guidance in the documentation. 2026-03-13 01:39:39 +01:00
Prince Canuma 7435facc52 Add support for DEV_TWO_STAGE pipeline and implement LoRA merging functionality in generate.py. Enhance video generation capabilities by allowing LoRA weights to be loaded and merged into the model, improving flexibility in model configurations. Update pipeline handling to accommodate the new two-stage generation process. 2026-03-13 01:22:45 +01:00
Prince Canuma e0aafd72fc Refactor generate.py to ensure temporal coordinates and position grids are processed in bfloat16 for consistency with PyTorch's precision behavior. Update denoise_dev_av function to apply standard ratio rescaling for audio and video guidance, enhancing numerical fidelity and model compatibility. 2026-03-12 21:26:38 +01:00
Prince Canuma b07b1e3213 Update .gitignore to exclude additional configuration and model files. Modify generate.py to enhance console output with rescale parameter and adjust default values for inference steps and CFG scale. Refactor text encoder to align positional embedding max position with PyTorch defaults, improving compatibility and performance. 2026-03-12 17:13:43 +01:00
Prince Canuma 3618966625 Wan2.1 and Wan2.2 model support, including LoRA support & more Poodles 2026-03-11 19:08:14 +01:00
Prince Canuma d1fa47722b Fix timestep_conditioning logic in infer_vae_decoder_config to ensure consistent behavior based on has_timestep flag. 2026-03-11 18:30:29 +01:00
Prince Canuma 207c223354 Add LTX-2.3 model architecture with prompt-conditioned adaptive layer normalization (adaln) support. Introduce gating mechanisms in attention modules and update transformer configurations to accommodate new parameters. Refactor video and audio processing to utilize adaptive normalization, improving model flexibility and performance. Update weight loading and initialization logic to support dynamic block structures in the decoder. 2026-03-10 16:47:36 +01:00
Prince Canuma d028b239fb Update LTX conversion script to support LTX-2.3 safetensors format. Enhance documentation and improve file matching logic for variant detection in local directories. 2026-03-10 08:01:26 +01:00
Prince Canuma 576e01da14 Implement linking of text encoder and tokenizer directories in conversion process. Enhance error handling in LTX2TextEncoder for tokenizer loading, providing a fallback model if the specified path is unavailable. 2026-03-09 18:25:32 +01:00
Prince Canuma 41ed62f7e8 Add LTX-2 conversion script for safetensors to MLX directory layout. Implement modular structure 2026-03-09 18:16:20 +01:00
Prince Canuma 9f37dab076 Refactor model loading in generate.py to use dynamic model paths for audio and video components. Simplify weight loading logic in LTX2TextEncoder to accommodate both monolithic and reformatted model structures. Introduce a check for existing model paths in get_model_path function to enhance robustness. 2026-03-09 15:51:21 +01:00
Prince Canuma d1dd30cbac Add Adaptive Projected Guidance (APG) support to denoising functions. Introduce apg_delta function for stable guidance by decomposing into parallel and orthogonal components. Update denoise_dev and generate_video functions to accept APG parameters, enhancing flexibility in video generation. Modify command-line arguments for APG integration. 2026-01-26 21:35:58 +01:00
Prince Canuma 87962c7f83 Enhance precision in denoising functions by ensuring all latents and calculations are consistently handled in float32. Update model input casting and return types to maintain dtype integrity across audio and video processing. Add precision parameter to video generation for improved memory management. 2026-01-24 15:40:42 +01:00
Prince Canuma cb2d19c84d fix loading 2026-01-24 01:37:38 +01:00
Prince Canuma ef76ec0921 add from pretrained 2026-01-23 18:13:51 +01:00
Prince Canuma ce39e744c3 Refactor VideoEncoder to initialize from VideoEncoderModelConfig, enhancing configuration management. Add methods for weight sanitization and loading from pretrained models, improving model usability and integration with existing workflows. 2026-01-23 17:59:57 +01:00
Prince Canuma f8f78aeab5 Add LTXModel with a from_pretrained class method for loading model weights from a specified path. Update weight sanitization to handle positional embeddings and dtype consistency. Refactor timestep and context preparation methods to accept hidden_dtype, improving flexibility in model processing. 2026-01-23 17:45:50 +01:00
Prince Canuma df753312c7 Refactor video generation and model loading processes to utilize from_pretrained methods for VideoEncoder and VideoDecoder. Update denoising functions to include a cfg_rescale parameter for improved artifact reduction. Ensure consistent dtype handling across audio and video processing, enhancing precision and aligning with PyTorch behavior. 2026-01-23 17:39:02 +01:00
Prince Canuma 02bfa228d9 Refactor weight loading and sanitization processes for audio models 2026-01-23 17:31:25 +01:00
Prince Canuma 7a74946c57 Merge pull request #14 from Blaizzy/pc/add-streaming 
Add --stream flag and chunked conv memory optimization for VAE decoding
 2026-01-21 15:42:55 +01:00
Prince Canuma ffdeec72a6 Merge branch 'main' into pc/add-streaming 2026-01-21 15:42:16 +01:00
Prince Canuma 7ad14e18ca Merge pull request #12 from Blaizzy/pc/add-vae-tiling 
Add VAE Tiling + BFloat16 Support for Memory-Efficient Video Generation
 2026-01-21 15:41:46 +01:00
Prince Canuma 2681f75d2f Refactor LTXModel to include a from_pretrained class method for loading and sanitizing model weights. Update generate.py to utilize this method, streamlining the transformer loading process and improving code clarity. 2026-01-20 12:56:29 +01:00
Prince Canuma bbb3de6aa7 Update audio decoder configuration to disable mid-block attention and ensure audio waveform is converted to float32 for consistency in processing. 2026-01-19 17:05:59 +01:00
Prince Canuma 8a2ea38c88 Refactor denoising functions in generate.py and utils.py to use float32 for improved precision, aligning with PyTorch behavior. Update calculations for latents and denoised outputs to ensure consistent dtype handling across audio and video processing. 2026-01-19 09:13:04 +01:00
Prince Canuma e0ee934b99 Update video generation completion message to display elapsed time in a more user-friendly format, showing minutes and seconds instead of just seconds. 2026-01-19 02:23:51 +01:00
Prince Canuma 4cd58f8b26 Refactor LTX2TextEncoder to utilize Rich for progress tracking during token generation. Replace tqdm with Rich's Progress for enhanced console output and user experience. Clean up imports and streamline the generation process. 2026-01-19 02:13:10 +01:00
Prince Canuma ac67ee8b1e Remove the generate_dev.py file, consolidating its functionality into generate.py. Enhance the video generation pipeline to support both distilled and dev models, integrating dynamic sigma scheduling and classifier-free guidance (CFG) for improved video quality. Update command-line interface to accommodate new pipeline options and refactor related functions for better maintainability. 2026-01-19 02:13:00 +01:00
Prince Canuma 0538af6554 Enhance video generation pipeline by integrating Rich for styled console output and progress tracking. Update dependencies in pyproject.toml to include Rich. Refactor print statements to use console methods for improved user experience during video and audio processing. 2026-01-19 01:43:14 +01:00
Prince Canuma cae11291a9 Remove the audio-video generation pipeline from generate_av.py and integrate audio capabilities into generate.py. This includes adding audio position grid creation, audio frame computation, and updating the denoising function to handle audio latents. Enhance the command-line interface to support audio generation options and update the model configuration accordingly. 2026-01-19 01:28:53 +01:00
Prince Canuma 749762a0b9 Update audio decoder configuration to use an empty set for attention resolutions in both generate_av.py and generate_dev.py. Add a print statement for loading audio VAE decoder weights in generate_dev.py. 2026-01-18 21:55:38 +01:00
Prince Canuma 7069cc39c9 Add audio generation capabilities to video pipeline, including audio position grid creation, audio frame computation, and integration of audio VAE and vocoder. Update tests to cover new audio functionalities. 2026-01-18 21:28:56 +01:00
Prince Canuma b36ad1e22d add tests 2026-01-18 11:18:18 +01:00
Prince Canuma e483eab039 Optimize positional embedding handling in TransformerArgsPreprocessor and improve RoPE frequency computation in _precompute_freqs_cis_double_precision for enhanced performance and precision. 2026-01-18 11:13:32 +01:00
Prince Canuma 62fc4805a0 Add LTX-2 Dev Model video generation pipeline 2026-01-18 11:13:11 +01:00
Prince Canuma b1bf9e2dc0 Enhance video generation with progress bar for streaming and remove debug prints from tiling decoder 2026-01-17 23:53:53 +01:00
Prince Canuma f256c5fb25 add tests 2026-01-17 23:36:39 +01:00
Prince Canuma 7f20840dc7 Add streaming support to video generation 2026-01-17 23:17:08 +01:00
Prince Canuma f33f496fba Merge branch 'main' into pc/add-vae-tiling 2026-01-17 19:37:21 +01:00
Prince Canuma e692b7a6b3 Add i2v 
Add i2v
 2026-01-17 19:37:06 +01:00
Prince Canuma 785b0b955d Merge branch 'main' into pc/add-i2v 2026-01-17 19:36:28 +01:00
Prince Canuma 26fa8919ed Merge pull request #13 from Blaizzy/Blaizzy-patch-1 
Update actions
 2026-01-17 19:36:14 +01:00
Prince Canuma c89de996eb Update GitHub Sponsors username in FUNDING.yml 2026-01-17 19:35:24 +01:00
Prince Canuma 0669998e15 Add audio support 
Add audio support
 2026-01-17 19:31:21 +01:00
Prince Canuma 61c56cd989 Add RoPE tests and warning for bfloat16 precision loss in RoPE calculations 2026-01-17 19:28:05 +01:00
Prince Canuma 78244a2d66 Cast dtype to bf16 in video and audio generation processes 2026-01-17 17:20:22 +01:00
Prince Canuma 883c6b0ad8 ensure dtype cast 2026-01-17 13:03:48 +01:00
Prince Canuma e4cdbb7eab add vae tiling 2026-01-17 07:51:54 +01:00
Prince Canuma f607112407 Refactor video and audio latent generation in generate_video and generate_video_with_audio 
- Removed direct initialization of latents with random noise, replacing it with a conditional approach based on I2V (Image-to-Video) conditioning.
- Introduced a structured flow for applying noise during the latent state creation, enhancing the conditioning process for both video and audio.
- Updated the noise application logic to ensure proper handling of conditioned and unconditioned frames in both stages of video generation.
- Improved code clarity and maintainability by consolidating latent shape definitions and restructuring noise application logic.
 2026-01-17 01:38:53 +01:00
Prince Canuma d52e567c56 Enhance precision in denormalization and normalization processes 
- Updated `denormalize` and `pixel_norm` methods in `LTX2VideoDecoder` and `PerChannelStatistics` classes to cast mean and standard deviation to float32 for improved precision.
- Ensured that the output of normalization operations retains the original data type of the input tensor.
 2026-01-17 01:14:29 +01:00
Prince Canuma ecda6d10e5 Merge pull request #9 from Blaizzy/pc/fix-text-encoder 
Fix text encoder
 2026-01-17 01:10:36 +01:00
Prince Canuma 146f5d2981 Add image-to-video (I2V) conditioning support 
- Introduced `load_image`, `prepare_image_for_encoding`, and `apply_conditioning` functions for handling image inputs and conditioning during video generation.
- Enhanced `generate_video` and `denoise_av` functions to accept optional image inputs for I2V conditioning.
- Updated command-line interface to include parameters for image conditioning, such as `--image`, `--image-strength`, and `--image-frame-idx`.
- Added new `VideoConditionByLatentIndex` and `LatentState` classes for managing latent states with conditioning.
- Implemented VAE encoder loading and image encoding for conditioning in the video generation process.d
 2026-01-17 00:19:52 +01:00
Prince Canuma 5f86e881d7 Update top_p parameter in sampler function to 1.0 for enhanced sampling control in LTX2TextEncoder 2026-01-16 21:08:14 +01:00
Prince Canuma f6e0e5d5a4 Update av_ca_timestep_scale_multiplier to 1000 in model configuration for consistency across modules 2026-01-16 15:59:22 +01:00
Prince Canuma e1bff927df Auto-detect timestep_cond from model metadata () 2026-01-16 14:55:50 +01:00
Prince Canuma a658911f98 add audio 2026-01-16 01:15:22 +01:00
Prince Canuma 81daf3f67d Add prompt enhancement feature to video generation 
- Introduced `enhance_prompt`, `max_tokens`, and `temperature` parameters in `generate_video` function for improved prompt handling.
- Implemented prompt enhancement logic using the new `enhance_t2v` method in the text encoder.
- Added command-line arguments for prompt enhancement options.
- Created new system prompt files for T2V and I2V generation to guide the enhancement process.
 2026-01-15 14:31:00 +01:00
Prince Canuma f5134fa172 adjust gelu and precision 2026-01-15 12:49:21 +01:00
Prince Canuma 349a82f763 Refactor GroupNorm3d: Optimize data type handling by casting input, weight, and bias to float32 for consistency and performance 2026-01-15 04:46:56 +01:00
Prince Canuma 09c2b460a7 Refactor LTX2VideoDecoder and ResBlockGroup: Change up_blocks and res_blocks from lists to dictionaries for better parameter tracking in MLX 2026-01-15 03:48:16 +01:00
Prince Canuma 3fcd8f90be Refactor LTXModel: Change transformer_blocks from list to dictionary 2026-01-15 03:47:52 +01:00
Prince Canuma e7067fea11 Refactor LTX2VideoDecoder: Remove redundant comments for residual parameter 2026-01-14 01:21:43 +01:00
Prince Canuma 957093c29b use numpy for improved float64 precision and performance 2026-01-14 00:03:00 +01:00
Prince Canuma 74af04718d Remove commented-out code and clean up text encoder initialization 2026-01-13 23:31:54 +01:00
Prince Canuma ea063f7550 Cast LM weights to bfloat16 2026-01-13 23:30:26 +01:00
Prince CanumaandHimanshU Mourya fc6ef20c1b Add custom text encoder with quantization 
Co-authored-by: HimanshU Mourya <40685364+codingstark-dev@users.noreply.github.com>
 2026-01-13 22:56:51 +01:00