[Wan 2.2 LoRA] add support for 2nd transformer lora loading + wan 2.2 lightx2v lora #12074
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
curious to see an example @linoytsaban would love to try this out |
sayakpaul
reviewed
Aug 5, 2025
Member
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Thanks for working on this. Left some comments.
| """ | ||
|
|
||
| _lora_loadable_modules = ["transformer"] | ||
| _lora_loadable_modules = ["transformer", "transformer_2"] |
Member
There was a problem hiding this comment.
Just to note that this loader is shared amongst Wan 2.1 and 2.2 as the pipelines are also one and the same. For Wan 2.1, we won't have any transformer_2.
Comment on lines
5283
to
5293
| else: | ||
| self.load_lora_into_transformer( | ||
| state_dict, | ||
| transformer=getattr(self, self.transformer_name) if not hasattr(self, | ||
| "transformer") else self.transformer, | ||
| adapter_name=adapter_name, | ||
| metadata=metadata, | ||
| _pipeline=self, | ||
| low_cpu_mem_usage=low_cpu_mem_usage, | ||
| hotswap=hotswap, | ||
| ) |
Collaborator
Author
There was a problem hiding this comment.
my thought process was that, as opposed to LoRAs with weights for the transformer and text encoder for example, that we load in one load_lora_weights op, here we can have a situation where we have different weights for each transformer, but the state_dict keys are identical. Also, this way we can load the lora into each transformer separately with different adapter names - making it easy to use different scales for each transformer lora (which was seen to be beneficial for quality). I'm happy to improve this logic, but these are the considerations to keep in mind
Member
There was a problem hiding this comment.
Yeah. So, in case users want to load both transformers, won't it just load one if load_into_transformer_2=True?
Collaborator
Author
There was a problem hiding this comment.
yep it would, they would need to load separately to each
Member
There was a problem hiding this comment.
Can you show some pseudo-code expected from the users? This is another way of loading another adapter into transformer_2:
#12040 (comment)
Collaborator
Author
There was a problem hiding this comment.
I don't feel strongly about it staying that exact way, but i do think it should remain possible to load different lora weights into the transformers and in different scales
Member
There was a problem hiding this comment.
Makes sense. Let's go with this but with a note in the docstrings saying it's experimental in nature.
Collaborator
Author
|
I2V example: using Wan2.2 with Wan2.1 lightning LoRA i2v_output-84.mp4 |
|
thanks a lot for the amazing work @linoytsaban just FYI issue #12047 also applies to this PR, I tried and I get the mismatch error with GGUF models, reporting as they are the most popular way to run Wan on consumer hardware. |
|
@linoytsaban are we sure if we don't put boundary_ratio args in our generation pipe would still choose transformer2 as low noise ? Bcs I can see first PR on wan2.2 #12004 by @yiyixuxu has these lines |
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Collaborator
Author
yes @mayankagrawal10198 it should still use |
luke14free
reviewed
Aug 7, 2025
Collaborator
Author
|
@bot /style |
Contributor
|
Style fix is beginning .... View the workflow run here. |
Collaborator
Author
|
@bot /style |
Contributor
|
Style bot fixed some files and pushed the changes. |
|
Hey Guys this is amazing work.. There is now a new concept to do this in 3 stages 3 stage approach==> The first stage uses the original WAN2.2 model, without Lightx2v lora. This allows for faster motions to be generated. The 2nd and 3rd stage uses the High and Low Lightx2v loras like normal. I will do some experiment on this :) |
sayakpaul
reviewed
Aug 18, 2025
Member
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Left some comments.
Let's also make a note about it in the docs?
Member
|
Hi @mayankagrawal10198 you can refer to this PR for the boundary explanation, it has some nice images explaining it. Also the steps can't be floats, so they get rounded. This is the same for every other If you want more details maybe you can open a question in the discussion tab so we can keep this PR clean and with comments about it and not more general unrelated questions. |
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Collaborator
Author
|
Thanks a lot @sayakpaul! |
|
I assume its not compatible with .gguf version rn? awesome work, thanks |
|
We merged the wan2.2 diffusers base model and wan2.2-lightning model to an new model: https://huggingface.co/FastDM/Wan2.2-T2V-A14B-Merge-Lightning-V1.0-Diffusers |
|
Thank you very much for your work. When I was using wan22 use wan21_lightx2v loracode to load Lora, I encountered this error code This result indicates that I seem to have encountered keys different from the original transformer and transformer_2 when loading Lora, but when I printed the structure of the model, I seemed to have successfully loaded Lora?Is this just a prompt error or is there a problem with my usage？ |
Member
|
Please provide a complete yet minimal snippet for debugging. |
torch.compile + channels_last support for Wan 2.2 (T2V / I2V) fails with RuntimeError + Dynamo Unsupported behaviorIssue Description Hi, I am trying to optimize Wan 2.2 T2V / I2V inference speed on a single RTX 4090, using:1 Wan2.2 (Diffusers) My goal is to achieve maximum throughput on a single 4090 GPU. However, when following the official docs for efficiency: https://huggingface.co/docs/diffusers/api/pipelines/wan#t2v-inference-speedI hit two different failures: 1. RuntimeError when calling .to(memory_format=torch.channels_last)According to the docs: I got the error 2 When skipping channels_last and compiling directly, torch.compile fails at runtimeI attempted:# Skipped channels_last I got the error full codeRequest1 Can Wan2.2 Transformer support channels_last? 3 Are there recommended compiler flags |
Member
|
Full code is not reproducible because it contains paths relative to your system.
No, because (you should open the issue on PyTorch repository because we don't control this behaviour):
We support it. From your error-trace, you seem to have enabled group offloading which is not compatible with from user code:
File "/data/conda_envs/haobang.geng/lib/python3.10/site-packages/diffusers/hooks/hooks.py", line 189, in new_forward
output = function_reference.forward(*args, **kwargs)
File "/data/conda_envs/haobang.geng/lib/python3.10/site-packages/diffusers/hooks/hooks.py", line 188, in new_forward
args, kwargs = function_reference.pre_forward(module, *args, **kwargs)
File "/data/conda_envs/haobang.geng/lib/python3.10/site-packages/diffusers/hooks/group_offloading.py", line 304, in pre_forward
self.group.onload_()
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Wan 2.2 has 2 transformers, the community has found it to be beneficial to load Wan LoRAs into both transformers and occasionally in different scales as well (this also applies for Wan 2.1 LoRAs, loaded into
transformerandtransformer_2).Recently, new lighting LoRA was released for Wan2.2 T2V- with separate weights for
transformer(High noise stage) andtransformer_2(Low noise stage)This PR adds support for LoRA loading into
transformer_2+ adds support for lightning LoRA (hasalphakeys)T2V example:
t2v_out-5.mp4