SCAIL-2
Official Implementation of SCAIL-2: Unifying Controlled Character Animation with End-to-end In-Context Conditioning
What is it?
What it is
SCAIL-2 is an open-source end-to-end character animation model that eliminates intermediate pose representations by utilizing a unified motion transfer interface with masking channels and RoPE design. It supports complex motions, cross-identity replacement, and animal-driven scenarios through in-context conditioning.
Why it exists
To address limitations in prior methods reliant on intermediate pose representations (e.g., skeleton maps), which hindered complex motion handling and identity generalization. SCAIL-2 bypasses intermediates by training on 60K motion pairs from off-the-shelf models and introduces bias-aware DPO for improved details.
Who should use it
Developers, researchers, and animators with technical expertise in AI/ML, Python, and character animation workflows.
Who should avoid it
Non-technical users, casual users without Python/ML experience, or those seeking a simple plug-and-play tool.
How it works
A quick walkthrough in plain English
How SCAIL-2 works
Step 1 of 3
You interact with it
Open SCAIL-2, send a request, or connect it to your stack.
Features
Advantages
- Eliminates reliance on intermediate poses, enhancing generalization
- Handles diverse driving sources beyond human movements
- Enables complex animations like cross-identity and animal-driven
- Zero-shot multi-reference allows varied inputs without prior training
- DPO LoRA improves visual details and temporal consistency
- Open-source with active community support and documentation
Disadvantages
- Complex setup requiring multiple components (SCAIL-Pose, specific input formats)
- Multi-reference mode not optimized, leading to potential quality degradation
- Dependency on external models for data synthesis may limit flexibility
- Requires precise mask creation for effective results
- High computational resources needed for inference
Installation
FAQ
What are the mask color semantics in SCAIL-2 and how do they affect animation vs replacement modes?
In SCAIL-2 the mask uses three color channels: black marks background that should be hidden, white marks background that should be visible, and colored regions encode the correspondence between character parts and the driving motion. For animation mode the mask provides control signals; for replacement mode the mask_video defines the region to be swapped. Supplying an incorrect mask can cause the model to fall back to replacement‑mode behavior even in animation mode.
How do I set up the environment and dependencies for running SCAIL-2 inference?
First ensure Python 3.10‑3.12 is installed. Then run `pip install -r requirements.txt` to install the core dependencies. Next initialize the SCAIL‑Pose submodule with `git submodule update --init --recursive` and follow its setup: install its requirements and download the pose‑preprocessing weights into `SCAIL-Pose/pretrained_weights` as described in the README.
What input files are required for generate.py and how should they be prepared using SCAIL-Pose?
The inference script expects four local files: a reference image (`--image`), its foreground mask (`--mask_image`), a driving/pose video (`--pose`), and a per‑frame mask video (`--mask_video`). For character replacement add `--replace_flag` and supply the replacement‑region mask via `--mask_video`. These files can be generated automatically with the SCAIL‑Pose tools (`process_animation_aio.py` for end‑to‑end or pose‑driven mode, and `process_replacement.py` for replacement).
How can I use the DPO LoRA or other LoRA checkpoints with SCAIL-2?
To apply a LoRA checkpoint, pass its path with `--lora_path` and set the strength via `--lora_alpha` (e.g., `--lora_alpha 1.0`). For the DPO LoRA you can either check out the `sat-scail2` branch to reproduce the original results, or convert the LoRA to safetensors and load it in the same way using `--lora_path` and an appropriate `--lora_alpha` value.
How does multi-reference inference work and what mask guidelines should I follow for extra reference images?
Multi‑reference inference is enabled by providing lists of extra reference images (`--additional_ref_image`) and matching masks (`--additional_ref_mask_image`). Mask semantics: use a white mask for background you want preserved, black for unrelated background, and colored regions that match the main reference’s character parts. To reduce artifacts you can mock the extra references as short videos, as demonstrated in community workflows like WanAnimatePlus.
Where can I download the SCAIL-2 model checkpoints and what are the recommended resolutions?
Model weights are available on Hugging Face (`zai-org/SCAIL-2`) and ModelScope (`ZhipuAI/SCAIL-2`). The download includes `Wan2.1_VAE.pth`, the `model/` directory with FSDP checkpoints, and the `umt5-xxl` folder. For end‑to‑end animation both 512p and 704p are supported; pose‑driven works best at 704p. Height and width must be multiples of 32 (e.g., 704×1280).
Featured in Videos
YouTube tutorials and walkthroughs for SCAIL-2
Alternatives
Similar projects ranked by category, topics, and text overlap.
