Spaces:
Running
on
Zero
Running
on
Zero
Commit
·
b3980c8
1
Parent(s):
13a5cf7
docs: update app page
Browse files- .gitignore +1 -3
- app.py +25 -13
.gitignore
CHANGED
|
@@ -28,6 +28,4 @@ preprocess/sam2_code
|
|
| 28 |
!preprocess/oneformer_code/oneformer/data/bpe_simple_vocab_16e6.txt
|
| 29 |
!config/*.json
|
| 30 |
!requirements.txt
|
| 31 |
-
!requirements/*
|
| 32 |
-
!__assets__/*
|
| 33 |
-
!__assets__/page/*
|
|
|
|
| 28 |
!preprocess/oneformer_code/oneformer/data/bpe_simple_vocab_16e6.txt
|
| 29 |
!config/*.json
|
| 30 |
!requirements.txt
|
| 31 |
+
!requirements/*
|
|
|
|
|
|
app.py
CHANGED
|
@@ -72,25 +72,37 @@ MARKDOWN = \
|
|
| 72 |
|
| 73 |
</div>
|
| 74 |
|
| 75 |
-
Frame In-N-Out
|
| 76 |
-
Further, it allows users to assign motion trajectories to existing objects, introduce new identities that enter the scene with their own trajectories, or both.<br>
|
| 77 |
The model we used here is [<b>Wan2.2-5B</b> V1.6](https://huggingface.co/uva-cv-lab/FrameINO_Wan2.2_5B_Stage2_MotionINO_v1.6) trained on our Frame In-N-Out control mechanism.
|
| 78 |
|
| 79 |
|
| 80 |
<br>
|
| 81 |
-
<
|
| 82 |
-
Choose one
|
|
|
|
|
|
|
| 83 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 84 |
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
<br>
|
| 86 |
-
❗️❗️❗️Instruction Steps:<br>
|
| 87 |
-
1️⃣ Upload your first frame image. Set the size you want to resize to for <b>Resized Height for Input Image</b> and <b>Resized Width for Input Image</b>. <br>
|
| 88 |
-
2️⃣ Set your <b>canvas top left</b> and <b>bottom right expansion</b>. The combined height and width should be the multiplier of 32. <br>
|
| 89 |
-
Recommend <b>Canvas HEIGHT = 704</b> and <b>Canvas WIDTH = 1280</b> for the best performance (Pre-trained training Resolution). <br>
|
| 90 |
-
3️⃣ Click <b>Build the Canvas</b>. <br>
|
| 91 |
-
4️⃣ Provide the trajectory of the main object in the canvas by clicking on the <b>Expanded Canvas</b>. <br>
|
| 92 |
-
5️⃣ Provide the ID reference image and its trajectory (optional). Also, write a detailed <b>text prompt</b>. <br>
|
| 93 |
-
Click the <b>Generate</b> button to start the Video Generation. <br>
|
| 94 |
|
| 95 |
|
| 96 |
If **Frame In-N-Out** is helpful, please help star the [GitHub Repo](https://github.com/UVA-Computer-Vision-Lab/FrameINO?tab=readme-ov-file). Thanks!
|
|
@@ -804,7 +816,7 @@ if __name__ == '__main__':
|
|
| 804 |
with gr.Row():
|
| 805 |
|
| 806 |
# Button
|
| 807 |
-
generation_btn = gr.Button(value="Generate")
|
| 808 |
|
| 809 |
|
| 810 |
with gr.Row():
|
|
|
|
| 72 |
|
| 73 |
</div>
|
| 74 |
|
| 75 |
+
Frame In-N-Out expands the first-frame to a larger canvas, where it allows users to assign motion trajectories to existing objects and introduce new identities that enter the scene with specified trajectories.<br>
|
|
|
|
| 76 |
The model we used here is [<b>Wan2.2-5B</b> V1.6](https://huggingface.co/uva-cv-lab/FrameINO_Wan2.2_5B_Stage2_MotionINO_v1.6) trained on our Frame In-N-Out control mechanism.
|
| 77 |
|
| 78 |
|
| 79 |
<br>
|
| 80 |
+
<p style="color: red;">
|
| 81 |
+
<b>Easiest way:</b> Choose one from <b>Examples</b> below and then simply click <b>Generate</b>.
|
| 82 |
+
</p>
|
| 83 |
+
<br>
|
| 84 |
|
| 85 |
+
❗️❗️❗️Instruction Steps:<br>
|
| 86 |
+
1️⃣ Upload your <b>Input Image 🖼️ </b>.
|
| 87 |
+
Next, set <b>Resized Height for Input Image</b> and <b>Resized Width for Input Image</b> for the size you want.
|
| 88 |
+
<br>
|
| 89 |
+
2️⃣ Set <b>Top-Left Expand Height</b>, <b>Top-Left Expand Width</b>, <b>Bottom-Right Expand Height</b>, and <b>Bottom-Right Expand Width</b> for the expansion amount.
|
| 90 |
+
<br> The Canvas Height (Resized Height + Top-Left Expand Height + Bottom-Right Expand Height) and Canvas Width (Resized Width + Top-Left Expand Width + Bottom-Right Expand Width) should be the multiplier of 32.
|
| 91 |
+
<br> Recommend <b>Canvas Height = 704</b> and <b>Canvas Width = 1280</b> for the best performance (pre-trained model default resolution).
|
| 92 |
<br>
|
| 93 |
+
3️⃣ Click <b>Build the Canvas</b>.
|
| 94 |
+
4️⃣ Provide the motion trajectory of the object by clicking on the <b>Expanded Canvas 🖼️ </b>.
|
| 95 |
+
You can make additional trajectory for the same object by clicking <b>Add New Traj Line (Same Obj)</b>.
|
| 96 |
+
Reset by <b>Clear All Traj</b>.
|
| 97 |
+
<br>
|
| 98 |
+
5️⃣ Provide the <b>Identity Reference</b> image and its trajectory (optional).
|
| 99 |
+
Since image is segmented by SAM first (providng center point as query), it will be nice for the inputs to be center cropped.
|
| 100 |
+
<br> New instance trajectory can be done by clicking <b>Add New Instance (New Obj, including new ID)</b>.
|
| 101 |
+
<br>
|
| 102 |
+
6️⃣ Write a detailed <b>text prompt</b>.
|
| 103 |
+
<br>
|
| 104 |
+
7️⃣ Click the <b>Generate!</b> button to start the Video Generation.
|
| 105 |
<br>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
|
| 107 |
|
| 108 |
If **Frame In-N-Out** is helpful, please help star the [GitHub Repo](https://github.com/UVA-Computer-Vision-Lab/FrameINO?tab=readme-ov-file). Thanks!
|
|
|
|
| 816 |
with gr.Row():
|
| 817 |
|
| 818 |
# Button
|
| 819 |
+
generation_btn = gr.Button(value="Generate!")
|
| 820 |
|
| 821 |
|
| 822 |
with gr.Row():
|