Learn, Generate, Rank: Generative Ranking of Motion Capture


We present a novel approach for searching and ranking videos for activities using generative models. This problem is a well established and is usually addressed using discriminative models, however, the decisions made by these models tend to be inexplainable. We believe that generative models are more explainable since they can generate instances of what they learned. We propose a novel model for generative searching and ranking using Generative Adversarial Networks (GANs). Our model learns human motion, generate realistic visual instances given textual inputs, and then uses the generated instances to search and rank videos in a database. We formulate the GAN discriminator using dense validation at each time-scale and perturb the discriminator input to make it translation invariant. Our model is capable of motion generation and completion. We show through our evaluations the resiliency to noise, generalization over actions, and generation of long diverse sequences. We evaluate our approach on Human 3.6M and CMU motion capture datasets using inception scores.

Our demo is available at: http://visxai.ckprototype.com/demo

Problem Statement

A large database of videos, extracted human poses, and a text query of a certain activity of interest.

Given

A text query of a certain activity.

Goal

1. Rank videos containing this activity putting the most likely ones first.
2. Highlight the video timeline with the location of the activity.
3. Explain the algorithm's reasoning behind this choice.

To illustrate our approach, we will use a running example of the query an individual was sweeping the floor.

00000.mp4
00001.mp4
00002.mp4
00003.mp4
00004.mp4
00005.mp4
00006.mp4
00007.mp4
00008.mp4
00009.mp4
00010.mp4
00011.mp4
00012.mp4
00013.mp4
00014.mp4
00015.mp4
00016.mp4
00017.mp4
00018.mp4
00019.mp4
00020.mp4
00021.mp4
00022.mp4
00023.mp4
00024.mp4
00025.mp4
00026.mp4
00027.mp4
00028.mp4
00029.mp4
00030.mp4
00031.mp4
00032.mp4
00033.mp4
00034.mp4
00035.mp4
00036.mp4
00037.mp4
00038.mp4
00039.mp4
00040.mp4
00041.mp4
00042.mp4
00043.mp4
00044.mp4
00045.mp4
00046.mp4
00047.mp4
00048.mp4
00049.mp4
00050.mp4
00051.mp4
00052.mp4
00053.mp4
00054.mp4
00055.mp4
00056.mp4
00057.mp4
00058.mp4
00059.mp4
00060.mp4
00061.mp4
00062.mp4
00063.mp4
00064.mp4
00065.mp4
00066.mp4
00067.mp4
00068.mp4
00069.mp4
00070.mp4
00071.mp4
00072.mp4
00073.mp4
00074.mp4
00075.mp4
00076.mp4
00077.mp4
00078.mp4
00079.mp4
00080.mp4
00081.mp4
00082.mp4
00083.mp4
00084.mp4
00085.mp4
00086.mp4
00087.mp4
00088.mp4
00089.mp4
00090.mp4
00091.mp4
00092.mp4
00093.mp4
00094.mp4
00095.mp4
00096.mp4
00097.mp4
00098.mp4
00099.mp4
00100.mp4
00101.mp4
00102.mp4
00103.mp4
00104.mp4
00105.mp4
00106.mp4
00107.mp4
00108.mp4
00109.mp4
00110.mp4
00111.mp4
00112.mp4
00113.mp4
00114.mp4
00115.mp4
00116.mp4
00117.mp4
00118.mp4
00119.mp4
00120.mp4
00121.mp4
00122.mp4
00123.mp4
00124.mp4
00125.mp4
00126.mp4
00127.mp4
00128.mp4
00129.mp4
00130.mp4
00131.mp4
00132.mp4
00133.mp4
00134.mp4
00135.mp4
00136.mp4
00137.mp4
00138.mp4
00139.mp4
00140.mp4
00141.mp4
00142.mp4
00143.mp4
00144.mp4
00145.mp4
00146.mp4
00147.mp4
00148.mp4
00149.mp4
00150.mp4
00151.mp4
00152.mp4
00153.mp4
00154.mp4
00155.mp4
00156.mp4
00157.mp4
00158.mp4
00159.mp4
00160.mp4
00161.mp4
00162.mp4
00163.mp4
00164.mp4
00165.mp4
00166.mp4
00167.mp4
00168.mp4
00169.mp4
00170.mp4
00171.mp4
00172.mp4
00173.mp4
00174.mp4
00175.mp4
00176.mp4
00177.mp4
00178.mp4
00179.mp4
00180.mp4
00181.mp4
00182.mp4
00183.mp4
00184.mp4
00185.mp4
00186.mp4
00187.mp4
00188.mp4
00189.mp4
00190.mp4
00191.mp4
00192.mp4
00193.mp4
00194.mp4
00195.mp4
00196.mp4
00197.mp4
00198.mp4
00199.mp4
00200.mp4
00201.mp4
00202.mp4
00203.mp4
00204.mp4
00205.mp4
00206.mp4
00207.mp4
00208.mp4
00209.mp4
00210.mp4
00211.mp4
00212.mp4
00213.mp4
00214.mp4
00215.mp4
00216.mp4
00217.mp4
00218.mp4
00219.mp4
00220.mp4
00221.mp4
00222.mp4
00223.mp4
00224.mp4
00225.mp4
00226.mp4
00227.mp4
00228.mp4
00229.mp4
00230.mp4
00231.mp4
00232.mp4
00233.mp4
00234.mp4
00235.mp4
00236.mp4
00237.mp4
00238.mp4
00239.mp4
00240.mp4
00241.mp4
00242.mp4
00243.mp4
00244.mp4
00245.mp4
00246.mp4
00247.mp4
00248.mp4
00249.mp4
00250.mp4
00251.mp4
00252.mp4
00253.mp4
00254.mp4
00255.mp4
00256.mp4
00257.mp4
00258.mp4
00259.mp4
00260.mp4
00261.mp4
00262.mp4
00263.mp4
00264.mp4
00265.mp4
00266.mp4
00267.mp4
00268.mp4
00269.mp4
00270.mp4
00271.mp4
00272.mp4
00273.mp4
00274.mp4
00275.mp4
00276.mp4
00277.mp4
00278.mp4
00279.mp4
00280.mp4
00281.mp4
00282.mp4
00283.mp4
00284.mp4
00285.mp4
00286.mp4
00287.mp4
00288.mp4
00289.mp4
00290.mp4
00291.mp4
00292.mp4
00293.mp4
00294.mp4
00295.mp4
00296.mp4
00297.mp4
00298.mp4
00299.mp4
00300.mp4
00301.mp4
00302.mp4
00303.mp4
00304.mp4
00305.mp4
00306.mp4
00307.mp4
00308.mp4
00309.mp4
00310.mp4
00311.mp4
00312.mp4
00313.mp4
00314.mp4
00315.mp4
00316.mp4
00317.mp4
00318.mp4
00319.mp4
00320.mp4
00321.mp4
00322.mp4
00323.mp4
00324.mp4
00325.mp4
00326.mp4
00327.mp4
00328.mp4
00329.mp4
00330.mp4
00331.mp4
00332.mp4
00333.mp4
00334.mp4
00335.mp4
00336.mp4
00337.mp4
00338.mp4
00339.mp4
00340.mp4
00341.mp4
00342.mp4
00343.mp4
00344.mp4
00345.mp4
00346.mp4
00347.mp4
00348.mp4
00349.mp4
00350.mp4
00351.mp4
00352.mp4
00353.mp4
00354.mp4
00355.mp4
00356.mp4
00357.mp4
00358.mp4
00359.mp4
00360.mp4
00361.mp4
00362.mp4
00363.mp4
00364.mp4
00365.mp4
00366.mp4
00367.mp4
00368.mp4
00369.mp4
00370.mp4
00371.mp4
00372.mp4
00373.mp4
00374.mp4
00375.mp4
00376.mp4
00377.mp4
00378.mp4
00379.mp4
00380.mp4
00381.mp4
00382.mp4
00383.mp4
00384.mp4
00385.mp4
00386.mp4
00387.mp4
00388.mp4
00389.mp4
00390.mp4
00391.mp4
00392.mp4
00393.mp4
00394.mp4
00395.mp4
00396.mp4
00397.mp4
00398.mp4
00399.mp4
00400.mp4
00401.mp4
00402.mp4
00403.mp4
00404.mp4
00405.mp4
00406.mp4
00407.mp4
00408.mp4
00409.mp4
00410.mp4
00411.mp4
00412.mp4
00413.mp4
00414.mp4
00415.mp4
00416.mp4
00417.mp4
00418.mp4
00419.mp4
00420.mp4
00421.mp4
00422.mp4
00423.mp4
00424.mp4
00425.mp4
00426.mp4
00427.mp4
00428.mp4
00429.mp4
00430.mp4
00431.mp4
00432.mp4
00433.mp4
00434.mp4
00435.mp4
00436.mp4
00437.mp4
00438.mp4
00439.mp4
00440.mp4
00441.mp4
00442.mp4
00443.mp4
00444.mp4
00445.mp4
00446.mp4
00447.mp4
00448.mp4
00449.mp4
00450.mp4
00451.mp4
00452.mp4
00453.mp4
00454.mp4
00455.mp4
00456.mp4
00457.mp4
00458.mp4
00459.mp4
00460.mp4
00461.mp4
00462.mp4
00463.mp4
00464.mp4
00465.mp4
00466.mp4
00467.mp4
00468.mp4
00469.mp4
00470.mp4
00471.mp4
00472.mp4
00473.mp4
00474.mp4
00475.mp4
00476.mp4
00477.mp4
00478.mp4
00479.mp4
00480.mp4
00481.mp4
00482.mp4
00483.mp4
00484.mp4
00485.mp4
00486.mp4
00487.mp4
00488.mp4
00489.mp4
00490.mp4
00491.mp4
00492.mp4
00493.mp4
00494.mp4
00495.mp4
00496.mp4
00497.mp4
00498.mp4
00499.mp4
00500.mp4
00501.mp4
00502.mp4
00503.mp4
00504.mp4
00505.mp4
00506.mp4
00507.mp4
00508.mp4
00509.mp4
00510.mp4
00511.mp4
00512.mp4
00513.mp4
00514.mp4
00515.mp4
00516.mp4
00517.mp4
00518.mp4
00519.mp4
00520.mp4
00521.mp4
00522.mp4
00523.mp4
00524.mp4
00525.mp4
00526.mp4
00527.mp4
00528.mp4
00529.mp4
00530.mp4
00531.mp4
00532.mp4
00533.mp4
00534.mp4
00535.mp4
00536.mp4
00537.mp4
00538.mp4
00539.mp4
00540.mp4
00541.mp4
00542.mp4
00543.mp4
00544.mp4
00545.mp4
00546.mp4
00547.mp4
00548.mp4
00549.mp4
00550.mp4
00551.mp4
00552.mp4
00553.mp4
00554.mp4
00555.mp4
00556.mp4
00557.mp4
00558.mp4
00559.mp4
00560.mp4
00561.mp4
00562.mp4
00563.mp4
00564.mp4
00565.mp4
00566.mp4
00567.mp4
00568.mp4
00569.mp4
00570.mp4
00571.mp4
00572.mp4
00573.mp4
00574.mp4
00575.mp4
00576.mp4
00577.mp4
00578.mp4
00579.mp4
00580.mp4
00581.mp4
00582.mp4
00583.mp4
00584.mp4
00585.mp4
00586.mp4
00587.mp4
00588.mp4
00589.mp4
00590.mp4
00591.mp4
00592.mp4
00593.mp4
00594.mp4
00595.mp4
00596.mp4
00597.mp4
00598.mp4
00599.mp4
00600.mp4
00601.mp4
00602.mp4
00603.mp4
00604.mp4
00605.mp4
00606.mp4
00607.mp4
00608.mp4
00609.mp4
00610.mp4
00611.mp4
00612.mp4
00613.mp4
00614.mp4
00615.mp4
00616.mp4
00617.mp4
00618.mp4
00619.mp4
00620.mp4
00621.mp4
00622.mp4
00623.mp4
00624.mp4
00625.mp4
00626.mp4
00627.mp4
00628.mp4
00629.mp4
00630.mp4
00631.mp4
00632.mp4
00633.mp4
00634.mp4
00635.mp4
00636.mp4
00637.mp4
00638.mp4
00639.mp4
00640.mp4
00641.mp4
00642.mp4
00643.mp4
00644.mp4
00645.mp4
00646.mp4
00647.mp4
00648.mp4
00649.mp4
00650.mp4
00651.mp4
00652.mp4
00653.mp4
00654.mp4
00655.mp4
00656.mp4
00657.mp4
00658.mp4
00659.mp4
00660.mp4
00661.mp4
00662.mp4
00663.mp4
00664.mp4
00665.mp4
00666.mp4
00667.mp4
00668.mp4
00669.mp4
00670.mp4
00671.mp4
00672.mp4
00673.mp4
00674.mp4
00675.mp4
00676.mp4
00677.mp4
00678.mp4
00679.mp4
00680.mp4
00681.mp4
00682.mp4
00683.mp4
00684.mp4
00685.mp4
00686.mp4
00687.mp4
00688.mp4
00689.mp4
00690.mp4
00691.mp4
00692.mp4
00693.mp4
00694.mp4
00695.mp4
00696.mp4
00697.mp4
00698.mp4
00699.mp4
00700.mp4
00701.mp4
00702.mp4
00703.mp4
00704.mp4
00705.mp4
00706.mp4
00707.mp4
00708.mp4
00709.mp4
00710.mp4
00711.mp4
00712.mp4
00713.mp4
00714.mp4
00715.mp4
00716.mp4
00717.mp4
00718.mp4
00719.mp4
00720.mp4
00721.mp4
00722.mp4
00723.mp4
00724.mp4
00725.mp4
00726.mp4
00727.mp4
00728.mp4
00729.mp4
00730.mp4
00731.mp4
00732.mp4
00733.mp4
00734.mp4
00735.mp4
00736.mp4
00737.mp4
00738.mp4
00739.mp4
00740.mp4
00741.mp4
00742.mp4
00743.mp4
00744.mp4
00745.mp4
00746.mp4
00747.mp4
00748.mp4
00749.mp4
00750.mp4
00751.mp4
00752.mp4
00753.mp4
00754.mp4
00755.mp4
00756.mp4
00757.mp4
00758.mp4
00759.mp4
00760.mp4
00761.mp4
00762.mp4
00763.mp4
00764.mp4
00765.mp4
00766.mp4
00767.mp4
00768.mp4
00769.mp4
00770.mp4
00771.mp4
00772.mp4
00773.mp4
00774.mp4
00775.mp4
00776.mp4
00777.mp4
00778.mp4
00779.mp4
00780.mp4
00781.mp4
00782.mp4
00783.mp4
00784.mp4
00785.mp4
00786.mp4
00787.mp4
00788.mp4
00789.mp4
00790.mp4
00791.mp4
00792.mp4
00793.mp4
00794.mp4
00795.mp4
00796.mp4
00797.mp4
00798.mp4
00799.mp4
00800.mp4
00801.mp4
00802.mp4
00803.mp4
00804.mp4
00805.mp4
00806.mp4
00807.mp4
00808.mp4
00809.mp4
00810.mp4
00811.mp4
00812.mp4
00813.mp4
00814.mp4
00815.mp4
00816.mp4
00817.mp4
00818.mp4
00819.mp4
00820.mp4
00821.mp4
00822.mp4
00823.mp4
00824.mp4
00825.mp4
00826.mp4
00827.mp4
00828.mp4
00829.mp4
00830.mp4
00831.mp4
00832.mp4
00833.mp4
00834.mp4
00835.mp4
00836.mp4
00837.mp4
00838.mp4
00839.mp4
00840.mp4
00841.mp4
00842.mp4
00843.mp4
00844.mp4
00845.mp4
00846.mp4
00847.mp4

Ranking - Success

We present and contrast two approaches to address this problem and contrast one another: discriminative ranking and generative ranking.

We can start searching through the clips by typing in the query of an action performed by the individual. Note that can also choose from an autocomplete dropdown list of all the actions the algorithm is trained on.

Discriminative Ranking

Not Explainable: The model is trained to rank a list of videos by extracting features from them and relate to the text query.

Generative Ranking

Explainable: The model is trained to generate video instances of what it thinks the text query should look like. Then it uses these instances to rank the videos.

Rank(Text, Video)

40. sweep floor (126 blocks)

705. sweep floor (127 blocks)

244. mop floor (143 blocks)

773. mickey sneaking walk (39 blocks)

99. mickey sneaking walk (40 blocks)

812. climb ladder (44 blocks)

788. laugh (72 blocks)

837. mickey sneaking walk (41 blocks)

542. laugh (79 blocks)

702. sweeping (34 blocks)

348. cleaning up (116 blocks)

669. mickey sneaky walk (38 blocks)

558. direct traffic wave point (116 blocks)

847. climb ladder (51 blocks)

466. moving a stool (40 blocks)

390. cold (37 blocks)

667. unscrew bottlecap drink soda screw on bottlecap (51 blocks)

132. no movie (40 blocks)

274. pulling a nail (36 blocks)

361. wash windows (133 blocks)

362. bolt loosening wrench (42 blocks)

144. salsa dance (66 blocks)

231. wash windows (156 blocks)

28. tiger (35 blocks)

108. old man pick up (106 blocks)

726. unscrew bottlecap drink soda (46 blocks)

278. direct traffic wave point (139 blocks)

621. salsa dance (71 blocks)

805. ring around the rosie (34 blocks)

591. jumping jacks jog squats side twists stretches (168 blocks)

665. creeping with limp (94 blocks)

357. jumping jacks side twists reach up bend over (146 blocks)

592. drink soda (35 blocks)

328. violin (53 blocks)

33. get up from floor (48 blocks)

631. chicken pick up (51 blocks)

417. creeping walk (60 blocks)

570. alien turn (39 blocks)

448. throw (35 blocks)

426. gorilla (45 blocks)

447. normal coffee mug (50 blocks)

295. salsa dance (59 blocks)

367. yoga (116 blocks)

73. playground climb sit dangle legs rock back lower self to ground (82 blocks)

186. salsa dance (53 blocks)

10. walk up to object squat pick up object set down in another place (33 blocks)

622. creeping walk (66 blocks)

593. old man walk (96 blocks)

69. friends meet hang out a sits b joins a 2 subjects subject b (40 blocks)

120. looking around (35 blocks)

6. looking around (35 blocks)

7. typing on a laptop (33 blocks)

580. punch and kick (42 blocks)

294. playground climb sit dangle legs descend (161 blocks)

443. climb 3 steps (39 blocks)

577. chopping wood (34 blocks)

22. walk and carry (46 blocks)

414. monkey human subject (123 blocks)

47. sit on stepstool stand up (83 blocks)

389. bolt tightening wrench (33 blocks)

258. putting on a coat (39 blocks)

763. climb 3 steps (39 blocks)

66. drag heavy object (44 blocks)

754. normal pick up (37 blocks)

770. creep and pause repeat (36 blocks)

377. salsa dance (81 blocks)

454. sit on stepstool hands against chin fidget stand up (138 blocks)

309. swordplay (33 blocks)

555. playground climb (136 blocks)

391. walk up to object squat pick up object set down in another place (42 blocks)

299. snake (74 blocks)

291. start walk left (56 blocks)

125. old man wait (157 blocks)

259. (empty) (58 blocks)

792. lean forward reach for (113 blocks)

534. get up from floor (62 blocks)

383. driving (40 blocks)

471. start duck underneath stop (40 blocks)

836. oiduki (68 blocks)

603. punch_sequence (71 blocks)

292. scared (139 blocks)

608. monkey human subject (89 blocks)

445. shy (141 blocks)

320. dance fold in from side arabesque curl inwards (36 blocks)

12. walk backwards on toes crouched (40 blocks)

761. climb step over sit on jump over stepstool