The　variety　of　multimedia　big　data　has　promoted　emerging　applications　of　multiple　sensorial　media　(mulsemedia)　types,　in　which　haptic　information　attracts　increasing　attentions.　Until　now,　the　interaction　between　haptic　signal　and　conventional　audio-visual　signals　have　not　been　fully　investigated.　In　this　work,　we　make　an　exploration　on　the　cross-modal　interactivity　in　task-driven　scenarios.　We　first　explore　the　correlation　between　visual　attention　and　haptic　control　in　three　designed　tasks:　random-trajectory,　fixed-trajectory　and　obstacle-avoidance.　Then,　we　propose　a　visual-haptic　interaction　model　that　estimates　kinesthetic　position　of　haptic　control　with　the　information　of　gaze　only.　By　incorporating　a　Long　Short-Term　Memory　(LSTM)　neural　network,　the　proposed　model　provides　effective　prediction　in　the　scenarios　of　fixed-trajectory　and　obstacle-avoidance,　with　its　performance　superior　to　other　selected　machine　learning-based　models.　To　further　examine　our　model,　we　execute　it　in　a　haptic　control　task　using　visual　guidance.　Implementation　results　show　a　high　task　achievement　rate.　©　2019　IEEE.