We wanted to share some of the work we’ve been doing at Toolkit around enabling AI chat & workflows for videos. Here’s a quick Loom video of how easily you can upload an existing Loom video (just paste the link), meeting recording, or any video into Toolkit, in order to then be able to chat with your video, and also set up workflows that can run on top of your videos, and integrate with other tools you use like Docs (Notion / Google), Project Management (Linear etc.), CRMs, Applicant Tracking Systems and more.
https://www.loom.com/share/a76a66f748d9476b938494a7dab8d2bf
To summarize:
What’s important to know about the way the AI runs on your videos, is that the AI not only understands the spoken content of your videos, but also what’s actually happening on your screen - leveraging new models like GPT-4 with Vision, we’re able to give the AI a complete understanding of what’s happening in the frames of your video.
What we’re exploring in the long-term for Toolkit (toolkit.ai), is how to build a helpful co-pilot / AI teammate that understands the work you’re doing, and knows how to use the tools you use in your day to day work.
With OpenAI shipping GPT-V last November, AI can now understand the contents of what is on your screen, and in images more broadly. This trend is only going to continue, with open source models like Lllava (https://llava-vl.github.io/) and other providers, also aiming to ship more multi-modal capabilities.
Our assumption is that for a lot of work that we do in our day to day life, and in processes more broadly, teaching and communicating visually is often more effective than describing something through voice alone. As a designer, and product thinker, when I’m talking to customers and trying to understand their jobs to be done, and how they’re doing their work today, I almost always ask if they’re ok sharing they’re screen with me. Seeing how they work, and what their step-by-step processes are, is always illuminating. This sort of visual communication is much more powerful for capturing what processes actually consist of, and then being able to reason about how to actually help those folks improve their processes.
⚡️✨ If you’re thinking about how to automate more of you and your team’s day-to-day work, and would like an automation thought partner, drop me a note: [email protected]
We’ve done a variety of explorations around what sorts of products might be actually helpful to make AI more productive. We don’t believe that a chat-box that you have to remember to go to is necessarily the best UX for helping folks get the most out of AI, and that we’ll over time see more and more proactive AI.
One of the questions that any tool that helps you automate / assist with tasks in your day to day work has to navigate is discovering what the work you’re doing actually is, and being able to get into your flow to help out.
What could a helpful AI teammate look like? One thought we had was that an AI teammate could sort of be like a personal, or executive assistant - an AI that gets to know you, and helps you with tasks ad hoc.
Delegating tasks through screen recordings using tools like Loom is common practice, and if you want to get a sense of how teams focused on these practices automate their work and leverage human assistants, you should listen to this Tim Ferris interview with Sam Cocros, the founder of Levels and process / assistant super-user : https://tim.blog/2023/09/20/sam-corcos/. The Levels team works with Athena (athena.go).