Multimodal
Multimodal AI
What is multimodal AI?
- Processes text plus images, audio, and video together
- Use real work materials as they are
- Supported by most recent models
What multimodal AI really changes in practice is the 'prep work' that comes before AI. In the past, you had to type up scanned contracts, copy dashboard numbers into tables, and transcribe meeting recordings before you could use AI. Simply uploading materials as they are, without processing them, removes that manual prep work entirely. So the question in an adoption review shifts from 'Can AI do this task?' to 'Can AI read this material?'.
When evaluating, look at input and output multimodality separately. Business use is mostly on the input side: you feed in images, PDFs, or audio and get text back. The output side, which generates images or audio, is often a separate model with different pricing. It also plays a different role from OCR. OCR is strong at extracting characters accurately along with their coordinates, while multimodal AI is stronger at interpreting what a table means. If you have hundreds of fixed-format documents, extracting the text with OCR and feeding that in is better on both cost and accuracy. Images consume tokens based on resolution, so uploading high-resolution screenshots as they are can push your bill well above what you expect.
The most common slip in practice is believing the model 'sees like a person.' It reads chart trends well but estimates values between gridlines, and it misaligns rows and columns in tables with many merged cells. Handwritten 1s and 7s and dimension lines on drawings are also frequent trouble spots. The problem is that instead of saying it can’t tell, it fills in a plausible number, so if that goes straight into a report, no one notices. Add an instruction to the prompt such as 'leave it blank if you can’t read it,' and keep one review step that checks against the original figures. It is also common for client names or personal data to slip through when a whole screen capture is uploaded, so build the habit of masking them before upload.
John KimInstructor John Kim, author of "B2B Marketing Design" and CEO of Nanum Consulting
Sungil KimGlobal pitching expert from KAIST and Samsung Electronics who coached the winning teams at the UN Tourism and Select USA IR pitch competitionsThe concept of processing multiple modalities, such as text and images, together. It spread quickly after OpenAI’s CLIP and DALL·E in 2021.
Let’s look at a hypothetical case. Company A, an auto parts maker (260 employees, KRW 48 billion (≈ US$35,912,000) in annual revenue), still writes its shop-floor inspection logs by hand. Across 3 lines it produces 40 sheets a day, or nearly 900 handwritten inspection logs a month, and two members of the quality team spend 14 hours a week just copying them into Excel. Defect photos circulate separately in a group chat, so they are never linked to the logs.
- Manager Kim on the quality team scanned last month’s 30 inspection logs, uploaded them as-is, and asked, 'Pull out just the items, values, and notes into a table.' It read 27 sheets; on 3, ink smudges left the values blank.
- To stop it from inventing missing values, they added the instruction 'If something cannot be read, always mark it as [Needs check]' to the prompt. Unreadable items stayed visible in the table, which narrowed the review to 4 of 30 sheets.
- When they uploaded defect photos together with the log and had the AI link the related items, Manager Park on the production team said, 'The biggest win is that nobody has to type up photo descriptions by hand anymore.'
- On a drawing capture with overlapping dimension lines, the model misread 12.5mm as 1.25mm twice, so the team set a firm rule: drawings are never left to multimodal AI; existing CAD values are pasted in instead.
- Three months later, time spent entering inspection logs fell from 14 hours a week to 3, and with photos now attached to the logs, prep for defect root-cause meetings dropped from 2 hours to 1.