Template and rule pipeline
- Capture a scan or export
- OCR extracts raw text
- Match rules written per layout
- Exceptions go to people
A new layout means a new template, and every unknown case becomes manual work.
Session · Vision-language models
VLMs for automation workflows.
Most automation stops where the data stops being clean: scanned documents, screenshots, charts, photos. Vision-language models read those the way a person does. This session shows what that changes.
At a glance
The session in brief
- Format
- Talk with live demos and Q&A
- Length
- About 60 minutes, adjustable
- Audience
- Engineers, operations and product teams
- Prerequisites
- None. No machine-learning background needed.
Fundamentals
What is a VLM?
A vision-language model (VLM) is an AI model that takes images together with text and answers in text. Give it a photo, a scanned page or a screenshot along with an instruction, and it can describe, extract, compare or decide.
-
In
An image and an instruction
A PDF page, a dashboard screenshot, a product photo, plus a plain-language task such as “list the line items.”
-
Model
Layout and language, read together
The model interprets where things are and what they say at once, instead of running separate OCR and rules.
-
Out
Text, data or an action
A summary, structured JSON that matches your schema, or the next step for an agent operating an interface.
The change
How a VLM improves your workflow
Traditional pipelines need a template or rule for every layout. When a supplier changes an invoice format, the pipeline breaks and the work lands on a person. A VLM workflow reads the page instead of matching it.
VLM workflow
- Capture an image, PDF or screenshot
- VLM reads layout and text with your instructions
- Validate against a schema and business checks
- Review only low-confidence or failing cases
New formats often work without new templates, and people focus on the cases that need judgment.
-
Handles variety
One instruction covers many layouts, languages and image qualities, instead of one template each.
-
Fewer hand-offs
Reading, understanding and deciding happen in one step, so there are fewer stages to build and maintain.
-
Automates what has no API
If a person can see it on screen, a VLM can often read it. That opens up legacy tools, portals and paper.
-
Faster to change
Adapting a workflow means updating instructions and examples, not rebuilding parsing logic.
Use cases
Where teams put VLMs to work
-
01
Document processing
Invoices, receipts, forms and contracts turned into structured records for downstream systems.
-
02
Screen and UI automation
Agents that read a screenshot and operate software with no API, such as legacy tools and web portals.
-
03
Visual quality inspection
Checking photos for defects, missing labels or wrong packaging, with a written reason for each flag.
-
04
Charts and reports
Pulling numbers and trends out of charts, slides and dashboards that were never available as data.
-
05
Catalog and content tagging
Consistent attributes, descriptions and moderation labels for large image collections.
-
06
Visual monitoring
Watching screens, camera stills or dashboards and raising a plain-language alert when something looks wrong.
Doing it responsibly
Reliable, not just impressive
VLMs are not always right. They can misread small or dense text, miscount, or state something with false confidence. The session spends real time on the practices that make a VLM workflow safe to depend on.
This is where our work on evaluation and monitoring applies directly.
- Structured output. Force answers into a schema and reject anything that does not validate.
- Business-rule checks. Totals add up, dates are plausible, required fields exist.
- Human review by exception. Route low-confidence and failing cases to a person.
- A test set of real samples. Measure accuracy before launch and after every change.
- Ongoing monitoring. Watch for drift as documents and interfaces change.
- Right-sized models. Use smaller, cheaper models for easy cases and larger ones for hard cases.
Agenda
What we will cover
-
VLM fundamentals
What VLMs can and cannot do today, in plain language.
-
From OCR pipelines to VLM workflows
Where the old approach breaks and what replaces it.
-
Live demo: document to structured data
An image goes in; validated JSON comes out, with checks and a review queue.
-
Making it reliable
Evaluation sets, validation, monitoring and cost control.
-
Choosing your first workflow
A simple way to find a high-value, low-risk process to start with.
-
Q&A
Bring a real process from your team and we will talk it through.
Takeaways
What you will leave with
A clear mental model
An accurate picture of what VLMs do well, where they fail and why.
A way to pick workflows
Criteria for choosing which processes to automate first and which to leave alone.
A reliability checklist
The validation, evaluation and monitoring steps to build in from the start.
Request this session
Bring it to your team.
Tell us about your team and the workflows you would like to automate. We will tailor the examples and length to fit.
hello@femtosol.com