Live Video Consultation

Product decision

The primary experience is a continuous live Urdu video call with an AI agricultural consultant. The farmer sees and hears an ongoing consultation while the system maintains durable case state across speech, video evidence, tool calls, research and policy verification.

Session modes

Media topology

Flutter camera + microphone
        ↓ WebRTC
LiveKit room / SFU / TURN
        ├── continuous audio track → streaming STT
        ├── continuous video track → adaptive sampler
        ├── network telemetry → quality controller
        └── officer participant → optional live join

Adaptive sampler
        ├── periodic low-rate samples
        ├── stable-frame detector
        ├── motion / scene-change trigger
        ├── model-requested diagnostic burst
        └── explicit full-resolution still capture
                ↓
        evidence store + reasoning model

Adaptive frame sampling

Sampling is policy-driven rather than fixed. A starting configuration may inspect a low-rate stream during normal conversation, increase sampling when the farmer points to a symptom, and request a high-resolution still when fine texture or an insect must be examined.

• Do not infer from frames that fail focus, exposure, crop-presence or subject-size gates.
• Deduplicate near-identical frames and retain only evidence-bearing selections.
• Label each selected frame with call time, camera state, requested view and quality result.
• Preserve the link between model observations and the exact frame IDs used.
• Never equate “many frames” with stronger evidence.

Camera guidance loop

1. Model states the next visual objective.
2. Realtime agent converts it into concise Urdu camera guidance.
3. On-device/server quality checks observe whether the view is usable.
4. The system acknowledges success or gives one corrective instruction.
5. A selected frame or still becomes an evidence item.
6. The reasoning agent updates hypotheses and chooses the next action.

Typical instructions include: show the whole plant, move closer, hold still, turn the leaf over, show an unaffected plant, walk to the field edge, or show standing water and drainage.

Reasoning context during the call

The model receives an evolving case file: finalized transcript turns, normalized entities, selected frame observations, farmer answers, weather/soil/topography, field history, retrieved local evidence, external research and exact-policy results. Partial transcripts and raw frame streams remain outside the durable reasoning loop.

Officer join and handoff

• The officer receives a concise pre-join brief: issue, risk, leading hypotheses, missing evidence, policy status and requested role.
• Role-based controls determine who may see precise location, prior treatments and retained media.
• The farmer is told when a human joins and whether the session is being recorded.
• The transcript and evidence timeline continue in the same case.
• The officer may approve, amend, request evidence, end AI speech or complete a warm handoff.

Network degradation ladder

Full live video
  → lower bitrate / resolution / frame rate
  → audio-first call with periodic stills
  → push-to-talk with compressed images
  → recorded case queued for later processing

Degradation must preserve the farmer’s case and explain the active mode. The application must not silently continue as though the model can still see live video.

Recording and retention

Live transport does not imply recording. Default pilot policy should process the call in real time, retain transcript and explicitly selected evidence frames, and avoid retaining the full raw video unless a separately approved purpose, consent and retention schedule require it.

• Show recording state continuously.
• Separate consent for service delivery, human review and evaluation use.
• Store selected frames with purpose and retention class.
• Support immediate recording disable and source deletion workflows.

Operational and acceptance metrics

• Call setup success and time to media.
• Median/P95 packet loss, reconnects and degradation transitions.
• Useful-frame acquisition rate and number of corrective prompts.
• Time from farmer indication to accepted diagnostic evidence.
• Farmer comprehension and live-call completion.
• Officer join time and handoff success.
• Model cost per selected frame and media cost per completed advisory.
• Privacy incidents, unintended recording and frame-provenance completeness.