When integrating image understanding capabilities with OpenAI Vision models via Azure OpenAI, developers often encounter a perplexing issue: bounding box coordinates returned by the model are consistently incorrect when analyzing high-resolution images. These errors are systematic and reproducible, not random hallucinations. In essence, the AI is performing computations on a version of the image it never actually received at full resolution.

This discrepancy is more than just an inconvenience—it can break workflows that require precise spatial understanding, such as document analysis, automated drawing interpretation, and UI element detection. The issue extends beyond OpenAI’s models to other multimodal AI offerings like Google Gemini, Anthropic Claude, and Qwen3-VL, although each handles image scaling differently.

In this post, we will unravel the root cause of these coordinate mismatches, describe how to fix the problem using C#, compare how various models handle resolution, and highlight the evolution in newer GPT versions that better preserve image fidelity. If you rely on bounding box precision in your AI apps, understanding these nuances is critical to getting reliable results.

Architecture Overview

┌─────────────────────────────────────────────┐
│            Input High-Resolution Image       │
│  (e.g., 4960 x 3507 pixels)                  │
├─────────────────────────────────────────────┤
│  Preprocessing Pipeline on OpenAI Servers   │
│  • Scale to fit within 2048x2048 bounding box│
│  • Further scale shortest side to 768 pixels│
│  • Divide into 512x512 tiles (tiled vision) │
└─────────────────────────────────────────────┘
               ↓
┌─────────────────────────────────────────────┐
│       Multimodal LLM Model Processing        │
│  • GPT-4, GPT-5.x, Claude, Gemini, Qwen3-VL │
│  • Model sees only downscaled tiled image    │
└─────────────────────────────────────────────┘
               ↓
┌─────────────────────────────────────────────┐
│       Returned Coordinates & Annotations     │
│  • Coordinates refer to downscaled image     │
│  • Require rescaling to original resolution  │
└─────────────────────────────────────────────┘

This flow illustrates the critical insight: the model’s “view” of the image is a significantly downscaled version optimized for token and computational efficiency. The coordinates provided correspond to this reduced coordinate space.

Figure: How OpenAI Vision’s preprocessing pipeline causes coordinate misalignments
Image courtesy of AzureTechInsider by Dieter Gobeyn

Key Technical Observations

• Systematic Downscaling Pipeline — OpenAI applies a fixed three-stage downscaling process before feeding images to models: fit within a 2048×2048 box while preserving aspect ratio, rescale the shortest side to 768 pixels, then tile into 512×512 pixel patches, causing a mismatch between original and perceived coordinates.

• ChatImageDetailLevel Parameter Controls Preprocessing — This SDK parameter (with values Low, High, and Auto) determines if and how much the tiled pipeline is applied. “High” triggers the full tile rescale. Importantly, “Auto” is not recommended when coordinate fidelity matters as it leaves scaling decisions to opaque internal defaults.

• Pre-Scaling Fix via C# Image Resize — By explicitly resizing your input image client-side to the effective resolution the model will perceive (calculated from the original image’s aspect ratio and scaling rules), you synchronize the coordinate spaces. This requires scale factors to correctly convert returned bounding box coordinates back to the original resolution.

• GPT-5.4 Introduced an Original Detail Level — This new mode supports images up to 10.24 million pixels, significantly reducing downscaling and improving coordinate accuracy. GPT-5.5 defaults to this original mode when detail level is unset or set to Auto, enhancing out-of-the-box precision.

• Model Differences in Effective Resolution — Other vendors and models handle image scaling differently: Anthropic Claude’s effective max resolution is ~2659×1880, Google Gemini around 3072×2172, while Qwen3-VL allows fully configurable resolution without fixed pipelines, enabling superior coordinate precision at high VRAM cost.

• Documentation Spread Across Multiple Platforms — Official pixel limits and token cost formulas are scattered across provider docs (OpenAI, Anthropic, Google Vertex AI, Hugging Face), requiring developers to update hardcoded scaling factors vigilantly as models evolve.

How It Works: Correcting Coordinate Errors with Pre-Scaling

Understanding the Effective Image Resolution

When sending an HD image (for example, 4960×3507 pixels) to OpenAI Vision models with ChatImageDetailLevel.High, the preprocessing pipeline does:

1. Fit inside 2048×2048 box (preserving aspect ratio):
   For 4960×3507, it scales to about 2048×1448.

2. Scale shortest side to 768 pixels:
   Further scaling down to approximately 1086×768.

The model effectively “sees” a 1086×768 pixel image, not the original high-res.

Pre-scale Your Image Before Calling the API

To align coordinate systems, resize the image locally to the effective resolution:

// Calculate scale factors based on original dimensions and effective resolution
const int EffectiveWidth  = 1086;
const int EffectiveHeight = 768;
const double ScaleX = 4960.0 / EffectiveWidth;   // ~4.57
const double ScaleY = 3507.0 / EffectiveHeight;  // ~4.57

// Load and resize the original image
using var originalImage = Image.Load(imageBytes);
originalImage.Mutate(x => x.Resize(EffectiveWidth, EffectiveHeight));

using var ms = new MemoryStream();
await originalImage.SaveAsPngAsync(ms);
var scaledBytes = ms.ToArray();

// Send the scaled image to the API noting the true dimensions in the prompt
var prompt = $"The image is {EffectiveWidth}x{EffectiveHeight}. " +
             "Return bounding box coordinates as [x, y, width, height].";

var imagePart = ChatMessageContentPart.CreateImagePart(
    BinaryData.FromBytes(scaledBytes),
    "image/png",
    ChatImageDetailLevel.High
);

// After receiving bounding box coordinates (modelX, modelY), rescale to original
var fullResX = (int)(modelX * ScaleX);
var fullResY = (int)(modelY * ScaleY);

Why This Matters

By pre-scaling:

• The model processes exactly the input image it “thinks” it receives.
• Coordinates returned match the downscaled image’s coordinate space.
• You can accurately map coordinates back to the full resolution domain.

This eliminates the offset and scaling errors seen in naive usage.

Dynamically Calculate Effective Resolution

Always compute effective width and height programmatically depending on your input image’s dimensions:

• Fit in 2048×2048 bounding box
• Scale shortest side to 768 pixels

This prevents brittle hardcoded values and adapts to varying image sizes.

Quick Tips & Tricks

1. Explicitly Set ChatImageDetailLevel to High or Original — Avoid “Auto” for workflows needing precise coordinates; Auto behavior can change across deployments and is undocumented.

2. Monitor SDK Updates for GPT-5.5 ‘Original’ Detail Level — The .NET SDK may lag behind the model capabilities. You can pass "original" via raw options before official enum support lands.

3. Use Qwen3-VL for Fully Configurable Resolution — If self-hosting is an option for you, Qwen3-VL's dynamic resolution means better coordinate fidelity at the cost of managing your own inference infrastructure.

4. Refer to Provider Docs for Pixel Limits and Token Costs — Hardcoded values get stale fast; always tie scaling calculations to official, versioned documentation.

5. Leverage Image Libraries for Efficient Resizing — Use performant native libraries like SixLabors.ImageSharp in .NET to handle resizing without quality loss.

6. Adjust Your Workflow for Model Updates — GPT-5.4 and 5.5 significantly changed default behavior; test after upgrades to leverage new detail levels and improved coordinate fidelity.

Conclusion

Precision in coordinate-based outputs from multimodal AI vision models hinges on understanding the image preprocessing pipeline that occurs behind the scenes. OpenAI Vision’s legacy tiled pipeline downscales high-resolution images significantly, causing bounding box coordinates to mismatch original dimensions. By applying a simple pre-scaling step in your C# client—resizing the input image to the effective resolution the model processes—you align coordinate spaces and restore spatial accuracy.

Newer GPT versions like 5.4 and 5.5 have improved this greatly by supporting near-original resolution input, but until SDKs fully support these modes, explicit control remains paramount. Meanwhile, other models like Anthropic Claude, Google Gemini, and Qwen3-VL offer alternative pipelines that might serve specific needs better, especially for tasks requiring detailed spatial reasoning.

As the AI vision ecosystem evolves, staying close to the latest official documentation and adapting your scaling logic accordingly will ensure your applications continue to deliver robust, precise results.

References

1. Why OpenAI Vision gets your coordinates wrong & how to fix it! - AzureTechInsider — Original detailed exploration and fix steps by Dieter Gobeyn
2. OpenAI Vision guide — Official documentation on image processing, token costs, and tiling pipeline
3. GPT-5.4 update announcement — Details on original detail level introduction improving resolution handling
4. Anthropic Vision documentation — Pixel limits and token cost formulas for Claude models
5. Gemini API vision documentation — Google’s approach to image resolution and cost in Gemini models
6. Qwen3-VL GitHub repository — Technical details on dynamic resolution and configuration for Qwen3-VL vision models
7. ChatMessageContentPart.CreateImagePart() - Azure OpenAI SDK docs — How to pass images with detail level in C# SDK

---