infer_nanonets_ocr2

Nanonets-OCR2 by Nanonets is a family of powerful, state-of-the-art image-to-markdown OCR models that go far beyond traditional text extraction. It transforms documents into structured markdown with intelligent content recognition and semantic tagging, making it ideal for downstream processing by Large Language Models (LLMs).

Nanonets-OCR2 is packed with features designed to handle complex documents with ease:

• LaTeX Equation Recognition: Automatically converts mathematical equations and formulas into properly formatted LaTeX syntax.
• Intelligent Image Description: Describes images within documents using structured tags, making them digestible for LLM processing. It can describe various image types, including logos, charts, graphs and so on, detailing their content, style, and context.
• Signature Detection & Isolation: Identifies and isolates signatures from other text, outputting them within a tag. This is crucial for processing legal and business documents.
• Watermark Extraction: Detects and extracts watermark text from documents, placing it within a tag.
• Smart Checkbox Handling: Converts form checkboxes and radio buttons into standardized Unicode symbols (☐, ☑, ☒) for consistent and reliable processing.
• `Complex Table Extraction: Accurately extracts complex tables from documents and converts them into both markdown and HTML table formats.
• Flow charts & Organisational charts: Extracts flow charts and organisational as mermaid code.
• Handwritten Documents: The model is trained on handwritten documents across multiple languages.
• Multilingual: Model is trained on documents of multiple languages, including English, Chinese, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Arabic, and many more. Visual Question Answering (VQA): The model is designed to provide the answer directly if it is present in the document; otherwise, it responds with "Not mentioned."

🚀 Use with Ikomia API

1. Install Ikomia API

We strongly recommend using a virtual environment. If you're not sure where to start, we offer a tutorial here.

pip install ikomia

2. Create your workflow

from ikomia.dataprocess.workflow import Workflow

# Init your workflow
wf = Workflow()

# Add algorithm
algo = wf.add_task(name="infer_nanonets_ocr2", auto_connect=True)

# Run on your image  
wf.run_on(url="https://github.com/NanoNets/Nanonets-OCR2/blob/main/assets/bank_statement.jpg?raw=true")

# Save output .json
nanonets = algo.get_output(1)
nanonets.save('nanonets_output.json')

☀️ Use with Ikomia Studio

Ikomia Studio offers a friendly UI with the same features as the API.

• If you haven't started using Ikomia Studio yet, download and install it from this page.
• For additional guidance on getting started with Ikomia Studio, check out this blog post.

📝 Set algorithm parameters

from ikomia.dataprocess.workflow import Workflow

# Init your workflow
wf = Workflow()

# Add algorithm
algo = wf.add_task(name="infer_nanonets_ocr2", auto_connect=True)

algo.set_parameters({
    "model_name": "nanonets/Nanonets-OCR2-3B",
    "cuda": "True",
    "prompt": "Describe the image in detail.",
    "max_new_tokens": "4096", 
    "input_size": "1024"
})

# Run on your image  
wf.run_on(url='https://github.com/Ikomia-dev/notebooks/blob/main/examples/img/img_text_inspiration.jpg?raw=true')

# Save output .json
qwen_output = algo.get_output(1)
qwen_output.save('nanonets_output.json')

🔍 Explore algorithm outputs

Every algorithm produces specific outputs, yet they can be explored them the same way using the Ikomia API. For a more in-depth understanding of managing algorithm outputs, please refer to the documentation.

from ikomia.dataprocess.workflow import Workflow

# Init your workflow
wf = Workflow()

# Add algorithm
algo = wf.add_task(name="infer_nanonets_ocr2", auto_connect=True)

# Run on your image  
wf.run_on(url='https://github.com/Ikomia-dev/notebooks/blob/main/examples/img/img_text_inspiration.jpg?raw=true')

# Iterate over outputs
for output in algo.get_outputs():
    # Print information
    print(output)
    # Export it to JSON
    output.to_json()