Shun-Hsiang Hsu

Shun-Hsiang Hsu UIUC

Machine learning
research & engineering.

Multimodal AI · Computer vision · LLM evaluation

I develop and evaluate multimodal models, geometry-grounded synthetic data, and LLM agent workflows.

Resume (PDF) Selected Work

Selected Work

Examples of geometry-grounded synthetic concrete defects and their annotations

Beyond Cracks · Research

Geometry-grounded synthetic data

Controllable 3D defect geometry generates synthetic imagery, aligned masks, and physical attributes for visual recognition.

  • Synthetic data
  • 3D geometry
  • Semantic segmentation
Project
Overview of the VL-Con construction vision-language dataset

VisualSiteDiary & VL-Con

Vision-language learning

Vision-language models and domain-specific image–text data for captioning, retrieval, and construction reporting.

  • Vision-language models
  • Image retrieval
  • Few-shot prompting
VisualSiteDiary paper VL-Con paper Code & data

Google Cloud · Gemini Cloud Assist · Summer 2026

LLM evaluation & agents

Controlled autorater experiments, scoring rubrics, and agent skill-graph design for retrieval and orchestration.

  • LLM evaluation
  • Rubric design
  • Agent skill graphs

TryTech

Applied AI prototypes

Customer-support prototypes with retrieval, tool orchestration, and human handoff; separate visual-inspection pilots deployed on AWS.

  • Retrieval
  • Tool orchestration
  • Human handoff

Experience

  1. Summer 2026

    GoogleGoogle Cloud · Gemini Cloud Assist

    Software Engineering Intern (PhD)

    • LLM autorater experiments and rubrics: over 90% repeated-score consistency and over 90% alignment with human ratings.
    • Designed agent skill graphs for sequencing, retrieval, and orchestration.
    • Experimental design
    • LLM autorating

  2. Jan 2024 – May 2026

    Reconstruct Inc.

    Computer Vision Engineer Intern

    • Benchmarked image matching and 3D reconstruction under challenging visual conditions.
    • Adapted Swin segmentation with FiLM orientation cues; trained models through the existing active-learning pipeline.
    • 3D reconstruction
    • Swin Transformer
    • FiLM

  3. Jan 2021 – present

    TryTech Co., Ltd.

    Founder / Technical Lead

    • Built customer-support LLM prototypes with retrieval, tools, transactions, and human handoff.
    • Developed inspection applications and deployed AWS SAM/Lambda modules for industry pilots.
    • AWS SAM
    • AWS Lambda

  4. May 2020 – Jul 2022

    NCREE–NTUCE Joint AI Research Center

    Research Assistant

    • Led six industry research projects on deep-learning inspection of concrete and steel structures.
    • Integrated visual recognition into field inspection and near-real-time reporting.
    • Deep learning
    • Visual inspection

Selected Publications

All publications
  1. Research illustration for Self-Evaluation of Single and Multi-Agent LLMs as Assistants in Inspection Reporting Workflows

    Self-Evaluation of Single and Multi-Agent LLMs as Assistants in Inspection Reporting Workflows

    Shun-Hsiang Hsu, Yoonhwa Jung, Junryu Fu, Mani Golparvar-Fard

    Joint CSCE Construction Specialty & CRC Conference · 2025

    Equal contribution: Shun-Hsiang Hsu and Yoonhwa Jung

    Compares single- and multi-agent vision-language workflows for inspection reporting and studies model-generated self-assessment of report quality.

    Contribution: Co-first author; evaluated reporting workflows and self-assessment strategies.

    Paper (PDF)
  2. Research illustration for Beyond Cracks: Synthetic Image and Geometry Generation for Computer Vision Detection and Severity Assessment of Diverse Concrete Surface Defects

    Beyond Cracks: Synthetic Image and Geometry Generation for Computer Vision Detection and Severity Assessment of Diverse Concrete Surface Defects

    Shun-Hsiang Hsu, Mani Golparvar-Fard

    Automation in Construction · 2026

    Under review

    Uses controllable 3D defect geometry to generate aligned imagery and supervision for recognizing concrete defects and assessing their severity. Manuscript under review.

    Contribution: First author; developed the geometry-grounded synthetic-data framework.

    Project
  3. Research illustration for VisualSiteDiary: A detector-free Vision-Language Transformer model for captioning photologs for daily construction reporting and image retrievals

    VisualSiteDiary: A detector-free Vision-Language Transformer model for captioning photologs for daily construction reporting and image retrievals

    Yoonhwa Jung, Ikhyun Cho, Shun-Hsiang Hsu, Mani Golparvar-Fard

    Automation in Construction · 2024

    Equal contribution: Yoonhwa Jung, Ikhyun Cho, and Shun-Hsiang Hsu

    A detector-free vision-language Transformer generates construction photolog captions and supports image retrieval from visual observations.

    Contribution: Equal contributor with Yoonhwa Jung and Ikhyun Cho; co-developed VisualSiteDiary.

    Paper