Shun-Hsiang Hsu / Resume

Research & experience.

Machine learning research in multimodal learning, computer vision, geometry-grounded synthetic data, and LLM evaluation. Experience in controlled experiments, benchmarking, model adaptation, and research prototypes.

UIUC Ph.D. student · Expected 2027 · Champaign, Illinois

Resume (PDF) ↗

Professional experience

Google Google Cloud · Gemini Cloud Assist

Software Engineering Intern (PhD)

2026 – 2026

Summer 2026 · LLM evaluation and skill-based agent systems.

  • Designed controlled experiments and refined scoring rubrics for an LLM autorater, achieving over 90% repeated-evaluation score consistency and over 90% alignment with human ratings.
  • Designed a graph representation of agent skills to encode sequencing and dependencies for relationship-aware retrieval, orchestration, and skill-library health checks.

Reconstruct Inc.

Computer Vision Engineer Intern

2024-01 – 2026-05

Computer vision and 3D method evaluation, model adaptation, and training iteration.

  • Built controlled image-matching and 3D reconstruction benchmarks under low-texture, repeated-pattern, sparse-view, and illumination-shift conditions to guide production method selection.
  • Integrated FiLM-conditioned camera-orientation cues into a Swin Transformer segmentation model and trained inspection models through an internal active-learning pipeline.

TryTech Co., Ltd.

Founder / Technical Lead

2021-01 – present

LLM applications, human-AI workflows, and industry prototypes.

  • Built a customer-support LLM backend prototype with retrieval, tool orchestration, transaction workflows, and human handoff.
  • Prototyped inspection applications with iOS and web interfaces and deployed machine-learning and custom algorithm modules on AWS using SAM and Lambda for industry pilots.

NCREE–NTUCE Joint AI Research Center

Research Assistant

2020-05 – 2022-07

  • Led six industry research projects developing and deploying deep-learning inspection systems for concrete and steel structures, integrating visual recognition into field inspection and near-real-time reporting.

Selected research

Geometry-Grounded Synthetic Data for Visual Recognition

Parameterized 3D defect models generate rendered imagery, pixel-level masks, and physical defect attributes without manual annotation.

  • Developed a synthetic-data framework that derives supervision directly from controllable defect geometry.
  • Generated 72K synthetic crack images and evaluated transfer across three public real-image segmentation benchmarks; realism enhancement improved zero-shot IoU by 3.7–13.6 points.
  • Developing a CAD-referenced extension for multi-view appearance and geometry representations, matching, and few-shot asset recognition.

Vision-Language Learning for Visual Reporting and Reasoning

Vision-language models and datasets for generating and evaluating reports from visual observations.

  • Co-developed VisualSiteDiary, a detector-free vision-language Transformer for construction photolog captioning and image retrieval (equal contribution).
  • Developed VL-Con, a construction vision-language dataset and few-shot prompting framework using in-domain visual-text demonstrations.
  • Evaluated single- and multi-agent vision-language workflows for inspection reporting, including model-generated self-assessment to identify low-quality outputs.
Full publication list ↗

Engineering projects

LLM Evaluation and Agent Workflows

Controlled experiments and rubric iteration for an autorater of troubleshooting hypotheses; agent skill-graph design.

  • Compared scoring formulations and rubric-grounded prompts, using disagreement analysis to refine the evaluator.
  • Achieved over 90% repeated-evaluation score consistency and over 90% alignment with human ratings on the evaluated tasks.
  • Designed a graph representation of skill sequencing and dependencies for relationship-aware retrieval, orchestration, and library checks.

Vision Model Adaptation and Benchmarking

Controlled benchmarks and model adaptation for infrastructure and facility imagery.

  • Compared image-matching and 3D reconstruction methods under low-texture, repeated-pattern, sparse-view, and illumination-shift conditions.
  • Integrated FiLM-conditioned camera-orientation cues into Swin Transformer segmentation and extended inspection-specific object classes.
  • Trained and iterated on domain-specific recognition models using the company's existing active-learning pipeline.

Applied AI Prototypes and Human Handoff

Customer-support prototype integrating retrieval, tools, backend transactions, and human escalation.

  • Built the LLM backend and agent-console handoff for an omnichannel support prototype.
  • Developed conversation-analysis workflows and an embedding-based classifier for identifying effective service utterances.
  • In separate inspection projects, built iOS and web interfaces and deployed modules on AWS using SAM and Lambda for industry pilots.

Education

Ph.D. (expected 2027)

Construction Engineering and Management · University of Illinois Urbana-Champaign

2022-08 – present

GPA: 4.0

  • Research: multimodal AI, computer vision, 3D vision, and geometry-grounded synthetic data.
  • Teaching Assistant, Computer Vision (Spring 2024); Government Fellowship for Studying Abroad (2024–2026).

Master of Computer Science

Computer Science · University of Illinois Urbana-Champaign

2023-08 – 2024-12

GPA: 4.0

Technical skills

Programming
Python, C++17, SQL
Machine Learning
PyTorch, Hugging Face Transformers, scikit-learn, OpenCV, Vision-language modeling, LLM/RAG evaluation
Vision / 3D
COLMAP, Structure-from-motion, Multi-view stereo, 3D reconstruction, Synthetic data generation
Systems
CUDA, Linux, Docker, Git, AWS