Shun-Hsiang Hsu / Resume
Research & experience.
Machine learning research in multimodal learning, computer vision, geometry-grounded synthetic data, and LLM evaluation. Experience in controlled experiments, benchmarking, model adaptation, and research prototypes.
UIUC Ph.D. student · Expected 2027 · Champaign, Illinois
Professional experience
Google Google Cloud · Gemini Cloud Assist
Software Engineering Intern (PhD)
2026 – 2026
Summer 2026 · LLM evaluation and skill-based agent systems.
- Designed controlled experiments and refined scoring rubrics for an LLM autorater, achieving over 90% repeated-evaluation score consistency and over 90% alignment with human ratings.
- Designed a graph representation of agent skills to encode sequencing and dependencies for relationship-aware retrieval, orchestration, and skill-library health checks.
Reconstruct Inc.
Computer Vision Engineer Intern
2024-01 – 2026-05
Computer vision and 3D method evaluation, model adaptation, and training iteration.
- Built controlled image-matching and 3D reconstruction benchmarks under low-texture, repeated-pattern, sparse-view, and illumination-shift conditions to guide production method selection.
- Integrated FiLM-conditioned camera-orientation cues into a Swin Transformer segmentation model and trained inspection models through an internal active-learning pipeline.
TryTech Co., Ltd.
Founder / Technical Lead
2021-01 – present
LLM applications, human-AI workflows, and industry prototypes.
- Built a customer-support LLM backend prototype with retrieval, tool orchestration, transaction workflows, and human handoff.
- Prototyped inspection applications with iOS and web interfaces and deployed machine-learning and custom algorithm modules on AWS using SAM and Lambda for industry pilots.
NCREE–NTUCE Joint AI Research Center
Research Assistant
2020-05 – 2022-07
- Led six industry research projects developing and deploying deep-learning inspection systems for concrete and steel structures, integrating visual recognition into field inspection and near-real-time reporting.
Selected research
Geometry-Grounded Synthetic Data for Visual Recognition
Parameterized 3D defect models generate rendered imagery, pixel-level masks, and physical defect attributes without manual annotation.
- Developed a synthetic-data framework that derives supervision directly from controllable defect geometry.
- Generated 72K synthetic crack images and evaluated transfer across three public real-image segmentation benchmarks; realism enhancement improved zero-shot IoU by 3.7–13.6 points.
- Developing a CAD-referenced extension for multi-view appearance and geometry representations, matching, and few-shot asset recognition.
Vision-Language Learning for Visual Reporting and Reasoning
Vision-language models and datasets for generating and evaluating reports from visual observations.
- Co-developed VisualSiteDiary, a detector-free vision-language Transformer for construction photolog captioning and image retrieval (equal contribution).
- Developed VL-Con, a construction vision-language dataset and few-shot prompting framework using in-domain visual-text demonstrations.
- Evaluated single- and multi-agent vision-language workflows for inspection reporting, including model-generated self-assessment to identify low-quality outputs.
Engineering projects
LLM Evaluation and Agent Workflows
Controlled experiments and rubric iteration for an autorater of troubleshooting hypotheses; agent skill-graph design.
- Compared scoring formulations and rubric-grounded prompts, using disagreement analysis to refine the evaluator.
- Achieved over 90% repeated-evaluation score consistency and over 90% alignment with human ratings on the evaluated tasks.
- Designed a graph representation of skill sequencing and dependencies for relationship-aware retrieval, orchestration, and library checks.
Vision Model Adaptation and Benchmarking
Controlled benchmarks and model adaptation for infrastructure and facility imagery.
- Compared image-matching and 3D reconstruction methods under low-texture, repeated-pattern, sparse-view, and illumination-shift conditions.
- Integrated FiLM-conditioned camera-orientation cues into Swin Transformer segmentation and extended inspection-specific object classes.
- Trained and iterated on domain-specific recognition models using the company's existing active-learning pipeline.
Applied AI Prototypes and Human Handoff
Customer-support prototype integrating retrieval, tools, backend transactions, and human escalation.
- Built the LLM backend and agent-console handoff for an omnichannel support prototype.
- Developed conversation-analysis workflows and an embedding-based classifier for identifying effective service utterances.
- In separate inspection projects, built iOS and web interfaces and deployed modules on AWS using SAM and Lambda for industry pilots.
Education
Ph.D. (expected 2027)
Construction Engineering and Management · University of Illinois Urbana-Champaign
2022-08 – present
GPA: 4.0
- Research: multimodal AI, computer vision, 3D vision, and geometry-grounded synthetic data.
- Teaching Assistant, Computer Vision (Spring 2024); Government Fellowship for Studying Abroad (2024–2026).
Master of Computer Science
Computer Science · University of Illinois Urbana-Champaign
2023-08 – 2024-12
GPA: 4.0
Technical skills
- Programming
- Python, C++17, SQL
- Machine Learning
- PyTorch, Hugging Face Transformers, scikit-learn, OpenCV, Vision-language modeling, LLM/RAG evaluation
- Vision / 3D
- COLMAP, Structure-from-motion, Multi-view stereo, 3D reconstruction, Synthetic data generation
- Systems
- CUDA, Linux, Docker, Git, AWS