Job Details

Duties/Responsibilities:
We focus on evaluating BizChat quality across various scenarios, such as summarizing content from documents, generating images, and projecting data into graphs. For each scenario, we have a set of concrete user tasks (e.g., "Could you extract the KPIs from the document and project them into a table?"). We will obtain responses from different versions of AI Assistants. The Applied Scientist (AS) will help us[KM7] [LY8] [YW9] develop an LLM-based labeler that can evaluate the quality of responses given the task.
To achieve this, the AS is expected to combine different methods as appropriate.

• For example, one approach is to define the correct answer for a given user task and match the chatbot response to this answer, such as a mathematical task like "Can you help me sort the following numbers: 67, 111, ...". The AS can generate these answers by combining results from multiple LLM models.
• For creative open-answer questions such as "Summarize the key points from section 4 in this document," the AS can help develop an LLM-based judge to evaluate the quality of the summary. To do so, they are expected to refine the prompt to the LLM judge, guiding the LLM-based judge to score different aspects of the summary based on dimensions such as accuracy, comprehensiveness, and conciseness.

Required Skills/Abilities:

• Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, Applied Mathematics, or a related field AND 3+ years of related experience (e.g., statistics, predictive analytics, research).
• Experience working on successful applied research projects in industry environments.
• Applicants with the following skills would be a plus:
  • Good coding/engineering skill to build production quality solution.
  • Applications of Foundation Models
  • Application of Vision, Audio, and Multimodal Foundation Models
  • Prompt Tuning
  • Natural Language Processing
  • Applied Machine Learning
  • Data Mining with Language Models
  • Data Annotation Specialty

Strong English reading comprehension and ability to quickly analyze and interpret complex information or Applied Linguistics background is also a plus.