Talk Title: Automatic Testing of Biases in LLMs

Abstract: LLMs tend to reflect the biases and ethical issues of our society. This is problematic for companies developing or adopting LLMs as new regulations in place prohibit biased and discriminatory decisions. In this talk, I’ll present LangBiTe, an open-source testing platform to systematically assess the presence of biases in an LLM. LangBiTe enables red teams to tailor their test scenarios, and automatically generate and execute the test cases according to a set of user-defined ethical requirements.

Talk Title: Affective Linguistic Programming: Hacking LLM Ethical Decision-Making with Emotion and Framing

Abstract: This presentation introduces the latest update to our ethical audit of 3 leading SOTA and 17 open-source LLM models. In our latest study, we find open-source models like Llama 3 are closing the gap in performance. We also update our previous study by benchmarking decision-making against graduated empathetic backstory and syntactic framing variation. In many cases, LLMs demonstrate striking reversals in decision-making. While LLM decision-making increasingly models human behavior, our findings also suggest vulnerabilities in LLM systems.

Talk Title: Using fine-tuned Lama models to build historical databases

Abstract: The main limitation on the productivity of a historian is how much he or she can read during a professional life. This means that a machine or a system that can systematically analyze the vast amounts of machine readable historical documents that are available today will have a transformative effect on the discipline of history.

In this talk I explain how I came to use large language models to extract information from large amounts of unstructured prose text into a relational database. I describe the journey of experimenting with various models and systems before landing on the LlaMa models. I finish by going through some of the lessons learned.

Talk Title: Embodied Social Intelligence in the Wild: Themes, Challenges, and Future Directions

Abstract: The field of artificial social intelligence aims to create agents that can perceive, reason about, and adapt to human social cues. This entails processing multimodal signals, including visual, vocal, and verbal messages. Operating beyond lab settings in the real world--"in the wild"--gives rise to unique challenges. These arise from inherent trade-offs between several factors: participant privacy, ecological validity, and data fidelity. In this talk, I begin by addressing these challenges within three core research themes: the modeling, analysis, and recording of the dynamics of unscripted human interactions in the wild. Here I draw upon diverse fields, including multimodal machine learning, affective computing, and computer vision. I will then present recent projects that lay the groundwork for an exciting new direction: autonomous AI agents capable of open-ended, lifelong social learning in real-world settings.

Talk Title: LLM use for Community Health Promotion in Low-Middle Income Countries

Abstract: Meet HIBA, the LLM-driven chatbot designed to revolutionize health information accessibility and health literacy in vulnerable communities. In a field where innovation always redraws boundaries, HIBA leverages cutting-edge Large Language Models (LLMs) to make quality health support more accessible. We will discuss how this WhatsApp-based tool aims to empower under-served and overlooked communities with readily accessible health information in text and audio form, in 28 languages. We will highlight some of the results of the deployment in Lebanon and Jordan.

Talk Title: LingOly: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages

Abstract: We present the LingOly benchmark, a new benchmark for advanced reasoning abilities in large language models. Using challenging Linguistic Olympiad puzzles, we evaluate (i) capabilities for in-context identification and generalisation of linguistic patterns in very low-resource or extinct languages, and (ii) abilities to follow complex task instructions. The LingOly benchmark covers more than 90 mostly low-resource languages, minimising issues of data contamination, and contains 1,133 problems across 6 formats and 5 levels of human difficulty. We assess performance with both direct accuracy and comparison to a no-context baseline to penalise memorisation. We find that in absence of memorisation, true multi-step out-of-domain reasoning remains a challenge for current language models.

Talk Title: Enhancing Educational Access for Deaf and Hard of Hearing Students through LLMs

Abstract: LLMs have the potential to assist in creating educational content, such as quizzes, instructional materials, and responses to student queries. Designing the educational content requires careful considerations of diverse learner needs, background, and abilities. I will present the opportunities and technical challenges of applying LLMs to improve the learning experience of deaf and hard of hearing students in recorded lectures. Enhancing inclusivity and accessibility poses new problems and methods in machine learning and NLP research.

Talk title: Using Open LLMs to Understand and Improve Human-centered Tasks and Outcomes

Abstract: Open LLMs are essential for bridging accessibility divides related to availability, cost, transparency, and compliance. In this talk, I discuss how state-of-the-art open LLMs have closed the gap with their closed-source counterparts across four common learning paradigms – embeddings, fine-tuning, few-shot learning, and generative agents. I illustrate this using real-world projects involving diverse populations in contexts such as depression detection and automated essay scoring.

Talk Title: Human-understandable Explanations for Deep Learning Models Handling Sequential Data

Abstract: Sequential data is prevalent across many real-world applications, such as natural language, financial data, signal data, etc. Deep Neural Networks (DNNs), such as Recurrent Neural Networks (RNNs) and Transformers, have shown promising results in modelling sequential data and extracting useful features for various real-world applications. However, these DNNs are often seen as black boxes, making it difficult for users to understand the reasons behind the decisions made by these models. This understanding is crucial, especially in critical applications like patient monitoring, legal document summarization, and stock price prediction. Current efforts have focused on using eXplainable Artificial Intelligence (XAI) techniques such as backpropagation, perturbation, attention, and approximation to shed light on these models. However, these XAI methods provide feature attributions, which can be difficult for lay users to understand due to the complex nature of sequential data, such as signal data. While XAI methods have been developed to generate human-understandable explanations for image data, similar efforts have not been widely explored in the domain of sequential data. In this talk, we will explore the need for human-understandable explanations for DNNs handling sequential data and review existing XAI methods designed to generate more human-understandable explanations.

Talk Title: Using LLMs to support policy advice in The Presidency of South Africa: Practical applications and ethical implications

Abstract: The application of large language models (LLMs) is particularly suited to environments where substantial amounts of English documents and other forms of textual input need to be processed daily by small teams. In collaboration with The Presidency of South Africa, the Policy Innovation Lab at Stellenbosch University is exploring specific use cases within a government unit focused on providing policy advice. This exploration includes identifying practical applications of LLMs, establishing ethical guardrails, and assessing the implications for existing policy frameworks in South Africa.

Talk Title: Pathway to Inclusivity in Human Language Technology for under-resourced languages in Kenya

Abstract: This talk will address the language digital divide in under-resourced languages in Africa and the need for tools, resources and applications for Kikamba, Kipsigis, Swahili, and Egekusii. This talk will review how we build a multilingual ecosystem for translation for these languages, and how we can continue to build this out further in the era of generative AI. 

Talk Title: Aligning Language Models with Expert-Defined Standards 

Abstract: Domain experts across engineering, healthcare, and education follow strict standards for producing quality content such as technical manuals, medication instructions, and children's reading materials. However, current works in controllable text generation have yet to explore using these standards as references for control. I will present our EMNLP 2024 paper introducing Standardize, a retrieval-style in-context learning-based framework to guide large language models to align with expert-defined standards. Focusing on English language standards in the education domain as a use case, we consider the Common European Framework of Reference for Languages (CEFR) and Common Core Standards (CCS) for the task of open-ended content generation. Our findings show that models can gain a 40% to 100% increase in precise accuracy for Llama and GPT-4, respectively, demonstrating that the use of knowledge artifacts extracted from standards and integrating them in the generation process can effectively guide models to produce better standard-aligned content.

Talk Title: Building Ethical AI: Practical Strategies for Responsible Implementation 

Abstract: As AI technologies become increasingly integral to society, the importance of implementing ethical guidelines throughout the development process is paramount. This conference focuses on practical, actionable strategies for building ethical design for AI systems.

Talk Title: Responsibility at Meta

Talk Title: Building an Open Source AI Clinical Trial Matching Platform

Abstract: Dana-Farber Cancer Institute (DFCI) was recently awarded a Llama Impact Grant to build a new open source AI platform to computationally match patients with cancer to clinical trials. We plan to use Llama 3 to summarize and comprehend unstructured clinical notes and unstructured clinical trial eligibility criteria, enabling rapid identification of appropriate trial options for individual patients. The new AI models will be integrated into DFCI’s existing open source MatchMiner platform and evaluated with clinical partners.

Talk Title: Farmer.Chat — Effective AI enabled advisory for farmers

Abstract: Digital Green has been working with smallholder farmers for more than a decade to improve their livelihoods. Farmer.Chat is our latest platform to provide scalable AI enabled advisory to farmers and extension agents. In this talk, we will describe a few key AI components built in the platform, discuss approaches to improve efficacy esp. in low-resource language settings.

Talk Title: Competence Framework for AI Technologies

Abstract: In this presentation, we will introduce a competence framework specifically designed for AI technologies. This framework will detail various Qualification Profiles, outlining the necessary levels of proficiency in AI and the corresponding Proficiency Areas associated with each profile. Furthermore, we will include a detailed table that illustrates the estimated time required to achieve proficiency in each qualification profile, providing a clear roadmap for learners and professionals aiming to enhance their skills in the field of artificial intelligence. This framework is based on the "European Competence Framework for Quantum Technologies."

Talk Title: Latest developments in Llama 3 and Meta’s perspective on open-source