[Theory] REVISED: 9/18 Talks at TTIC: Kanishka Misra, UT Austin

[Mary Marre]

*When:*        Monday, September 18, 2023 at* 11:30** a**m CT   *


*Where:       *Talk will be given *live, in-person* at

                   TTIC, 6045 S. Kenwood Avenue

                   5th Floor, Room 530


*Virtually:*   * via* Panopto *livestream*
<https://uchicago.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=f2ca8655-9ba3-4cb9-b45e-b07e003e833c>


*Who: *     Kanishka Misra, *UT Austi**n*(Linguistics)


------------------------------

*Title:          *Analyzing Robust Conceptual Knowledge and Property
Inheritance in Language Models

*Abstract: *A characteristic feature of human semantic cognition is the
ability to not only store and retrieve the properties of concepts observed
through experience, but to also facilitate the inheritance of properties
(e.g., can breathe) from superordinate concepts (animal) to their
subordinates (dog)---i.e. demonstrate property inheritance. In this talk, I
will present COMPS, a collection of minimal pair sentences that jointly
tests pre-trained language models (PLMs) on their ability to attribute
properties to concepts and their ability to demonstrate property
inheritance behavior. Analyses of 31 different PLMs on COMPS reveal that
they can easily distinguish between concepts on the basis of a property
when they are trivially different but find it relatively difficult when
concepts are related on the basis of explicit knowledge representations.
Further analyses find that PLMs can show behaviors suggesting successful
property inheritance in simple contexts, but fail in the presence of
distracting information, which decreases the performance of many models,
sometimes even below chance. This lack of robustness in demonstrating
simple reasoning raises important questions about PLMs’ capacity to make
correct inferences even when they appear to possess the prerequisite
knowledge. I will also discuss preliminary results on a potential few-shot
setting of this task, complemented with additional controls for positional
heuristics.

*Bio: *Kanishka Misra is a postdoc at the department of linguistics at UT
Austin with Kyle Mahowald. He received his PhD at Purdue University with
Julia Rayz and Allyson Ettinger (UChicago). His research focuses on
analyzing and evaluating language models and neural networks from the
perspective of semantic cognition, analyzing their ability to extract
conceptual meaning and demonstrate reasoning compatible with the nuances of
human conceptual knowledge. He and his collaborators have been recognized
with a best paper award at EACL 2023 and an outstanding paper award at ACL
2023.



*Host: *Avrim Blum <avrim at ttic.edu>



Mary C. Marre
Faculty Administrative Support
*Toyota Technological Institute*
*6045 S. Kenwood Avenue, Rm 517*
*Chicago, IL  60637*
*773-834-1757*
*mmarre at ttic.edu <mmarre at ttic.edu>*


On Thu, Sep 14, 2023 at 10:54 PM Mary Marre <mmarre at ttic.edu> wrote:

> *When:*        Monday, September 18, 2023 at* 11:30** a**m CT   *
>
>
> *Where:       *Talk will be given *live, in-person* at
>
>                    TTIC, 6045 S. Kenwood Avenue
>
>                    5th Floor, Room 530
>
>
> *Virtually:*   * via* Panopto *livestream*
> <https://uchicago.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=f2ca8655-9ba3-4cb9-b45e-b07e003e833c>
>
>
> *Who: *         Kanishka Misra, Purdue
>
>
> ------------------------------
>
> *Title:          *Analyzing Robust Conceptual Knowledge and Property
> Inheritance in Language Models
>
> *Abstract: *A characteristic feature of human semantic cognition is the
> ability to not only store and retrieve the properties of concepts observed
> through experience, but to also facilitate the inheritance of properties
> (e.g., can breathe) from superordinate concepts (animal) to their
> subordinates (dog)---i.e. demonstrate property inheritance. In this talk, I
> will present COMPS, a collection of minimal pair sentences that jointly
> tests pre-trained language models (PLMs) on their ability to attribute
> properties to concepts and their ability to demonstrate property
> inheritance behavior. Analyses of 31 different PLMs on COMPS reveal that
> they can easily distinguish between concepts on the basis of a property
> when they are trivially different but find it relatively difficult when
> concepts are related on the basis of explicit knowledge representations.
> Further analyses find that PLMs can show behaviors suggesting successful
> property inheritance in simple contexts, but fail in the presence of
> distracting information, which decreases the performance of many models,
> sometimes even below chance. This lack of robustness in demonstrating
> simple reasoning raises important questions about PLMs’ capacity to make
> correct inferences even when they appear to possess the prerequisite
> knowledge. I will also discuss preliminary results on a potential few-shot
> setting of this task, complemented with additional controls for positional
> heuristics.
>
> *Bio: *Kanishka Misra is a postdoc at the department of linguistics at UT
> Austin with Kyle Mahowald. He received his PhD at Purdue University with
> Julia Rayz and Allyson Ettinger (UChicago). His research focuses on
> analyzing and evaluating language models and neural networks from the
> perspective of semantic cognition, analyzing their ability to extract
> conceptual meaning and demonstrate reasoning compatible with the nuances of
> human conceptual knowledge. He and his collaborators have been recognized
> with a best paper award at EACL 2023 and an outstanding paper award at ACL
> 2023.
>
>
>
> *Host: *Avrim Blum <avrim at ttic.edu>
>
>
>
>
>
>
> Mary C. Marre
> Faculty Administrative Support
> *Toyota Technological Institute*
> *6045 S. Kenwood Avenue, Rm 517*
> *Chicago, IL  60637*
> *773-834-1757*
> *mmarre at ttic.edu <mmarre at ttic.edu>*
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.uchicago.edu/pipermail/theory/attachments/20230915/66b6ed8b/attachment.html>