Archive for ChatGPT

« Older Entries

The Counties of the United Kingdom (according to ChatGPT)

Posted in Artificial Intelligence with tags , , , , on December 23, 2025 by telescoper

Regular readers will know that I sometimes use this blog to educate the Great Unwashed about the facts of British geography (including where the North begins). I have decided to enlist the help of Generative AI to support me with this task so, with a little help from social media, here is a response from ChatGPT to a prompt requesting a map showing all the counties of the United Kingdom with their names. The result, as you can see, is truly spectacular:

I began my research career at the University of Bulgaria, by the way.

1 Comment »

Generative AI in Physics?

Posted in Artificial Intelligence, Education, mathematics, Maynooth with tags , , , , , , , , , on August 11, 2025 by telescoper

As a new academic year approaches we are thinking about updating our rules for the use of Generative AI by physics students. The use of GenAI for writing essays, etc, has been a preoccupation for many academic teachers. Of course in Physics we ask our students to write reports and dissertations, but my interest in what we should do about the more mathematical and/or computational types of work. A few years ago I looked at how well ChatGPT could do our coursework assignments, especially Computational Physics, and it was hopeless. Now it’s much better, though still by no means flawless, and now there are also many other variants on the table.

The basic issue here relates to something that I have mentioned many times on this blog, which is the fact that modern universities place too much emphasis on assessment and not enough on genuine learning. Students may use GenAI to pass assessments, but if they do so they don’t learn as much as they would had they done the working out for themselves. In the jargon, the assessments are meant to be formative rather than purely summative.

There is a school of thought that has the opinion that formative assessments should not gain credit at all in the era of GenAI since “cheating” is likely to be widespread. The only secure method of assessment is through invigilated written examinations. Students will be up in arms if we cancel all the continuous assessment (CA), but a system based on 100% written examinations is one with which those of us of a certain age are very familiar.

Currently, most of our modules in theoretical physics in Maynooth involve 20% coursework and 80% unseen written examination. That is enough credit to ensure most students actually do the assignments, but the real purpose is that the students learn how to solve the sort of problems that might come up in the examination. A student who gets ChatGPT to do their coursework for them might get 20%, but they won’t know enough to pass the examination. More importantly they won’t have learnt anything. The learning is in the doing. It is the same for mathematical work as it is in a writing task; the student is supposed to think about the subject not just produce an essay.

Another set of issues arises with computational and numerical work. I’m currently teaching Computational Physics, so am particularly interested in what rules we might adopt for that subject. A default position favoured by some is that students should not use GenAI at all. I think that would be silly. Graduates will definitely be using CoPilot or equivalent if they write code in the world outside university so we should teach them how to use it properly and effectively.

In particular, such methods usually produce a plausible answer, but how can a student be sure it is correct? It seems to me that we should place an emphasis on what steps a student has taken to check an answer, which of course they should do whether they used GenAI or did it themselves. If it’s a piece of code to do a numerical integration of a differential equation, for example, the student should test it using known analytic solutions to check it gets them right. If it’s the answer to a mathematical problem, one can check whether it does indeed solve the original equation (with the appropriate boundary conditions).

Anyway, my reason for writing this piece is to see if anyone out there reading this blog has any advice to share, or even a link to their own Department’s policy on the use of GenAI in physics for me to copy adapt for use in Maynooth! My backup plan is to ask ChatGPT to generate an appropriate policy…

16 Comments »

Publisherballs

Posted in Uncategorized with tags , , , , , on January 26, 2025 by telescoper

Private Eye has a regular feature called “Commentatorballs” which contains various verbal gaffes perpetrated by sports commentators, usually sent in by members of the public. It used to be called “Colemanballs” after David Coleman, but the name was changed after he passed away in 2013. Years ago, I had a couple of my own contributions published, actually.

Anyway, I’m seriously thinking about running a similar feature on this blog devoted to outrageous blunders made by academic publishers.

Here’s one that would make a good entry in “Publisherballs” from the journal Radiology Case Reports, published by – you guessed it – Elsevier:

On the left you see the title, author data and abstract and on the right an extract from the text that clearly shows it was generated by ChatGPT or some other AI bot. The authors cut and pasted the output without even noticing that the software they were using was explaining it could not respond to the prompt they gave it. Clearly the careful editorial process at the journal didn’t even go as far as reading the paper…

4 Comments »

The Dangers of AI in Science Education

Posted in Education with tags , , , , on January 17, 2025 by telescoper

I’m taking the liberty of reblogging this post from an experienced university teacher of chemistry and physics outlining some of the dangers posed by the encroachment of Artificial Intelligence into science education. It’s quite a long piece, but well worth reading in its entirety

The ethical case for resisting AI

4 Comments »

On Papers Written Using Large Language Models

Posted in Uncategorized with tags , , , , , , , on March 26, 2024 by telescoper

There’s an interesting preprint on arXiv by Andrew Gray entitled ChatGPT “contamination”: estimating the prevalence of LLMs in the scholarly literature that tries to estimate how many research articles there are out there that have been written with the help of Large Language Models (LLMs) such as ChatGPT. The abstract of the paper is:

The use of ChatGPT and similar Large Language Model (LLM) tools in scholarly communication and academic publishing has been widely discussed since they became easily accessible to a general audience in late 2022. This study uses keywords known to be disproportionately present in LLM-generated text to provide an overall estimate for the prevalence of LLM-assisted writing in the scholarly literature. For the publishing year 2023, it is found that several of those keywords show a distinctive and disproportionate increase in their prevalence, individually and in combination. It is estimated that at least 60,000 papers (slightly over 1% of all articles) were LLM-assisted, though this number could be extended and refined by analysis of other characteristics of the papers or by identification of further indicative keywords.

Andrew Gray, arXiv:2403.16887

The method employed to make the estimate involves identifying certain words that LLMs seem to love, of which usage has increased substantially since last year. For example, twice as many papers call something “intricate” nowadays compared to the past; there are also increases in the use of the words “commendable” and “meticulous”.

I found this a commendable paper, which is both meticulous and intricate. I encourage you to read it.

P.S. I did not use ChatGPT to write this blog post.

Leave a comment »

Assessment in the Age of AI

Posted in Education, Maynooth with tags , , , , on May 5, 2023 by telescoper

The arrival of AI bots such as ChatGPT continues to cast a shadow over student assessment in the third-level institutions, as academics are realizing that these algorithms are getting better and better at the tasks asked of students, especially straightforward writing tasks (perhaps including simple calculations) as well as the traditional student essay.

Before going further I have to admit that I’ve never really understood the obsession in some parts of academia with “the Essay” as a form of assessment. I agree that writing skills are extremely important but they’re not the only skills it is important for students to acquire during the course of a degree. Learning how to do things seems to me to be more important than writing about things other people have done. While forms of assessment in science subjects have evolved considerably over the last 50 years, some other domains still seem to concentrate almost exclusively on “The Essay”.

Systems such as ChatGPT can produce text on demand (with a variable degree of success) using sources on the internet. This is not great at dealing with technically complex specialist topics but can produce plausible if somewhat superficial offerings in many circumstances where something less demanding is required. I know that staff in some science departments find that these systems can score essentially 100% on their first-year coursework assignments. Urgent meetings are being called and working groups being set up about this. Panic is in the air.

My immediate response to the situation is very twofold:

  1. Don’t panic!
  2. If an assessment can be aced by a bot then it should not contribute towards credit unless the students do it in a supervised environment, e.g. as an in-class test rather than a take-home assignment.
  3. More importantly, if a student with only a superficial knowledge can score a high mark on an assessment, what is the value of the assessment anyway?

It seems to me that the intervention of ChatGPT should cause academics to reflect much more deeply on what it is that they are trying to assess, and that should lead to new forms of assessment that can’t be performed by AI bots as well as the scrapping of many existing assessment activities, many of which (in my opinion) are pointless. There is so much inertia in academia, however, that such a radical rethink will be forthcoming on the timescale required.

All of which waffly nonsense reminded me of a joke I heard many years ago.

Q: How many academics does it take to change a lightbulb?

A: What do you mean, change?

11 Comments »

A Draft Strategic Plan for Maynooth (via ChatGPT)

Posted in Education, Maynooth with tags , , on April 23, 2023 by telescoper

Senior managers at Maynooth University are currently engaged in the expensive and time-consuming process of compiling a new “Strategic Plan“. All universities seem to feel the need to have one of these, although I’ve never understood what purpose is served by such a document. They are all remarkably similar to one another in any case. Anyway, this is what ChatGPT produced for free in a matter of seconds in response to the query “please write a first draft strategic plan for Maynooth University”:

Introduction:

Maynooth University is a leading university in Ireland founded in 1795. The University aims to provide high-quality education to its students and prepare them for successful careers and to contribute positively to society as a whole. As a forward-thinking institution, Maynooth University recognizes the need for strategic planning to continue its legacy of excellence and ensure that it remains at the forefront of education and research.to conduct innovative research that address the challenges of the 21st century. In order to achieve its goals, the university needs to develop a strategic plan that outlines its priorities and the steps it needs to take to achieve them.

Vision:

The vision for Maynooth University is to become a world-class institution of higher education that promotes excellence in teaching, research and community engagement. The university will strive to provide a stimulating and inclusive environment that inspires its students, faculty and staff to achieve their full potential.

Mission:

The mission of Maynooth University is to provide a transformative student experience that prepares graduates to be leaders in their fields. The university will contribute to the creation and dissemination of new knowledge through its research and scholarship. It will engage with local, national and international communities to promote social and economic development.

Goals:

The following are the key goals of Maynooth University:

1. To enhance the quality of teaching and learning:

Maynooth University will provide its students with a challenging and engaging learning experience that equips them with the knowledge and skills required to succeed in their chosen fields. The university will invest in innovative teaching methods, modern teaching equipment, and professional development opportunities for its faculty to enhance their teaching skills and delivery.

2. To advance research and scholarship:

Maynooth University aims to become a leading research institution known for its excellence in research, innovation, and scholarship. The university will prioritize research funding and support for its faculty, students, and research teams. It will also invest in research infrastructure, facilities and equipment to support cutting-edge research across a range of disciplines.

3. To foster community engagement:

Maynooth University will strive to promote social and economic development through community engagement, public service, and outreach. The university will work with communities, government, and other stakeholders to address societal challenges and create positive social impact. Maynooth University will also prioritize outreach and engagement with schools, businesses, and cultural organizations in the region.

Strategies:

The following are the strategies that Maynooth University will use to achieve its goals:

  1. Develop innovative and relevant degree programs that meet the needs of employers and society.
  2. Support faculty development, recruitment, and retention to ensure that Maynooth University maintains its position as a leading research institution.
  3. Expand research partnerships, collaborations and funding through strategic alliances with other universities, government and industry.
  4. Foster a culture of interdisciplinary research, linking research with teaching and learning, and research with community engagement.
  5. Promote a sense of community and belonging among the Maynooth University community through effective communication and engagement strategies.

Conclusion:

Maynooth University is committed to providing a world-class education, conducting innovative research, and contributing to the social and economic development of communities. The strategic plan outlines the goals and strategies that the university will undertake to achieve its vision and mission. The university will continually review and update its strategies to ensure they are aligned with changes in the higher education landscape to maintain its standing as a leading institution of higher learning.

5 Comments »

Meanwhile, in an Arts & Humanities Faculty near you…

Posted in Education with tags on April 22, 2023 by telescoper
From Private Eye Magazine

P.S. I can’t wait to see what ChatGPT produces for Maynooth University’s new Strategic Plan…

Leave a comment »

Essays and (Computational) Physics

Posted in Biographical, Education, Maynooth with tags , , , , , on March 14, 2023 by telescoper

There have been more news stories about ChatGPT and assessment in universities going around. There’s one here from The Journal and another here from The Conversation to give just two examples.

I wrote about this myself a couple of months ago in a post that included this:

I have to admit that I’ve never really understood the obsession in some parts of academia with “the student Essay” as a form of assessment. I agree that writing skills are extremely important but they’re not the only skills it is important for students to acquire during the course of a degree. Of course I’m biased because I work in Theoretical Physics, an area in which student essays play a negligible role in assessment. Our students do have to write project reports, etc, but writing about something you yourself have done seems to me to be different from writing about what other people have done. While forms of assessment in science subjects have evolved considerably over the last 50 years, other domains still seem to concentrate almost exclusively on “The Essay”.

Whatever you think about the intrinsic value of The Essay (or lack thereof) it is clear that if it is not done in isolation (and under supervision) it is extremely vulnerable to cheating.

A few people have retorted that communication skills are very important in higher education. I agree with that wholeheartedly, but it seems to me that (a) there are other ways of communicating than via formal essays and (b) there are, should be, more to academic study than  writing about things.

That said, I do think we could be doing more in some disciplines, including my own, to cultivate communication skills in general and writing skills in particular. In Theoretical Physics we certainly don’t do this as much as we should. I do have a project report in my 3rd Year computational physics module, but that is a relatively short document and the report itself counts only one-third of the marks (and the project is only 40% of the module mark).

These thoughts somehow reminded me of this. You can click on it to make it bigger if it’s difficult to read. It was the first paper (called colloquially Paper Zero) of my finals examination at the University of Cambridge way back in 1985, getting on for 40 years ago:

wpid-wp-1425648226410.jpeg

As you can probably infer from the little circle around number 4, I decided to write an Essay about topic 4. I’ve always been interested in detective stories so this was an easy choice for me, but I have absolutely no idea what I wrote about for three hours. Nor do I recall actually ever getting a mark for the essay, so I never really knew whether it really counted for anything. I do remember, however, that I had another 3-hour examination in the afternoon of the same day, two three-hour examinations the following day, and would have had two the day after that had I not elected to do a theory project which let me off one paper at the end and for which I got a good mark.

Anyway, to get back to the essay paper, we certainly don’t set essay examinations like that here in the the Department of Theoretical Physics at Maynooth University and I suspect they no longer do so in the Department of Physics at Cambridge either. At the time I didn’t really see the point of making students write such things under examination conditions but then we didn’t have ChatGPT way back then. No doubt it could generate a reasonable essay on any of the topics given.

I am skeptical about whether any of my 3rd year computational physicists would use ChatGPT to write their reports, but they might. But ChatGPT can write Python code too. Am I worried about that? Not greatly. I’ve asked it to write scripts for the various class exercises I’ve set so far and the code it has produced has usually failed. It will get better though….

7 Comments »

What should it mean to be an author of a scientific paper?

Posted in Open Access, The Universe and Stuff with tags , , , on February 12, 2023 by telescoper

The implementation of artificial intelligence techniques in tools for generating text (such as ChatGPT) has caused a lot of head-scratching recently as organizations try to cope with the implications. For instance, I noticed that the arXiv recently adopted a new policy on the use of generative AI in submissions. One obvious question is whether ChatGPT can be listed as an author. This has an equally obvious answer: “no”. Authors are required to acknowledge the use of such tools when they have used them in writing a paper.

One particular piece of the new policy statement caught my eye:

…by signing their name as an author of a paper, they each individually take full responsibility for all its contents, irrespective of how the contents were generated. If generative AI language tools generate inappropriate language, plagiarized content, biased content, errors, mistakes, incorrect references, or misleading content, and that output is included in scientific works, it is the responsibility of the author(s).

The first sentence of this quote states an obvious principle, but there are situations in which I don’t think it is applied in practice. One example relates to papers emanating from large collaborations or consortia, where the author lists are often very long indeed, sometimes numbering in the thousands. Not all the “authors” of such papers will have even read the paper, so do they “each individually take full responsibility”? I don’t think so. And how can this principle be enforced as policy?

All large consortia have methods for assigning authorship rights as a way of assigning credit for contributions made. But why does “credit” have to mean “authorship”? Papers just don’t have thousands of authors, in the meaningful sense of the term. It’s only ever a handful of people who actually do any writing. That doesn’t mean that the others didn’t do any work. The project would probably not have been possible without them. It does mean, however, that pretending that they participated in writing the article that describes the work isn’t be the right way to acknowledge their contribution. How are young scientists supposed to carve out a reputation if their name is always buried in immensely long author lists? The very system that attempts to give them credit at the same renders that credit worthless.

As science evolves it is extremely important that the methods for disseminating scientific results evolve too. The trouble is that they aren’t. We remain obsessed with archaic modes of publication, partly because of innate conservatism and partly because the lucrative publishing industry benefits from the status quo. The system is clearly broken, but the scientific community carries on regardless. When there are so many brilliant minds engaged in this sort of research, why are so few willing to challenge an orthodoxy that has long outlived its usefulness.

In my view the real problem is not so much the question of authorship but the very idea of the paper. It seems quite clear to me that the academic journal is an anachronism. Digital technology enables us to communicate ideas far more rapidly than in the past and allows much greater levels of interaction between researchers. The future for many fields will be defined not in terms of “papers” which purport to represent “final” research outcomes, but by living documents continuously updated in response to open scrutiny by the community of researchers. I’ve long argued that the modern academic publishing industry is not facilitating but hindering the communication of research. The arXiv has already made academic journals redundant in many of branches of  physics and astronomy; other disciplines will inevitably follow. The age of the academic journal is drawing to a close. Now to rethink the concept of “the paper”.

In the meantime I urge all scientists to remember that by signing their name as an author of a paper, they individually take full responsibility for all its contents. That means to me that at the very least you should have read the paper you’re claiming to have written.

7 Comments »

« Older Entries