In the 6 years we’ve been operating the TUC Digital Lab programme, the pilot projects, events and shared learning we’ve been running with our affiliated unions has generated a huge amount of content.
There are reports, how-to-guides, case studies, workshop write-ups and blogs. It runs to over 330,000 words in all. As it’s spread over so many documents and such a long period of time, there’s probably nobody who’s actually read it all.
I wanted to see what new uses we could make of this resource, especially for people coming new to the Digital Lab.
So I’ve been working with the generative AI assistant tool Microsoft 365 Copilot to repurpose it by scanning it all and drafting a blog series. The aim is to draw together starting points for people with different interests, and ways into the further resources they might want to check out.
Start the series here
We’ll be releasing the blogs weekly over the next couple of months. We hope it’ll build up into a useful way of thinking about achieving digital change in your unions. And of course, we’d love to hear your thoughts or questions if you want to get in touch.
- Digital unionism means meeting members’ expectations
- Digital unionism means systems and investment
- Digital unionism means skills and organisational culture
- Digital unionism means a deeper relationship with data
- Digital unionism means showing leadership and managing change
How AI was used
At the TUC, we’ve been evaluating the Microsoft 365 Copilot, and Windows Copilot generative AI assistants with a formal pilot project, to see whether and how they might have value for the organisation – and how we might need to develop internal policies and guidance for it, to ensure we’re working to our values and maintaining safety and quality.
I used Copilot more extensively with this series than I have up until now. You’ll often see an AI usage disclaimer on newer Digital Lab reports as I use it to kick around ideas in the planning stage, or when writing to generate different ways of talking about a particular concept, if it is proving hard to explain. But I don’t tend to use it for drafting content to this degree.
I started by grouping together some of the ideas that I think our work has uncovered, and came up with these eight broader themes that I wanted to explore. I wanted to look for common threads that run through our work, no matter which specific technology or area of union operations we were focusing on.
I came up with:
- Meeting member expectations
- Investing in tech infrastructure
- Developing skills and culture
- Increasing use of data
- Digital leadership and change management
- Keeping safe and compliant
- Digital decentralisation
- Innovating and future-scanning
I set Copilot up to scan a SharePoint folder containing all our reports and guides. To get at the content across our blogs, I had Copilot write a python script to convert a database export from WordPress into a long Word doc – filtering out posts from irrelevant categories.
I set it some tone guidelines and got it to draft a post around my first theme, revising the prompt and regenerating to get the tone closer to what I wanted. I edited it line by line, to what I thought was more suitable for a union audience, and to bring out specific Digital Lab subtext that Copilot hadn’t identified. I uploaded the edits to refine the tone further and then set about generating more posts in the same vein.
I still needed to do a line-by-line edit of the whole series, re-ordering or expanding on different points, or deleting chunks where it was going into unnecessary detail just to conform to a standard format.
Overall, this process still took considerable time in preparing materials, constructing prompts, lost time in false starts and a lot of effort in manual editing. But that was maybe half the time that writing the series from scratch would have taken. And whilst I had to add in concepts I felt were missing, it also surfaced many good points across the themes that I probably would have missed if doing it purely by myself from memory.
How Copilot performed
Trying this also helped me learn some shortcomings of the Copilot 365 tool. Despite being marketed as being closely integrated with the Microsoft Graph for each user, this does have severe limitations. It struggled dealing consistently with a large number of documents, and on some themes I noticed its results missing important concepts or giving more generic responses. The 330,000 words I was trying to synthesize overall was much larger than it was able to cope with in any kind of detail.
Importantly, it won’t highlight these kind of limitations to you itself. Generative AI doesn’t “know” anything and isn’t trying to get to the truth behind a question. It’s just coming up with statistically most likely ways that such a question could be answered as coherent text. It will draft any amount of content supportive to the argument in your prompt – even if that means it loses track of evidence and starts making things up.
Ignoring that risk could end up playing along with a hallucination, or generating something that easy to create but isn’t worth reading – the dreaded work slop that makes your life easier but wasting the time of people who receive the work done.
After I noticed and queried this, it recommended me a potential solution to this drift – drafting a new prompt for every post that scanned the documents to build up a ‘corpus index’ (picking relevant points for that theme in each document, one document at a time), so it would not need to operate across all the data at once. I also highlighted the documents I thought were most relevant to each theme, and between these methods it was able to get the volume of content down enough to synthesise and reformat more consistently.
If you have any feedback on this series, or you have experiences you’d like to talk about in working with generative AI in a union context, I’d love to hear it – please do get in touch.
Read the first post… Digital unionism means meeting members’ expectations