I honestly found the whole "Moralmatics" thing pretty funny and, in a way, quite fitting, because… honestly… an integrated conscience box for AI sounds totally weird at first, but in principle it really wouldn't be a bad idea. Maybe this is a dumb question, but: If we "build in" an ethical corrective—who gets to decide what counts as correct? That's where the dilemma already begins, isn't it? For example: I once tested a model on HuggingFace (OpenAssistant Llama2, in case anyone's interested) that played it so extremely safe with certain keywords that almost nothing got through anymore—you kind of feel patronized then too… well, you know what I mean.
What I've noticed in practice is that the biggest challenge isn't even the technology, but everything around it. A lot of people (especially in companies or smaller dev teams) simply use language models without giving much thought to what is actually happening in the background and how badly things can go wrong when the data foundation is garbage or the model carries social prejudices into its answers. It actually happened to me once that a chatbot we were testing internally accidentally came out with misogynistic remarks because that kind of nonsense was included in the training dataset. Obviously, there was frustration at first, until it became clear that it wasn't malicious intent, but simply bad data… still, super uncomfortable.
Akkh already raised an important point with education and awareness. I'd even go a step further: It would be cool if there were something like mandatory "ethics checks" for new models, like a TÜV inspection for cars. A bit like penetration tests for software, except checking for bias and ethical pitfalls. Sounds bureaucratic, but maybe that's what it takes to stop every bit of nonsense from ending up in production somewhere.
I'm also not sure how that would work in everyday life—whether it would paralyze everything or perhaps even promote innovation because developers are forced to think things through more carefully. Has anyone had any experience with this? Or are there already such testing bodies somewhere that one could read more about? I'm still puzzling over how to make this practical… 🤔
