Consciousness in the Loop

When we talk about automation and artificial intelligence, we often use terms like human in the loop, human on the loop, and full automation. They describe where the human stands in relation to the system: whether it should approve each action, merely monitor and intervene as needed, or let the system operate independently.

But it is not enough for a human to be present in the loop. What matters is which part of human thinking is present. One can easily be placed in the middle of a decision-making process without really thinking about it — approving, accepting, and continuing almost automatically. In that case, one is technically in the loop, but functions in practice as yet another automated component of the system.

This is where Daniel Kahneman's System 1 and System 2 become interesting. The future of AI governance may be less about having a human in the loop, and more about having conscious reflection in the loop.

Humans are already an automated system

Kahneman's Dual Process Theory describes two fundamental forms of thinking. System 1 is fast, intuitive, and automatic — it reacts to patterns, experiences, associations, and expectations, and requires relatively little mental energy. System 2 is slower, more conscious, and more demanding. It is activated when we need to analyze a problem, perform a calculation, compare alternatives, or challenge our initial intuitive responses.

System 1 is the brain's layer of automation. When we walk through a building, read a facial expression, navigate traffic, or perform a familiar work task, the brain constantly makes judgments without involving our full attention. We do not consciously analyze every traffic sign, every person around us, or every object in our field of vision — a large part of the work is done automatically, and that is necessary. If we had to process all sensory inputs, movements, and small choices through slow, conscious reflection, we would quickly become mentally overloaded. Automation is therefore not a flaw in the human brain, but a prerequisite for us to function.

But automation comes at a cost.

System 1 works with simplifications: it recognizes patterns even when they are incomplete and uses past experiences even when the situation has changed. It reacts to what feels familiar, likely, or socially acceptable — which makes it both extremely competent and systematically fallible.

System 1 as biological full automation

In AI, we talk about full automation when a system can carry out a process without human approval of individual decisions. In this way, System 1 can be seen as a form of biological full automation.

This does not make System 1 unintelligent — it can encompass decades of experience, training, and intuitive expertise. An experienced nurse, mechanic, or designer can often detect that something is wrong before they can consciously explain why. But System 1 is not designed to explain or challenge itself. It delivers an answer, a feeling, or an action — and we can therefore react quickly, but also based on assumptions, habits, biases, and outdated mental models.

Much of our automatic behavior is shaped by biological evolution, cultural influence, social norms, and personal experiences — patterns developed under conditions different from those we encounter in a modern digital society. This does not automatically make them outdated, but it means they do not necessarily fit the situation we are in now.

The brain does not automatically ask:

Is this mental model still relevant?

It often reacts based on:

This looks like something I have seen before.

It is efficient, but not always correct.

System 2 as consciousness in the loop

System 2 becomes more prominent when the automatic response is insufficient — when we encounter something new, when information contradicts itself, or when the consequences of a mistake are significant. We pause, examine the assumptions, consider alternatives, and try to understand what could go wrong.

In AI language, one could call this consciousness in the loop, or perhaps more precisely deliberation in the loop. It is not about consciousness in a philosophical sense, but about the automatic reaction being subjected to active, critical, and purposeful evaluation.

System 2 thus functions as a human control mechanism. It does not monitor all decisions — that would be too slow and resource-intensive — but can be activated when uncertainty, complexity, or risk demands it. It resembles how we should ideally design AI systems: routine and reversible decisions can be automated, uncertain or atypical situations can be elevated to human oversight, and critical decisions may require active approval and reflection.

The question then becomes not whether everything should be automated or controlled by humans, but when the system should escalate from automation to reflection.

Human in the loop is not necessarily thinking in the loop

In many organizations, human in the loop is treated as a safety mechanism in itself: an AI system produces a suggestion, a human looks at it and presses approve. The process is thus not fully automated — on paper, there is human control.

But what happens if the human approves most suggestions without real examination because they are busy, mentally tired, or expect that the AI system is usually correct? Then the human may be physically and organizationally placed in the loop, while System 2 is actually absent. The human acts through System 1: the approval becomes a habit, the automated system suggests, and the human accepts. This creates an illusion of control without real critical assessment.

The problem is whether the human has the necessary prerequisites to realize that they should intervene.

Humans can become part of the automation

The paradox is that human in the loop can in some cases create less security than one might imagine. When a person knows they have the final say, the organization may feel protected. But if the workflow consists of hundreds of uniform approvals, the human quickly becomes trained to accept the system's recommendation — it does not become an independent control function, but an extension of the system.

A biological approval button.

This can happen even when the person is skilled and responsible — the problem lies not with the individual, but in the design of the process. If a person has to go through a large number of AI decisions where 99 percent are correct, attention will naturally wane, and System 1 takes over because the process becomes recognizable and routine. When the one serious error then occurs, the likelihood of it being detected may be lower than the system design suggests.

Therefore, we should not only ask if there is a human in the loop — we should ask if the process is designed to activate human reflection at the right time.

Human on the loop and cognitive escalation

A more realistic model is that humans do not control all decisions, but monitor the system at a higher level — often called human on the loop. The system operates independently, but the human can follow developments, investigate deviations, and intervene. It resembles the relationship between System 1 and System 2: System 2 does not analyze everything System 1 does, but can be activated when something is amiss or when the consequences require greater attention.

The challenge is that this activation does not always happen automatically in humans. We do not necessarily detect our own biases, do not always register that the situation requires slower thinking, and may be convinced that our initial assessment is correct. Similarly, an AI system cannot determine when human control is necessary unless it is built into the system's architecture.

Therefore, modern AI systems should work with cognitive escalation.

This could include:

• Unusual or unknown situations.

• Low confidence in the system's output.

• Conflicting data sources.

• Decisions with significant economic consequences.

• Decisions that affect human rights, health, or safety.

• Actions that are not easily reversible.

• Situations where the system operates outside its normal application area.

Here, the AI system should not just continue — it should escalate. But the escalation should not merely send a message to a human. It should present the situation in a way that helps the human activate System 2.

Design for System 2

If we want real human control, the user interface and workflow must be designed for reflection. It is not enough to show a big green button with the text "Approve" — the human should be able to understand:

• What the system recommends.

• What data the recommendation is based on.

• How uncertain the system is.

• What alternatives were rejected.

• What consequences an approval may have.

• What information is missing.

• Whether the situation resembles previously known error scenarios.

A system can also require the human to actively justify a critical decision rather than just accept it. It can present counterarguments or alternative interpretations, clarify when the recommendation lies outside the system's normal competence, and vary the workflow so that the control task does not become a mechanical series of identical approvals.

Human oversight is therefore as much a design problem as it is a question of roles and responsibilities: if the workflow makes it easiest to accept the AI's recommendation without reflection, that is often what will happen.

AI also has a cognitive budget

There is also another parallel between human and artificial intelligence. The human brain tries to save energy and uses automatic processes because deep concentration is demanding. AI systems operate similarly under resource constraints — token budgets, response times, costs, and limits on how much context they can process. An AI system will therefore not necessarily perform maximum analysis in all situations.

Some tasks can be solved quickly through a single model run. Others may require multiple models, external tools, validation, simulation, or human assessment. In this way, AI systems can be designed with different forms of "thinking": a fast and cheap layer handles routine tasks, a more thorough layer is activated in cases of uncertainty or risk, and a human is involved when the system cannot create sufficient confidence on its own.

This resembles System 1 and System 2, but with an important difference. In humans, the architecture has arisen biologically and culturally. In AI systems, we can design it consciously.

When should humans be in the loop?

It is tempting to say that humans should always be involved in important decisions. But human involvement is not automatically a guarantee of quality — humans are inconsistent, become tired, are influenced by time pressure, and may hold onto decisions based on status, emotions, or organizational interests.

The question is less about whether humans or AI are generally better, and more about how their strengths and weaknesses are combined. AI is strong at processing large amounts of data, detecting patterns, and executing uniform processes. Humans are strong at understanding context, interpreting ambiguous goals, assessing ethical dilemmas, and taking responsibility for consequences that cannot be reduced to a simple objective function.

Human in the loop makes particular sense when:

• The consequences are serious.

• The goal cannot be precisely defined.

• There are ethical or legal considerations.

• Data does not tell the whole story.

• The situation is new or unusual.

• A decision requires accountability, explanation, and legitimacy.

• Errors are difficult or impossible to correct.

In other situations, human approval may simply slow down the process without making it better. The degree of human involvement should therefore be risk-based and situation-dependent.

From human in the loop to intelligence in the loop

One could argue that we should talk about intelligence in the loop rather than human in the loop — what matters is that the right form of intelligence is applied at the right time. But the term has a limitation: the AI system is already a form of intelligence in the loop, so it does not tell us what specific function the human should perform. The same applies to some extent to consciousness in the loop — strong as a metaphor, but philosophically unclear.

The most precise term is probably deliberation in the loop: conscious, critical, and purposeful reflection in the loop. It is this reflection that we want to activate when the automatic process is insufficient. But as an article title, Consciousness in the Loop is stronger because it challenges our normal understanding of human control — it reminds us that a human does not necessarily think just because they are present.

Consciousness is not the default setting

We tend to imagine that human behavior is primarily the result of conscious decisions. But large parts of our lives consist of automatic reactions, learned routines, and pattern recognition — the same goes for the workplace. We are often hired for our ability to analyze, assess, and think innovatively, but over time even complex work processes can become automated habits. We reuse solutions, interpretations, and arguments that have worked in the past. It is efficient, but it can also blind us to changes.

Similarly, organizations can develop a collective System 1: "This is how we usually do it." "This is the process we have always followed." "It has worked until now." Routines become automated even when the original assumptions no longer apply.

AI can both reinforce and challenge these patterns. If trained on the organization's historical data and existing decisions, it risks automating past assumptions. But it can also be used to highlight deviations, challenge patterns, and present alternative perspectives — it depends on whether we design AI as a machine that confirms our System 1 or as a system that helps us activate System 2.

The future of automation does not require more control everywhere

The first reaction to AI is often to place humans in all loops. This is understandable — when a technology is new and uncertain, we want extra control. But as systems become more reliable, it becomes neither realistic nor desirable for humans to review all decisions.

The future model will likely not be human in the loop everywhere, but a combination:

• Full automation at low risk and high certainty.

• Human on the loop with continuous monitoring.

• Human in the loop for significant or unclear decisions.

• Human in command for setting goals, frameworks, and responsibilities.

• Deliberation in the loop when the system detects that the situation requires real reflection.

The last point is crucial. Human presence should not be a ceremonial approval — it should add something that the automatic system cannot provide.

Context. Judgment. Critique. Accountability.

And the ability to ask the question:

What if the entire premise is wrong?

Conclusion

Kahneman's System 1 and System 2 provide us with a useful language to understand automation. System 1 shows that automation is not something that began with artificial intelligence — humans have always relied on fast, automatic processes. System 2 shows that thorough reflection is a limited resource that cannot be applied to all decisions.

The central question thus becomes the same for both human and artificial intelligence: when can we trust automation, and when should we pause and think?

Human in the loop is only valuable if the human adds real assessment. If the human merely accepts the system's recommendation, we have not created real control — we have placed a human inside the automation.

The purpose of human oversight is therefore not to place a human in the process. It is to ensure that the right form of human thinking is activated at the right time. Otherwise, the human does not control the automation. The human has merely become yet another automated component in it.